Instruction-Based Embedding: Enhancing Text Representation and Performance

Instruction-based embedding generates diverse task-specific embeddings without fine-tuning, outperforming existing models by 3.4% across various domains and tasks.

### One Embedding to Instruct Them All: Instruction-Based Text Embedding for Diverse Tasks
* 📚 Text embedding involves representing text as **fixed-length vectors**, which are then used for tasks like **re-ranking or recommendation**.
* 🎯 Existing embeddings often suffer from performance degradation when applied to new tasks or domains, requiring **fine-tuning on task-specific datasets**.
* 🧠 The proposed "Instructor" framework generates **task-specific embeddings without the need for fine-tuning**, enabling **seamless application to various domains and tasks**.
* 🛠 By embedding the input and providing task-specific instructions, the Instructor framework creates **task-specific representations** for the same input.
* 📊 Through instruction-based fine-tuning on a dataset of 330 annotated text embedding datasets, the model focuses on **syntactic relationships** and achieves a **3.4% performance improvement** across various domains and tasks.
* 🔍 The Instructor model outperforms existing models by leveraging **instruction-based learning** to enhance **task ability** and **domain suitability**.
* 🏆 In benchmarking against 70 downstream tasks using the Billboard Prompt Retriever, the Instructor model **outperforms** other models, including the GTR model.
* 🌐 The Instructor model demonstrates **improved generalization performance** compared to task-specialized models, showcasing its **robustness in handling diverse tasks and domains**.

Hello, this is NLP. Today's topic is Instruction-based Embedding. The title of the paper is One Embedding to Instruct Them All: Instruction Text Embedding presented at ACL 2023. So, what is text embedding? It is representing text as vectors of fixed length and size. After representing, these embeddings are utilized and applied to tasks such as re-ranking or recommendation. However, existing embeddings face several challenges. One of them is performance degradation for new tasks or domains. For example, embeddings trained in a supervised manner like DP or CS may show significant performance differences depending on the domain or task. Therefore, the typical way to address this is by applying them to different domains or tasks through fine-tuning on task-specific datasets.

I need data, so today, the methodology being discussed is Instructor. Instruction-based generation is a way to create Instruction-Based Stations. One advantage is that it can generate Snadoware embeddings without the need for fine-tuning, which has been widely used in the past. Therefore, there is no need for dataset annotation. Secondly, it has the advantage of being applied seamlessly to various domains or tasks without fine-tuning. So, how does this Instructor work? It embeds the input and provides instructions tailored to each task domain. For example, when embedding a Cinderella love story, different instructions are given at the beginning, depending on the task. Then, different vectors are created for the same input, such as classifiers. Let's read this together. Yes, it's called Rekay Question Detection, a task like this.

It is said that they applied it, and they collected data from 330 datasets using contrastive loss, uploading similar pairs and reducing dissimilar ones to instruct the learning process. So, when training this, they directly constructed the dataset, which consists of 330 annotated text embedding datasets. Through instruction-based fine-tuning, they focused on seeking syntactic relationships and improving performance across various domains and tasks. When applied to various domains and tasks, it outperforms existing models by about 3.4%. Following this, the analysis will delve into the importance of instruction-based fine-tuning. Let's look into it in detail later. So, the core idea is that the instructor has a limited method of encoding the input.

During the process, encoding is done along with text instructions, and this can create task-specific presentations for the same input x. Key features do not require additional training, and it is possible to differentiate multiple tasks with a single language model. So, let's delve into how to create this instructor. They used a single encoder architecture, the GTR model, for the encoder. Therefore, after combining the input text x and instructions, they pass through the encoder, apply PL, and finally create embeddings through the hidden station. That's how they will create embeddings. Now, let's briefly explain this training object. The learning is done by dividing the output into good output in the text text representation. For example, the tasks vary greatly, but the way of learning is.

It seems like the input that was put in together is the input, and the output will be the output, right? So, after calculating the cosine, it's like cheating like this, right? Through this, by using the inch sample loss, we increase the similarity among similar positives and among negatives, it's a way to do it. It means that these two could change now, or something like that, right? And in this medial multitask embedding data inception, gathering data, the existing data is insufficient, right? It's not there, right? In a way, it's not there. Therefore, from 330 diverse datasets in different task categories and domains, they created a dataset called Inception Harmony. They obtained 300 datasets from there, and for the rest, they took 30 datasets from existing datasets for embedding training. And they used the Sentence T5 embedding to calculate this cosine similarity.

In this way, you choose positive and negative pairs evenly in the learning process. So, you calculate the cosine similarity to find the highest for positive and the lowest for negative. This is how you obtain the hard negative. By doing this, you mention that it is not feasible for people to do this manually, that's why they select positive and negative through sentence T5 embeddings. They explain that they created an instruction template by putting in what the data tab is, what its purpose is, and what the domain is. Therefore, people have created prompts one by one. For example, for open-domain queries, it is created like ENT DOC foral.

There is an instruction here. It says that you need to handle cute and lee separately. So, the point is that two different instructions need to be included. The trick is that even if two instructions are the same, it's okay. For example, in cases like classification or STS, when you want to find a model or do classification, there's no need for two different instructions. So, by collecting these two different types of datasets, you can create instructions like this. So, when you learn like this, it is said that task ability, domain suitability, has improved. What does this mean? It means that there was no need to specialize in a specific task, but rather more general life and additional training in various domains were not needed. Now, we need to verify this. So, how do we verify this benchmarking MT embedding?

It is said that it was done through the Billboard Prompt Retriever, and the metadata selected 70 downstream tasks for the metadata and trained them. Now we need to look at the results. So, it is to inform about the characteristics of each data. Especially, in the case of MTB, it verifies the tasks of the categories in 56 datasets and is most commonly used when testing embeddings. Therefore, it is said that it is directly compared to the GTR model. Because GTR was additionally fine-tuned, my model outperformed all other models today. GTR Large improved performance by nearly 18.3%, and particularly showed significant improvement in the Text Valueation Classification Clustering Task. And it says that it showed a significant improvement in the Text Valueation Classification Clustering Task compared to other SOTA models like Cent T5 XXL.

In this case, even though it's a SOTA model, today's instructor won with a much smaller model size of 4.8 million instead of 4.3 million, which is 335 million. It won by a much smaller margin, and when compared to the same gtr, the performance has improved like this. Since this one also learned as a gtr base, there is relative gain here. Yes, it outperformed all other base models. When doing this, there is a rebase model and a similarity-based model. This retriever base, for example, performs well in retrieval or ranking because it is trained for the task at hand and has limitations. But they say these models cannot handle semantic text tasks or classifications. On the contrary, models that perform well on sts similarity-based tasks cannot do retrieval well. So, they show each other here.

It's about having different task domains. Even if it's not good when the task domain is different, through instructors, the embeddings generated are showing good performance in various tasks. It is explaining that the performance has improved in terms of generalization compared to models specialized in tasks. Now, let's look at the analysis. First of all, it says that this allows various training. It talks about instructions like this. As I mentioned earlier, metrics like text similarity are used for this dataset. These metrics are asymmetric, retrieval-based, and so on. When divided into these two categories and trained with each of them, when training metrics and training embeddings separately, and training them together, with or without instructions, it is seen that the performance is much better when trained with instructions.

It was good, and furthermore, when mixing these two, instructions should be provided to improve performance. When mixing two, the performance did not improve. Therefore, instructions allow for various training together, enhancing performance when done together. Even if only one is done, having instructions shows stronger performance. This is about showing the importance of these instructions. Secondly, it is showing about the robustness. This is about whether there is a dealer i' or a super ni. This super ni provides instructions in over 300 diverse datasets, so training with various datasets increases robustness. Even so, there were times when the best performance was good, but usually, the models are a bit weak in precision. However, with a bit more, now five.

When testing the LaFRE instruction, a model that learns various tasks together was much more robust. Therefore, it is good to train on various tasks together. The instruction is talking about these things. Next, the analysis is about the complexity of instructions. There can be different levels of instruction complexity, from no instruction at all to simply giving tags, giving basic instructions, or giving detailed instructions. It is said that as more detailed information is provided and instruction tuning is done, the performance improves. Even just giving very simple instruction tags has been shown to improve performance. So, the higher the instruction complexity, the better the performance. Then, there is a discussion about size and fine-tuning of instructions. This is about GTR and GTR fine-tuning instructions, showing the difference in scores here.

So, the larger the model, the performance difference is increasing across all sizes, but especially excels in larger sizes. This is what they are talking about. Next, regarding domain shift, this instruction is questioning whether the mitigation [music] base tuning was very effective for domain shift. Next, when it comes to how much improvement there was in embeddings in TSN, in the case of insertion, when different entities were close, that's not good, right? Different entities should be farther apart, and similar ones should be closer. However, with instruction together, the embeddings clustered more closely, is what they are saying. So, the embedding space was better distinguished. That's the discussion. Therefore, there are similar text embeddings, but embeddings are really important in NLP, vision, and many other areas.

It's the way they are used, so there are various methods to create embeddings now. However, they lack diversity. Specialized models and zero-shot transfer, which is a concept similar to domain shift, exist. When transferring with zero-shot, the ability is not very good. Here, the benchmark of MTB Data Set has been limited, but due to comprehensive benchmarks, various model performances for various tasks have been consistently good. If a model performance is good, it can be even better. Therefore, the need for better models is emphasized. Also, there are words about instruction fine-tuning. In the past, instruction tuning was conducted for L language models, but here, it was done when creating embeddings. How.

You could say that Beding is also an embedding language model, but that's right. You could say that, but when creating embeddings, it seems that they were initially limited. So, in terms of the trick data I mentioned earlier, in the case of retrieval, and in the case of seeking linguistic structure, that's why they combined that balanced training approach and utilized such instruction tuning, which was more effective in various tasks. They are talking about this. Moreover, in any case, they were able to adapt more diversely to different domains or tasks. This is the story. So, in conclusion, as a model that can perform various functions, now, the way to generate Natural Language Instruction text embeddings. Here, through the limited media dataset, various datasets and instructions are gathered. It has been learned, and it has.

I have achieved high scores in various text embedding benchmarks and performed well in prompt retrieval. Yes, you did very well. Moreover, I resolved various tasks through minimal data-specific tuning. When training the Jerrys, of course, instruction tuning was introduced for various language models to perform different tasks. This was done to understand the words well. Similarly, it is said that when this model with the embedder encoder underwent instruction tuning, it was able to generate embeddings diversely.

Instruction-finetuned한 Embedding[One Embedder, Any Task: Instruction-Finetuned Text Embeddings]|꼬꼬엔

NLP와 프롬프트 엔지니어링에 대해서 알아보자! 자연어처리 마스터가 되기 위해서... 
LLM(ChatGPT, GPT, Bard...)는 미래가 아니라 현재입니다. 잘 알아야 잘 쓸 수 있고 그것을 응용할 수 있습니다. 같이 Large Language Model의 흐름에 올라타세요.
 

#ai #nlp #chatgpt #gpt #prompt #promptengineering #bard #deeplearning #machinelearning #llm #largelanguagemodel #prompting #prompt #프롬프트 #프롬프팅 #프롬프트엔지니어링 #rag


NLP 코기

Instruction-Based Embedding: One Embedding to Instruct Them All

Text embedding is the representation of text as fixed-length vectors, used for tasks like re-ranking and recommendation.

Instruction-based embedding, using the Instructor method, provides task-specific instructions for the input, allowing for seamless application to different domains without the need for fine-tuning.

The Instructor method uses a single encoder architecture, the GTR model, to combine input text and instructions, creating embeddings through a hidden station, and has been shown to outperform existing models by 3.

Creating the Inception Harmony Dataset for Improved Embedding Training

By gathering 330 diverse datasets, the Inception Harmony dataset was created, using 300 of these datasets for embedding training.

The Sentence T5 embedding was then used to calculate cosine similarity, allowing for the even selection of positive and negative pairs in the learning process.

This method aimed to automate the selection process, which was previously unfeasible to do manually.

The Inception Harmony dataset also involved creating instruction templates for different task categories and domains, ultimately improving task ability and domain suitability without the need for specialized training.

The Impact of Billboard Prompt Retriever on Model Performance

The Billboard Prompt Retriever metadata was used to train 70 downstream tasks, with a focus on MTB for testing embeddings.

The model outperformed other SOTA models, showing a significant 18.

3% improvement, especially in Text Valuation Classification Clustering Task, despite its smaller size.

The retriever base model excelled in retrieval and ranking tasks but had limitations in handling semantic text tasks, while similarity-based models showed the opposite, demonstrating the impact of different task domains on performance.

Training with instructions significantly improved model performance in various tasks, including text similarity metrics, retrieval-based tasks, and more, highlighting the importance of instructional training.

The Impact of Instructions on Performance and Robustness in Training

Providing instructions for training models together improves performance, especially when dealing with diverse datasets.

The complexity of instructions and fine-tuning also plays a significant role in enhancing performance.

Additionally, the size of the model and the effectiveness of instructions in dealing with domain shift are crucial factors to consider for improved results in various tasks, including NLP and vision.

Various methods for creating embeddings exist, but there is a lack of diversity and limited benchmarking for the MTB Data Set.

The need for better models and the effectiveness of instruction fine-tuning, especially in diverse tasks, are emphasized.

The model has shown high performance in text embedding benchmarks and prompt retrieval, with the introduction of instruction tuning for generating diverse embeddings.