인스트럭션 기반 임베딩: 텍스트 표현 및 성능 향상

인스트럭션 기반 임베딩은 미세 조정 없이 다양한 작업별 임베딩을 생성하여 다양한 도메인과 작업에서 기존 모델보다 3.4% 더 뛰어난 성능을 발휘합니다.

### 하나의 임베딩으로 모든 것을 지시하기: 다양한 작업을 위한 지침 기반 텍스트 임베딩
* 📚 텍스트 임베딩은 텍스트를 고정 길이 벡터로 표현한 다음 **순위 변경이나 추천과 같은 작업에 사용**하는 것을 포함합니다.
* 🎯 기존 임베딩은 새로운 작업이나 도메인에 적용할 때 종종 성능 저하가 발생하여 **작업별 데이터 세트에 대한 미세 조정**이 필요합니다.
* 🧠 제안된 '강사' 프레임워크는 **미세 조정 없이도 작업별 임베딩을 생성**하여 다양한 도메인과 작업에 원활하게 적용할 수 있습니다.
* 🛠 강사 프레임워크는 **입력을 포함**하고 작업별 지침을 제공함으로써 **동일한 입력에 대한 작업별 표현**을 생성합니다.
* 📊 이 모델은 **330개의 주석이 달린 텍스트 임베딩 데이터 세트에 대한 명령어 기반 미세 조정**을 통해 구문 관계에 초점을 맞추고 다양한 도메인과 작업에서 **3.4%의 성능 향상**을 달성했습니다.
* 🔍 강사 모델은 **교수법 기반 학습**을 활용하여 작업 능력과 도메인 적합성을 향상시킴으로써 **기존 모델보다 뛰어난 성능**을 발휘합니다.
* 🏆 빌보드 프롬프트 리트리버를 사용하여 70개의 다운스트림 작업을 벤치마킹한 결과, **강사 모델은 GTR 모델을 포함한 다른 모델보다 우수한 성능**을 보였습니다.
* 🌐 강사 모델은 작업 전문 모델에 비해 **일반화 성능이 향상**되어 다양한 작업과 도메인을 처리하는 데 있어 견고함을 보여줍니다.

Hello, this is NLP. Today's topic is Instruction-based Embedding. The title of the paper is One Embedding to Instruct Them All: Instruction Text Embedding presented at ACL 2023. So, what is text embedding? It is representing text as vectors of fixed length and size. After representing, these embeddings are utilized and applied to tasks such as re-ranking or recommendation. However, existing embeddings face several challenges. One of them is performance degradation for new tasks or domains. For example, embeddings trained in a supervised manner like DP or CS may show significant performance differences depending on the domain or task. Therefore, the typical way to address this is by applying them to different domains or tasks through fine-tuning on task-specific datasets.

I need data, so today, the methodology being discussed is Instructor. Instruction-based generation is a way to create Instruction-Based Stations. One advantage is that it can generate Snadoware embeddings without the need for fine-tuning, which has been widely used in the past. Therefore, there is no need for dataset annotation. Secondly, it has the advantage of being applied seamlessly to various domains or tasks without fine-tuning. So, how does this Instructor work? It embeds the input and provides instructions tailored to each task domain. For example, when embedding a Cinderella love story, different instructions are given at the beginning, depending on the task. Then, different vectors are created for the same input, such as classifiers. Let's read this together. Yes, it's called Rekay Question Detection, a task like this.

It is said that they applied it, and they collected data from 330 datasets using contrastive loss, uploading similar pairs and reducing dissimilar ones to instruct the learning process. So, when training this, they directly constructed the dataset, which consists of 330 annotated text embedding datasets. Through instruction-based fine-tuning, they focused on seeking syntactic relationships and improving performance across various domains and tasks. When applied to various domains and tasks, it outperforms existing models by about 3.4%. Following this, the analysis will delve into the importance of instruction-based fine-tuning. Let's look into it in detail later. So, the core idea is that the instructor has a limited method of encoding the input.

During the process, encoding is done along with text instructions, and this can create task-specific presentations for the same input x. Key features do not require additional training, and it is possible to differentiate multiple tasks with a single language model. So, let's delve into how to create this instructor. They used a single encoder architecture, the GTR model, for the encoder. Therefore, after combining the input text x and instructions, they pass through the encoder, apply PL, and finally create embeddings through the hidden station. That's how they will create embeddings. Now, let's briefly explain this training object. The learning is done by dividing the output into good output in the text text representation. For example, the tasks vary greatly, but the way of learning is.

It seems like the input that was put in together is the input, and the output will be the output, right? So, after calculating the cosine, it's like cheating like this, right? Through this, by using the inch sample loss, we increase the similarity among similar positives and among negatives, it's a way to do it. It means that these two could change now, or something like that, right? And in this medial multitask embedding data inception, gathering data, the existing data is insufficient, right? It's not there, right? In a way, it's not there. Therefore, from 330 diverse datasets in different task categories and domains, they created a dataset called Inception Harmony. They obtained 300 datasets from there, and for the rest, they took 30 datasets from existing datasets for embedding training. And they used the Sentence T5 embedding to calculate this cosine similarity.

In this way, you choose positive and negative pairs evenly in the learning process. So, you calculate the cosine similarity to find the highest for positive and the lowest for negative. This is how you obtain the hard negative. By doing this, you mention that it is not feasible for people to do this manually, that's why they select positive and negative through sentence T5 embeddings. They explain that they created an instruction template by putting in what the data tab is, what its purpose is, and what the domain is. Therefore, people have created prompts one by one. For example, for open-domain queries, it is created like ENT DOC foral.

There is an instruction here. It says that you need to handle cute and lee separately. So, the point is that two different instructions need to be included. The trick is that even if two instructions are the same, it's okay. For example, in cases like classification or STS, when you want to find a model or do classification, there's no need for two different instructions. So, by collecting these two different types of datasets, you can create instructions like this. So, when you learn like this, it is said that task ability, domain suitability, has improved. What does this mean? It means that there was no need to specialize in a specific task, but rather more general life and additional training in various domains were not needed. Now, we need to verify this. So, how do we verify this benchmarking MT embedding?

It is said that it was done through the Billboard Prompt Retriever, and the metadata selected 70 downstream tasks for the metadata and trained them. Now we need to look at the results. So, it is to inform about the characteristics of each data. Especially, in the case of MTB, it verifies the tasks of the categories in 56 datasets and is most commonly used when testing embeddings. Therefore, it is said that it is directly compared to the GTR model. Because GTR was additionally fine-tuned, my model outperformed all other models today. GTR Large improved performance by nearly 18.3%, and particularly showed significant improvement in the Text Valueation Classification Clustering Task. And it says that it showed a significant improvement in the Text Valueation Classification Clustering Task compared to other SOTA models like Cent T5 XXL.

In this case, even though it's a SOTA model, today's instructor won with a much smaller model size of 4.8 million instead of 4.3 million, which is 335 million. It won by a much smaller margin, and when compared to the same gtr, the performance has improved like this. Since this one also learned as a gtr base, there is relative gain here. Yes, it outperformed all other base models. When doing this, there is a rebase model and a similarity-based model. This retriever base, for example, performs well in retrieval or ranking because it is trained for the task at hand and has limitations. But they say these models cannot handle semantic text tasks or classifications. On the contrary, models that perform well on sts similarity-based tasks cannot do retrieval well. So, they show each other here.

It's about having different task domains. Even if it's not good when the task domain is different, through instructors, the embeddings generated are showing good performance in various tasks. It is explaining that the performance has improved in terms of generalization compared to models specialized in tasks. Now, let's look at the analysis. First of all, it says that this allows various training. It talks about instructions like this. As I mentioned earlier, metrics like text similarity are used for this dataset. These metrics are asymmetric, retrieval-based, and so on. When divided into these two categories and trained with each of them, when training metrics and training embeddings separately, and training them together, with or without instructions, it is seen that the performance is much better when trained with instructions.

It was good, and furthermore, when mixing these two, instructions should be provided to improve performance. When mixing two, the performance did not improve. Therefore, instructions allow for various training together, enhancing performance when done together. Even if only one is done, having instructions shows stronger performance. This is about showing the importance of these instructions. Secondly, it is showing about the robustness. This is about whether there is a dealer i' or a super ni. This super ni provides instructions in over 300 diverse datasets, so training with various datasets increases robustness. Even so, there were times when the best performance was good, but usually, the models are a bit weak in precision. However, with a bit more, now five.

When testing the LaFRE instruction, a model that learns various tasks together was much more robust. Therefore, it is good to train on various tasks together. The instruction is talking about these things. Next, the analysis is about the complexity of instructions. There can be different levels of instruction complexity, from no instruction at all to simply giving tags, giving basic instructions, or giving detailed instructions. It is said that as more detailed information is provided and instruction tuning is done, the performance improves. Even just giving very simple instruction tags has been shown to improve performance. So, the higher the instruction complexity, the better the performance. Then, there is a discussion about size and fine-tuning of instructions. This is about GTR and GTR fine-tuning instructions, showing the difference in scores here.

So, the larger the model, the performance difference is increasing across all sizes, but especially excels in larger sizes. This is what they are talking about. Next, regarding domain shift, this instruction is questioning whether the mitigation [music] base tuning was very effective for domain shift. Next, when it comes to how much improvement there was in embeddings in TSN, in the case of insertion, when different entities were close, that's not good, right? Different entities should be farther apart, and similar ones should be closer. However, with instruction together, the embeddings clustered more closely, is what they are saying. So, the embedding space was better distinguished. That's the discussion. Therefore, there are similar text embeddings, but embeddings are really important in NLP, vision, and many other areas.

It's the way they are used, so there are various methods to create embeddings now. However, they lack diversity. Specialized models and zero-shot transfer, which is a concept similar to domain shift, exist. When transferring with zero-shot, the ability is not very good. Here, the benchmark of MTB Data Set has been limited, but due to comprehensive benchmarks, various model performances for various tasks have been consistently good. If a model performance is good, it can be even better. Therefore, the need for better models is emphasized. Also, there are words about instruction fine-tuning. In the past, instruction tuning was conducted for L language models, but here, it was done when creating embeddings. How.

You could say that Beding is also an embedding language model, but that's right. You could say that, but when creating embeddings, it seems that they were initially limited. So, in terms of the trick data I mentioned earlier, in the case of retrieval, and in the case of seeking linguistic structure, that's why they combined that balanced training approach and utilized such instruction tuning, which was more effective in various tasks. They are talking about this. Moreover, in any case, they were able to adapt more diversely to different domains or tasks. This is the story. So, in conclusion, as a model that can perform various functions, now, the way to generate Natural Language Instruction text embeddings. Here, through the limited media dataset, various datasets and instructions are gathered. It has been learned, and it has.

I have achieved high scores in various text embedding benchmarks and performed well in prompt retrieval. Yes, you did very well. Moreover, I resolved various tasks through minimal data-specific tuning. When training the Jerrys, of course, instruction tuning was introduced for various language models to perform different tasks. This was done to understand the words well. Similarly, it is said that when this model with the embedder encoder underwent instruction tuning, it was able to generate embeddings diversely.

Instruction-finetuned한 Embedding[One Embedder, Any Task: Instruction-Finetuned Text Embeddings]|꼬꼬엔

NLP와 프롬프트 엔지니어링에 대해서 알아보자! 자연어처리 마스터가 되기 위해서... 
LLM(ChatGPT, GPT, Bard...)는 미래가 아니라 현재입니다. 잘 알아야 잘 쓸 수 있고 그것을 응용할 수 있습니다. 같이 Large Language Model의 흐름에 올라타세요.
 

#ai #nlp #chatgpt #gpt #prompt #promptengineering #bard #deeplearning #machinelearning #llm #largelanguagemodel #prompting #prompt #프롬프트 #프롬프팅 #프롬프트엔지니어링 #rag


NLP 코기

인스트럭션 기반 임베딩: 하나의 임베딩으로 모든 것을 지시하기

텍스트 임베딩은 텍스트를 고정 길이 벡터로 표현하는 것으로, 순위 재지정 및 추천과 같은 작업에 사용됩니다.

인스트럭션 기반 임베딩은 인스트럭터 방식을 사용하여 입력에 대한 작업별 인스트럭션을 제공하므로 미세 조정 없이도 다양한 도메인에 원활하게 적용할 수 있습니다.

인스트럭터 방식은 단일 인코더 아키텍처인 GTR 모델을 사용하여 입력 텍스트와 인스트럭션을 결합하고 히든 스테이션을 통해 임베딩을 생성하며, 다양한 도메인과 작업에서 기존 모델보다 3.

4% 더 우수한 성능을 발휘하는 것으로 나타났습니다.

향상된 임베딩 훈련을 위한 인셉션 하모니 데이터 세트 생성하기

330개의 다양한 데이터 세트를 수집하여 임베딩 학습을 위해 이 중 300개의 데이터 세트를 사용하여 Inception Harmony 데이터 세트를 만들었습니다.

그런 다음 문장 T5 임베딩을 사용하여 코사인 유사도를 계산하여 학습 과정에서 긍정과 부정 쌍을 고르게 선택할 수 있도록 했습니다.

이 방법은 이전에는 수동으로 할 수 없었던 선택 프로세스를 자동화하는 것을 목표로 했습니다.

또한 다양한 작업 범주와 도메인에 대한 지침 템플릿을 생성하여 궁극적으로 전문 교육 없이도 작업 능력과 도메인 적합성을 향상시킬 수 있도록 Inception Harmony 데이터 세트를 만들었습니다.

빌보드 프롬프트 리트리버가 모델 성능에 미치는 영향

빌보드 프롬프트 리트리버 메타데이터는 임베딩 테스트를 위한 MTB에 중점을 두고 70개의 다운스트림 작업을 훈련하는 데 사용되었습니다.

이 모델은 다른 SOTA 모델보다 성능이 뛰어났으며, 특히 텍스트 평가 분류 클러스터링 작업에서 더 작은 크기에도 불구하고 18.

3%의 상당한 개선을 보였습니다.

검색 기반 모델은 검색 및 순위 지정 작업에서는 탁월했지만 시맨틱 텍스트 작업 처리에는 한계가 있었고, 유사도 기반 모델은 그 반대의 결과를 보여줌으로써 다양한 작업 영역이 성능에 미치는 영향을 보여주었습니다.

지침을 통한 훈련은 텍스트 유사도 메트릭, 검색 기반 작업 등 다양한 작업에서 모델 성능을 크게 향상시켜 지침 훈련의 중요성을 강조했습니다.

교육에서 지침이 성과와 견고성에 미치는 영향

훈련 모델에 대한 지침을 함께 제공하면 특히 다양한 데이터 세트를 다룰 때 성능이 향상됩니다.

명령어의 복잡성과 미세 조정도 성능 향상에 중요한 역할을 합니다.

또한 모델의 크기와 도메인 전환을 처리하는 데 있어 명령어의 효과는 NLP 및 비전을 비롯한 다양한 작업에서 결과를 개선하기 위해 고려해야 할 중요한 요소입니다.

임베딩을 생성하는 다양한 방법이 존재하지만, 다양성이 부족하고 MTB 데이터 세트에 대한 벤치마킹이 제한되어 있습니다.

특히 다양한 작업에서 더 나은 모델의 필요성과 인스트럭션 미세 조정의 효과가 강조되고 있습니다.

이 모델은 다양한 임베딩을 생성하기 위한 인스트럭션 튜닝을 도입하여 텍스트 임베딩 벤치마크와 신속한 검색에서 높은 성능을 보여주었습니다.