트랜스포머의 주의 집중 메커니즘 이해하기

트랜스포머의 주의 메커니즘은 단어의 문맥적 의미를 모델링하는 데 매우 중요합니다. 이 메커니즘은 키, 쿼리 및 값 행렬을 사용하여 임베딩을 업데이트합니다.

### 트랜스포머의 주의 집중 메커니즘의 기초
* 🧠  트랜스포머의 주의 메커니즘은 임베딩을 점진적으로 조정하여 보다 풍부한 문맥적 의미를 인코딩함으로써 데이터를 처리할 수 있게 해줍니다.
* 📊  주의 메커니즘을 통해 모델은 단어 간의 관계를 추론하고 문맥에 따라 의미를 업데이트할 수 있습니다.
* 🔢  주의 메커니즘은 주의 패턴을 계산하여 다른 모든 단어의 의미를 업데이트할 때 각 단어의 관련성을 결정합니다.
* 🤖  주의 메커니즘은 여러 개의 머리가 병렬로 실행되어 다양한 문맥적 의미를 학습하는 다중 머리 주의로 구성됩니다.
* 💡  병렬화가 가능한 주의 메커니즘의 특성은 딥러닝 모델에서 높은 성능과 확장성을 제공하는 데 기여합니다.

In the previous chapter, we examined the internal operation principle of the transformer. This is one of the key technologies in many large-scale language models and the latest AI waves. Attention mechanism was first introduced in the famous 2017 paper 'Attention is All You Need', and in this chapter, we will take a close look at what this attention mechanism is and explain how data is processed by visualizing it. In summary, please keep in mind the following important context. The goal of the model that you and I are researching is to predict what word will come next after receiving a piece of text. The input text is divided into small pieces called tokens, which are often words or word fragments, but for the sake of simplification in this video, let's assume tokens are always words to make it easier for you and me to think.

The first step of the transformer is to connect each token to a high-dimensional vector, which is called embedding. The most important idea I had in mind is how all possible directions of embeddings in this high-dimensional space can correspond to semantic meanings. In the previous chapter, we saw an example of how directions can correspond to gender, meaning that by adding a specific step in this space, we can move from the implication of a masculine noun to the implication of the corresponding feminine noun. This is just one example of how many different directions in this high-dimensional space can correspond to many different aspects of word meanings. The goal of the transformer is to gradually adjust these embeddings to not just encode individual words but to contain much richer contextual meanings.

This is it. I would like to inform you in advance that many people find the core element of attention mechanism in transformers very confusing, so there is no need to worry if it takes time to adapt. I believe it is necessary to consider some examples of types of operations that require attention before delving into detailed calculations and all matrix multiplications. Think of phrases like true mole of the United States, carbon dioxide dot, and mole biopsy. You and I are aware that the word mole has different meanings depending on the context. However, after splitting the text and connecting each token with vectors following the first step of the transformer, the initial token embeddings practically do not reference the context as they are a lookup table, so the vectors associated with mole are the same in all cases. Only in the next step of the transformer, the surrounding embeddings are referenced.

You will have the opportunity to convey information through transformers. In this embedding space, there are various directions encoding various meanings of the word mole, and a well-trained attention block can be considered to calculate what needs to be added to the general embedding based on the function of the context and move in one of these specific directions. For example, let's think about embedding the word tower. This would probably be a direction in a very general and nonspecific space associated with many other large and tall nouns. If the word Eiffel is directly attached in front of this word, you can imagine that the mechanism would update to point the vector more specifically encoding the Eiffel Tower, i.e., vectors associated with Paris, France, and things made of steel. If the word miniature is attached in front, the vector would no longer be about large and tall objects.

You need to update it more so that it is not associated. Attention blocks do more than just fine-tune the meaning of words; they allow the model to move from the information encoded in one embedding to information in another embedding, potentially information that is quite far away, much richer than a single word. In the previous chapter, after all vectors, including various attention blocks, passed through the network, we saw that the calculations performed to predict the next token are entirely a function of the last vector of the sequence. For example, imagine that the input text occupies most of a detective novel and is written 'Therefore, the culprit was the murderer' close to the end. For the model to predict the next word accurately, the last vector of the sequence updated by all attention blocks needs to be much richer than individual words.

You need to represent a lot of information and somehow encode all the information in the entire context window related to predicting the next word. However, let's look at a much simpler example to examine the calculations step by step. Imagine the phrase fluffy blue creatures wandering in the blue forest in the input. For now, let's assume that the only type of update we care about is when an adjective adjusts the meaning of the noun it describes. What I am describing now can be considered as one attention head, and later, we will see how multiple different heads make up an attention block and run in parallel. Again, the initial embedding for each word is a high-dimensional vector that encodes the meaning of a specific word without context. In fact, it is not. It also encodes the position of the word. More about how the position is encoded.

There is a story, but for now, all you need to know is that you can determine what a word is and where it exists in the context as an item of this vector. Now, let's represent these embeddings with the character 'e'. The goal is to create a new sophisticated embedding set through a series of computations, where, for example, a word corresponding to a noun derives meaning from an adjective. In deep learning games, most calculations are intended to look like matrix-vector multiplication, where the matrix is full of adjustable weights, and the model learns based on the data. To be clear, I have created this example to explain the type of behavior that a person with attention does when adjectives update nouns. Like many deep learnings, the actual operation is much more difficult to analyze as it is based on adjusting numerous parameters to minimize the cost function.

It's difficult. However, when looking at various matrices filled with parameters related to this process, it would be really helpful if there were imaginary examples that could help understand everything more concretely. In the first step of this process, you can imagine asking each noun like a living thing, Are there any adjectives in front of me? Then, for words like fluffy and blue, you should be able to answer, Yes, I am an adjective and I am at that position. This question is somehow encoded into another vector, a list of numbers, which is called a query for this word. However, this query vector has a much smaller size (e.g. 128) compared to the embedding vector. Calculating this query seems to involve bringing a specific matrix and multiplying embeddings here (we will label it as wq).

Let's write the query vector a little compressed as q, and what is shown next to the arrow is to indicate that when you multiply the vector at the beginning of the arrow by this matrix, you get the vector at the end of the arrow. In this case, by multiplying all embeddings of the context by this matrix, a query vector is generated for each token. The entries of this matrix are the model's parameters, meaning they learn actual behavior from the data, and it is difficult to parse what this matrix actually does in a specific attention head. However, to imagine an example we hope to learn, let's assume that this query matrix somehow encodes the concept of mapping the inclusion of a noun in this small query space in a specific direction to find an adjective in the previous position. Who knows what impact it will have on other embeddings? Perhaps with that, we can aim for a different goal.

It may also try to achieve simultaneously. Right now, it is focusing on nouns. At the same time, it multiplies the key matrix, which is the second matrix related to this, to all embeddings. This way, a second vector sequence called keys is generated. Conceptually, keys can be thought of as potential answers to queries. This key matrix is also full of adjustable parameters and, like the query matrix, maps the embedding vectors to the same small-dimensional space. Think of keys as matching closely when queries match each other closely. In this example, you can imagine that the key matrix maps to vectors that closely match queries generating words like fluffy and blue to the word creature. To measure how well each key matches each query, calculate the dot product between each possible key-query pair. I visualize a grid full of dots.

I like things, a bigger point corresponds to a bigger point product, that is, the location where the key and query are aligned. In the case of an adjective noun example, it is shown as follows. If the key generated by Fluffy and Blue actually closely matches the query generated by the creature, the dot product of these two points will be a large positive number. In machine learning terminology, the embedding of fluff and blue can be said to represent the embedding of a creature. On the other hand, the dot product between the key of different words like the and the query for creature reflects a small value or a negative value that is unrelated to each other. Therefore, through a value grid that can be any real number from negative infinity to positive infinity, a score is given to determine how relevant each word is in updating the meanings of all other words. The method of using this score is to assign weights based on relevance according to each column and calculate a specific weighted sum.

Therefore, instead of the value range from negative infinity to infinity, I prefer values between 0 and 1, similar to a probability distribution where the sum of each column's numbers adds up to 1. If you have read from the last chapter, you probably know what to do. Normalize the values by calculating softmax along each column. In the figure, fill the grid with these normalized values after applying softmax to all columns. At this point, it is safe to think that each column assigns weights based on how relevant the word on the left is to the corresponding value at the top. We call this grid an attention pattern. Now, looking at the original Transformer paper, there is a concise way to write all of this. Here, variables q and k represent the entire arrays of query and key vectors, respectively, multiplying the query and key matrices in embeddings to obtain small vectors. The expression in the numerator represents the key and query.

This is a very concise way of representing all possible dot products in a grid between pairs. One small technical detail I didn't mention is that dividing all these values by the square root of the dimension in the corresponding key query space is helpful for numerical stability. Then, the softmax wrapping the entire expression should be understood to be applied column-wise. I will explain the term V shortly. Before that, there is one more technical detail that we have skipped so far. During the training process, when running this model on the given text examples and slightly adjusting all weights to reward or penalize based on how much probability is allocated to the actual next word of a passage, it is much more efficient to predict all possible next tokens simultaneously after each initial token of this passage.

It has been revealed. For example, in the case of the phrase we are focusing on, we can also predict which words come after the creature and which words come after the creature. This is a very good thing because it means that one educational example works effectively like multiple educational examples. For the purpose of attention patterns, it is necessary to ensure that the words that come later do not influence the previous words, otherwise, it can reveal the answer to the words that come next. What this means is that at every point here, the point representing the tokens that affect the previous token must somehow become 0. The simplest way to think about this is to set it to 0, but if you do that, the column will no longer be 1 and will not be normalized. Therefore, the common way to perform this task is to apply softmax before all.

Setting an item to negative infinity. By doing this, even though all columns turn to 0 after applying softmax, the columns are kept in a normalized state. This process is called masking. There are caution versions that do not apply this masking, but in the case of GPT examples, it is more relevant in the training phase than in running as a chatbot, always applying this masking to prevent subsequent tokens from affecting previous tokens. Another fact worth considering about this attention concentration pattern is that its size is equal to the square of the context size. Therefore, in large-scale language models, the context size can become a significant bottleneck, and extending it is not easy. Driven by the desire for a larger context window, some variations are being made to the attention mechanism to increase the scalability of the context recently.

We are focusing on the basics here. When we calculate this pattern, the model can infer which words are related to each other. Now we need to actually update the embeddings so that words can pass information to other related words. For example, let's assume we want to encode fluffy creatures more specifically by somehow moving the fluffy embedding to a different part of the 12,000-dimensional embedding space. Here, we would like to show you the simplest way first, but there are slight variations in situations that many people pay attention to. The simplest way is to use the third matrix, called the value matrix, to multiply the embedding of the first word (e.g., Fluffy). The result can be called a value vector, which is added to the embedding of the second word, in this case, added to the embedding of Creature.

Therefore, this value vector exists in a very high-dimensional space, just like embeddings. If we multiply the embedding of a word by this value matrix, you might think about what exactly needs to be added to the embeddings of other words to reflect this if the word is related to adjusting the meaning of other words. After calculating attention patterns, we will take this value matrix and multiply it by all embeddings to create a series of value vectors, so looking back at the diagram, we will set aside both keys and queries. These value vectors can be considered to have some relevance to their respective keys. For each column in this diagram, multiply each value vector by the corresponding weight of that column. For example, embedding a creature here would add a large portion of the value vectors for Fluffy and Blue, making all other value vectors either become 0 or at least very close to 0. Finally, the actual embeddings related to this column are actually multiplied.

The method of updating is to encode the meaning of Creature without the context previously, sum up all the readjusted values ​​of the columns with enthusiasm, create changes to be added (which I will label as delta-e), and then add them to the original embedding. I hope that a more sophisticated vector encoding rich meanings like a fluffy blue organism will be created. Of course, this operation is not performed on just one embedding, but by applying the same weight to all columns in this image, a series of changes are generated. By adding all these changes to the embedding, a more sophisticated embedding pops out in the area of interest, creating a complete sequence. By simplifying all these processes, you can focus at once. As explained so far, this process is parameterized by three different matrices filled with adjustable parameters: key, query, and value. Continuing from the previous chapter, GPT-3's...

Let me briefly explain the scoring calculation for counting the total number of model parameters using numbers. These key and query matrices consist of 12,288 columns matching the embedding dimension each and 128 rows matching the dimension of a smaller key-query space. This way, around 1.5 million parameters are additionally provided for each. Comparatively, looking at these value matrices, in the manner described so far, both input and output are in very large embedding spaces, so it can be seen as a square matrix with 12,288 columns and rows each. In fact, this means that approximately 150 million parameters have been added. To be clear, it can be done that way. More parameters can be allocated to the value map than to the key and query. However, in reality, it is much more efficient to allocate the same number of parameters to the value map as to the key and query.

This is particularly relevant to settings that involve running multiple attention heads simultaneously. This approach decomposes the value map into the product of two small matrices. Conceptually, it is good to think of this as a full linear map with inputs and outputs in a larger embedding space, for example, embedding the color blue to be added to a noun in the direction of blue. However, it is only a smaller number of rows like the key-query space in general. In other words, think of mapping a large embedding vector to a much smaller space. This is not a common naming convention, but I will call it a value-down matrix. The second matrix remaps from this small space back to the embedding space to generate vectors used for actual updates. I will call this a value-up matrix, which is also a different approach from the conventional one. The approach you can see in most papers is slightly different.

I will tell you in a moment. In my opinion, there is a tendency to become a bit more confusing conceptually. In terms of linear algebra specialized terms, essentially, it is restricting the entire value map to a lower-ranked transformation. Going back to the number of parameters, these four matrices are all the same size, and adding them all up results in about 6.3 million parameters for a single attention head. To be more precise, what has been described so far is called 'self-attention head' to distinguish it from variations like cross-attention that appear in other models. Although it is not related to the GPT example, for those who are curious, to explain, cross-attention is related to models that handle two types of data, such as text in one language and text or speech input in progress in another language, like ongoing transcription. Cross-attention heads look almost the same.

The only difference is that the key map and query map work on different datasets. For example, in a model that performs translation, keys may come from one language while queries may come from another language, and attention patterns can explain which word in one language corresponds to which word in another language. In this setup, masking usually does not occur because there is no concept that tokens affect previous tokens later. However, if you focus on self-attention and have understood everything so far, and if you stop here, you will understand the essence of attention. Now all that remains for us is to develop a sense of performing this task in various ways several times. In the core example, the focus was on updating nouns with adjectives, but of course, there are many ways in which context can influence the meaning of words.

When the conflicting word comes before the word car, it affects the shape and structure of the car. Also, many related words may not be grammatical. If the word wizard is in a phrase like Harry, it may refer to Harry Potter, but instead, if the words queen, sex, and William are in the phrase, the embedding of Harry may need to be updated to refer to a prince. Various parameters of these key and query matrices change to capture various interest patterns in updating context of all imaginable types, and the parameters of the value map will also vary depending on the items to be added to the embedding. The actual operation of these maps is much more difficult to interpret, as the weights are set to perform all the tasks necessary for the model to best achieve the goal of predicting the next token.

As mentioned earlier, everything explained before is a single attention block, and the entire attention block inside the transformer is composed of what is called multi-head attention, executing many tasks in parallel and using unique key queries and value maps for each. For example, GPT-3 uses 96 attention heads within each block. Considering that each one is already a bit confusing, it is certainly a lot to keep in mind. In simple terms, having 96 different key and query matrices within each head generates 96 different patterns of interest. Then, each head has a unique value matrix used to generate a sequence of 96 value vectors. All of these are summed using the respective interest pattern weights. In other words, for each position in the context, for each token, all these heads contribute to their embeddings at that position.

Generating proposed changes means creating additional change suggestions. Therefore, all proposed changes should be aggregated for each head, and then the result should be added to the original embedding at that location. This overall sum is a piece of what is output in the multi-head attention block, corresponding to one of the sophisticated embeddings that pop out at the opposite end. Again, don't worry at all if it takes time to adapt because there is a lot to consider. The general idea is to provide the model with the ability to learn different ways in which meaning can vary depending on the context by running multiple heads in parallel. Including each variant of these four matrices, aggregating the parameters for 96 heads results in approximately 600 million parameters for each multi-head attention block. For those who want to learn more about transformers, there is one more thing I would like to mention.

The value map is divided into these two different matrices, and you will remember that I mentioned labeling them as a value-down matrix and a value-up matrix. The way I have structured it is to think that there is this pair of matrices inside each attention head, and it can be implemented in this way. This could be a valid design. However, the way it is described in the paper and the way it is actually implemented are slightly different. All these value-up matrices for each head appear together processed into one giant matrix called the output matrix connected to the entire multi-head attention block. When people mention the value matrix for a given attention head, they often only refer to the first step, which I labeled as value projection into a small space. I have left a note on the screen for those who are curious.

There is a risk of deviating from the main concepts with detailed information, but I wanted to mention it so that you can know if you have read about it from other sources. Setting aside all technical nuances, in the preview of the previous chapter, we observed that data flowing through the Transformer does not simply pass through one attention block. First, it also goes through another task called a multi-layer perceptron. I will explain this in detail in the next chapter. And these two tasks are repeated several times. In other words, after a specific word partially absorbs the context, it is more likely to be influenced by subtler surrounding environments. As we go down the network, each embedding brings more meaning from all other embeddings, gradually acquiring more subtle differences itself.

I have developed a hope that goes beyond simple descriptors and grammatical structures to encode higher-level abstract ideas in the given input. I consider emotions, tones, whether it is poetic, what scientific truths are related to the work, and more. Going back to score management, GPT-3 includes 96 unique layers, so multiplying 96 to the total number of key queries and value parameters results in a total of less than 58 billion unique parameters allocated to all attention heads. Certainly, it is a large number, but it is only about one-third of the 175 billion in the entire network. Therefore, while attention is focused on all, most parameters occur in blocks between these stages. In the next chapter, I will delve into these different blocks and the training process in more detail. The most significant aspect where attention mechanisms could succeed

The reason is not the specific type of operation that this mechanism implements, but the fact that this mechanism is highly parallelizable, allowing it to execute a huge number of calculations in a short time using GPUs. One of the big lessons in deep learning over the past 10-20 years was that simply scaling up the model greatly improves performance, so parallelizable architectures have a significant advantage in enabling this. To learn more about this, I have left many links in the description. Especially, most products created by Andrei Karpathy and Chris Olah are often top-notch. While this video wanted to focus on the current state, if you are curious about more history, how we got here, and how to recreate this idea, my friend Vivek has just uploaded some more motivating videos.

Furthermore, Britt Cruz of The Art of the Problem channel provides a fantastic video on the history of large language models. Thank you.

Attention in transformers, visually explained | DL6

My name is Grant Sanderson. Videos here cover a variety of topics in math, or adjacent fields like physics and CS, all with an emphasis on visualizing the core ideas. The goal is to use animation to help elucidate and motivate otherwise tricky topics, and for difficult problems to be made simple with changes in perspective.

For more information, other projects, FAQs, and inquiries see the website: https://www.3blue1brown.com

3Blue1Brown

트랜스포머에서 주의 집중 메커니즘의 역할

주의 메커니즘은 트랜스포머 모델에서 중요한 요소로, 단어 임베딩을 조정하여 풍부한 문맥적 의미를 포착할 수 있게 해줍니다.

이 메커니즘을 통해 모델은 문맥에 따라 한 임베딩에서 다른 임베딩으로 이동하면서 단어의 의미를 미세 조정하고 풍부하게 만들 수 있습니다.

궁극적으로 주의 메커니즘은 시퀀스의 최종 벡터를 생성하는 데 중요한 역할을 하며, 이는 트랜스포머 모델에서 정확한 예측을 위해 매우 중요합니다.

주의 메커니즘을 통해 향상된 단어 임베딩 구축하기

단어 임베딩은 문장에서 단어의 의미와 위치를 인코딩하는 고차원 벡터입니다.

주의 메커니즘은 임베딩 벡터에 쿼리 및 키 매트릭스를 곱하여 각 단어에 대한 쿼리 및 키 벡터를 생성하는 것입니다.

각 키-쿼리 쌍 사이의 도트 곱을 계산하여 각 키가 각 쿼리와 얼마나 잘 일치하는지 측정함으로써 모델이 명사를 업데이트하는 형용사와 같은 단어 간의 관계에 집중할 수 있도록 합니다.

트랜스포머 모델에서 주의 패턴의 중요성

트랜스포머 모델의 관심도 패턴은 쿼리와 키 벡터 사이의 도트 곱 그리드를 사용하여 관련성에 따라 단어에 가중치를 할당합니다.

이러한 값을 소프트맥스로 정규화하면 트레이닝 과정에서 가능한 모든 다음 토큰을 동시에 효율적으로 예측할 수 있습니다.

주의 집중 패턴의 크기는 대규모 언어 모델의 확장성에 중요한 요소이며, 컨텍스트 창 확장성을 높이기 위한 변형의 필요성을 불러일으킵니다.

키, 쿼리 및 값 매트릭스로 단어 임베딩 업데이트하기

키, 쿼리, 값 행렬은 단어 임베딩을 업데이트하고 그 관계를 캡처하는 데 사용됩니다.

값 행렬은 모든 임베딩에 곱해져 일련의 값 벡터를 생성한 다음 키 및 쿼리 행렬을 사용하여 조정하여 원래 임베딩을 업데이트합니다.

이 프로세스는 세 가지 행렬로 매개변수화되며, 값 행렬은 키 행렬과 쿼리 행렬에 비해 훨씬 더 많은 수의 매개변수를 필요로 합니다.

트랜스포머에서 멀티 헤드 어텐션의 개념

트랜스포머의 다중 헤드 주의는 여러 키, 쿼리 및 값 매트릭스 세트를 사용하여 데이터에서 다양한 관심 패턴을 포착하는 것을 포함합니다.

이를 통해 모델은 컨텍스트에 따라 의미가 달라질 수 있는 다양한 방식을 학습할 수 있습니다.

GPT-3의 각 다중 헤드 관심도 블록은 96개의 관심도 헤드로 구성되며, 각 블록에는 많은 수의 매개변수(약 6억 개)가 포함됩니다.

이러한 접근 방식을 통해 모델은 다양한 주의 패턴을 동시에 고려함으로써 정교한 임베딩을 생성할 수 있습니다.

트랜스포머 모델에서 가치 행렬 구현하기

트랜스포머 모델의 가치 맵은 가치-다운 행렬과 가치-업 행렬로 나뉘며, 각 주의력 헤드에는 한 쌍의 행렬이 포함되어 있습니다.

그러나 모든 밸류업 행렬이 전체 멀티 헤드 주의 블록에 연결된 하나의 거대한 행렬로 결합되기 때문에 논문에 설명된 방식과 실제로 구현되는 방식이 다릅니다.

이 구현은 원래의 개념에서 벗어난 것입니다.

트랜스포머 모델의 데이터 흐름은 주의 블록을 통과할 뿐만 아니라 다층 퍼셉트론을 통과하며, 이 두 가지 작업을 여러 번 반복하여 입력의 미묘한 차이와 의미를 점차적으로 흡수합니다.