Analyzing Transformer Architectures and Their Implications

The main subject is Hyung Won's analysis of transformer architectures and their relevance in the context of exponentially cheaper compute driving AI research.

### Analyzing Transformer Architectures and the Dominant Driving Force of AI Research
* 🧠 Hyung Won, a research scientist at OpenAI, discusses the **importance of understanding the history and key structures of transformers** to project the **future of AI research**.
* 🔮 The **exponentially cheaper compute**, which leads to **scaling up AI models**, is identified as the **dominant driving force** in AI research.
* 💡 The common approach in AI research is to model how humans think and incorporate that into mathematical models. However, this poses a **short-term shortcut** and becomes a **bottleneck for scaling**.
* 📈 According to the "bitter lesson," the past 70 years of AI research can be summarized as developing **more general methods with weaker assumptions** and **scaling up with more data and compute**.
* 🤖 Models with less structure tend to be more scalable, but the **optimal amount of structure** depends on the **current level of compute**.
* The encoder-decoder, encoder-only, and decoder-only transformer architectures are analyzed to understand their **differences and implications for scaling**.
* 🔍 By examining the assumptions and additional structures of these architectures, it is possible to identify which ones are **no longer relevant** due to advancements in compute and algorithms.
* 🚀 Hyung Won concludes that by **revisiting and questioning the assumptions in AI problems**, we can collectively **shape the future of AI research**.

now we have Hyung Won give a talk. So he's 
currently a research scientist at on the OpenAI ChatGPT team. He has worked on various aspects of 
large language models. Things like pre-training, instruction fine-tuning, reinforcement learning 
with human feedback, reasoning, and so forth. And some of his notable works include the scaling 
flan papers such as FLAN-T5, as well as Flan-PALM, and T5X, the training framework used to train 
the PALM language model. And before OpenAI, he was at Google Brain, and he received his 
PhD from MIT. So give a hand for Hyung Won. Do we switch the slides?
All right. My name is Hyung Won, and I'm really happy to be here 
today. And this week I was thinking about-- by the way, is mic working fine? Yeah. So this 
week I thought about-- OK, I'm giving a lecture

on transformers at Stanford. What should I 
talk about? And I thought, OK, some of you in   this room and in Zoom will actually go shape the 
future of AI, so maybe I should talk about that.  It's a really important goal and ambitious, and 
we really have to get it right. So that could   be a good topic to think about. And when 
we talk about something into the future,   the best place to get an advice is to 
look into the history. And in particular,   we'll look at the early history of transformer, 
and try to learn many lessons from there.  And the goal will be to develop a unified 
perspective in which we can look into many   seemingly disjoint events. And from 
that we can probably hope to project   into the future what might be coming. And 
so that will be the goal of this lecture,   and we'll look at some of the architectures 
of the transformers, so let's get started.

Everyone I see is saying, AI is so advancing so 
fast that it's so hard to keep up. And it doesn't   matter if you have years of experience, there 
are so many things are coming out every week   that it's just hard to keep up. And I do see 
many people spend a lot of time and energy   catching up with this latest developments, the 
cutting edge and the newest thing, and then not   enough attention goes into old things because 
they become deprecated and no longer relevant.  But I think it's important, actually, to look into 
that because we really need to-- when things are   moving so fast beyond our ability to catch up 
what we need to do is study the change itself.   And that means we can look back at the previous 
things, and then look at the current thing,   and try to map how we got here, and from which we 
can look into what where we are heading towards.

So what does it mean to study the change itself? 
First, we need to identify the dominant driving forces behind the change. So here, dominant is an important word because typically, a change has many, many driving forces, and we only care about the dominant one because we're not trying to get really accurate, we just want to have the sense of directionality. Second, we need to understand the driving force really well, and then after that, we can predict the future trajectory by rolling out the driving force and so on. And you heard that right, I mentioned about predicting the future, this is a computer science class not like an astrology or something. But we do-- I think it's actually not that impossible to predict some future trajectory of a very narrow scientific domain. And that endeavor is really useful to do because let's say you do all these and then make your prediction accuracy to from 1% to 10%, and then

you make say 100 predictions, 10 of them will be 
correct, say one of them will be really, really   correct. Meaning it will have an outside impact 
that outweighs everything. And I think that is   kind of how many-- I've seen many general thing in 
life that you really have to be right a few times.  So why-- so if we think about why predicting 
the future is difficult or maybe even think   about the extreme case where we can all 
do the prediction with perfect accuracy,   almost perfect accuracy. So here, I'm going to 
do a very simple experiment of dropping this pen,   and follow this same three-step process. So we're 
going to identify the dominant driving force.  First of all, what are the driving forces 
acting on this pen? Gravity downwards,   and is that all? We also have say air friction, 
right? If I drop it, and that will cause what's

called a drag force acting upwards. And 
actually, depending on how I drop this, the orientation, the aerodynamic interaction 
will be so complicated that we don't currently have any analytical way of modeling that.
We can do it with the CFD, the computational fluid dynamics, but it will be non-trivial. So 
we can neglect that. This is heavy enough that gravity is probably the only dominant force, so 
we simplify the problem. Second, do we understand this dominant driving force, which is gravity? And 
we do because we have this Newtonian mechanics, which provides a reasonably good model.
And then with that, we can predict the future trajectory of this pen. And if we remember 
from this dynamics class, if we have this initial velocity is 0, I'm not going to put any 
velocity. And then let's say position is 0 here, and then 1/2 GT squared will give a precise 
trajectory of this pen as I drop this. So

if there is a driving force, single driving 
force that we really understand, it's actually   possible to predict what's going to happen.
So then, why do we really fear about predicting   the future in the most general sense? And I argue 
that, among many reasons, the number of driving   force, the sheer number of dominant driving forces 
acting on the general prediction is so complicated,   and their interaction creates a complexity 
that we cannot predict the most general sense.  So here is my cartoon way of thinking about the 
prediction of future. x-axis, we have a number   of dominant driving forces, y-axis we have 
a prediction difficulty. So on the left hand   side, we have a dropping of pen. It's a very 
simple case. It's a difficulty it's very small,   you just need to learn physics. And then, as 
you add more stuff, it just becomes impossible.  So how does this fit into the AI research? And 
you might think, OK, I see all the time things

are coming in, we are bombarded by new things, 
and some people will come up with the new agent, new modality, new MMLU score, or whatever. We just 
see so many things. It just I'm not even able to catch up with the latest thing. How can I even 
hope to predict the future of the AI research? But I argue that it's actually simpler 
because there is a dominant driving force that is governing a lot, if any, if not all 
of the AI research. And because of that, I would like to point out that it's actually 
closer to the left than to the right than we actually may perceive. So what is the driving 
force? Maybe before that, I would like to caveat that when I do this kind of talk, I would like 
to not focus too much on the technical stuff, which you can probably do better in your own 
time, but rather I want to share how I think.

And for that, I want to share how my opinion is, and so it will be very strongly opinionated. And by no means, I'm saying this is correct or not, I just wanted to share my perspective. So coming back to this driving force for AI, what is that dominant driving force? And here is a plot from Rich Sutton. And on the y-axis, we have the calculations plot if you pay $100, and how much computing power do you get? And it's in log scale. And then x-axis, we have a time of more than 100 years. So this is actually more than exponential. And I don't know any trend that is as strong and as long-lasting as this one. So whenever I see this kind of thing, I should say, OK, I should not compete with this. And better, I should try to leverage as much as possible. And so

what this means is, you get 10x more compute every 
five years if you spend the same amount of dollar.  And so in other words, you get the cost of 
compute is going down exponentially. And   this associated scaling is really dominating the 
AI research, and that is somewhat hard to take,   but that is, I think, really important to think 
about. So coming back to this AI research,   how is this exponentially cheaper compute 
drive the AI research? Let's think about   the job of the AI researchers. It is to teach 
machines how to think in a very general sense.  And one somewhat unfortunately common approach 
is, we think about how we teach machine how we   think we think. So meaning, we model how 
we think and then try to incorporate that   into some kind of mathematical model and teach 
that. And now, the question is, do we understand   how we think at the very low level? I don't 
think we do. I have no idea what's going on.

So it's fundamentally flawed in the sense that we try to model something that we have no idea about. And what happens if we go with this kind of approach is that it poses a structure that serves as a shortcut in the short term, and so you can maybe get a paper or something. But then it becomes a bottleneck when-- because we don't know how this will limit further scaling up. More fundamentally what this is doing is, we are limiting the degree of freedom we are giving to the machines, and that will backfire at some point. And this has been going on for decades. And bitter lesson is, I think the single most important piece of writing in AI. And it says, this is my wording by the way. Say past 70 years of entire AI research can be summarized into developing progressively more general method with weaker modeling assumptions or inductive biases, and add more data and compute, in other words scale up.

And that has been the recipe of entire AI research, not fancy things. And if you think about this, the models of 2000 is a lot more difficult than what we use now. And so it's much easier to get into AI nowadays from a technical perspective. So this is, I think, the really the key information. We have this compute cost is going down exponentially, and it's getting cheaper faster than we are becoming a better researcher. So don't compete with that and just try to leverage that as much as possible, and that is the driving force that I wanted to identify. And I'm not saying this is the only driving force, but this is the dominant driving force, so we can probably neglect the other ones. So here's a graphical version of that. x-axis we have a compute, y-axis we have a performance of some kind. Let's think about some general intelligence.

And let's look at two different methods. One, 
with more structure, more modeling assumptions,   fancier math, whatever. And then the other one 
is a less structure. What you see is typically,   you start with a better performance when you have 
a low compute regime, and then but it plateaus,   because of some kind of structure backfiring.
And then with the less structure because we give   a lot more freedom to the model, it doesn't work 
in the beginning. But then as we add more compute,   it starts working, and then it gets better, we 
call this more scalable method. So does that mean   we should just go with the least structure, 
most freedom to the model possible way from   the get go? And the answer is, obviously, no.
Let's think about even less structured case. This   red one here is, it will pick up a lot later and 
requires a lot more compute. So it really depends

on where we are. We cannot indefinitely wait 
for the most general case. And so let's think   about the case where our compute situation is 
at this dotted line. If we're here, we should   choose this last structure one as opposed to 
this even less structure one because the other   one doesn't really work and the other one works.
But crucially, we need to remember that we are   adding some structure because we don't have 
compute, so we need to remove that later. And   so the difference between these two methods is 
that additional inductive biases or structure we   impose, somewhat impose that typically don't get 
removed. So adding this. What that means is that,   at the given level of compute data algorithmic 
development and architecture that we have,   there's like an optimal inductive bias or 
structure that we can add to the problem   to make the progress. And that has been 
really how we have made so much progress.

But these are like shortcuts that hinder further 
scaling later on, so we have to remove them later on when we have more compute, better 
algorithm, or whatever. And as a community, we do adding structure very well, and because 
there's an incentive structure with like papers. You add a nice one, then you get a paper. But 
removing that doesn't really get you much, so that we don't really do that. And I 
think we should do a lot more of those. So maybe another implication of this bitter lesson 
is that, because of this, what is better in the long-term almost necessarily looks worse now. And 
this is quite unique to AI research because the AI research of the current paradigm is a learning-based 
method. Meaning that we are giving models freedom, the machines choose how they learn.
So because we need to give more freedom, it gives-- it's more chaotic at the beginning, so 
it doesn't work. But then when it started working,

we can put in more compute, and then it can be better. So it's really important to have this in mind. So to summarize, we have identified this dominant driving force behind the AI research, and that is this exponentially cheaper compute, and associated scaling up. Now that we have identified, if you remember back from my initial slides, we-- the next step is to understand this driving force better. And so that we're going to spend most of the time doing that. And for that, we need to go back to some history of transformer because this is a transformers class and analyze key structures and decisions that were made by the researchers at the time, and why they did that. Whether that was an optimal structure that could have been added at the time, and why they might be irrelevant now, and should we remove that? And we'll go through some of the practices of this. And hopefully,

this will give you some flavor of what 
scaling research looks like. So now, we'll go into a little bit of the technical 
stuff. Tech transformer architecture, there are some variants. I'll talk about 
three of them. First, is the encoder-decoder, which is the original transformer, 
which has a little bit more structure. Second one is the encoder-only, which is 
popularized by Bert. And then third one is decoder-only, which you can think of it as a 
current like GPT-3 or other language models. This has a lot less structure than the encoder-decoder. 
So these are the three types we'll go into detail. Second, the encoder-only is actually not that 
useful in the most general sense. It still has some place, but we will just briefly go over 
that, and then spend most of the time comparing 1 and 3. So one has more structure, what's the 
implication of that and so on? So first of all,

let's think about what a transformer is, just at a very high level. Or first principles, what is a transformer? It is a sequence model. And a sequence model has an input of a sequence. So a sequence can be a sequence of elements, such as words, or images, or whatever. It's a very general concept. In this particular example, I show you with the words, a sentence is a sequence of words. And then the first step is to tokenize it because we have to represent these words in computers, which requires some kind of encoding scheme, so we just do it with a fixed number of integers, and then we have now a sequence of integers. And then the dominant paradigm nowadays is to represent each sequence element as a vector, a dense vector, because we know how to multiply them well. And then we have a sequence of vectors. And finally, this sequence model will do the following. We just want to model the interaction

between sequence elements. And we do that by let 
them take the dot product of each other. And if the dot product is high, we can say semantically 
they are more related than the dot products that is low, and that's kind of the sequence model.
And the transformer is a particular type of sequence model that uses what's called attention 
to model this interaction. So let's get into the details of this encoder-decoder, which was 
the original transformer. It's quite many, many pieces. So let's go into a little bit a piece 
at a time. So starting with the encoder. So here I'm going to show you an example of machine 
translation, which used to be a very cool thing. And so you are-- you have an English sentence that 
is good, and then we're going to translate it into German. So first thing is to encode this into a 
dense vector. So here I'm representing it with

this is a vector of size three or something. 
And then we have to let them take the dot product. So this lines represent which element 
can talk to which element, other elements. And here because it's an input, we take what is 
called a bidirectional attention, so any token can talk to any other token. And then we have 
this MLP or feedforward layer which is per token, it doesn't have any interaction, we just do 
some multiplication just because we can do it. And then that's one layer, and we repeat that 
n times, and that's just the transformer encoder. And at the end, what you get is the sequence of 
vectors each representing the sequence element in this case a word. So that's the output of this 
encoder. Now, let's look at the decoder, which is similarly shaped stack of layers. So here we put 
in as an input what the answer should be. So here,

BOS is the beginning of a sequence, and then das ist gut. I don't know how to pronounce it, but that's the German translation of that is good. And so we kind of go through the similar process. Here we have a causal self-attention, meaning that the tokens of time step t can only attend to t and before because when we start generating it, we don't have the future tokens, so we cannot. When we train it, we should limit that, and that way, this is done by masking, but this is different from the encoder. So after this, you can get, after again, n layers, you get this sequence output, and you have this, the output of sequence to sequence to sequence mapping. This is a general encoder-decoder architecture. And when you get this end of sequence, you stop generating it. So this is the overall picture. Now

I'll point out some important attention patterns.
So we are translating into German what is into input to the encoder. So there has to be some connection between the decoder and the encoder that is done by this cross-attention mechanism shown in this red, which is just that each vector representation of each sequence in the output decoder should attend to some of them in the encoder, and that is done. In particular, the design feature which is interesting is that all the layers in the decoder attend to the final layer output of the encoder, and we'll come back to the implication of this design. So yeah, that's that. And now, move on to the second type of architecture, which is encoder-only. We'll spend a little bit of time here. So again, we have to-- we have the same input, and we go through a similar structure. And then in this case, the final output is a single vector. Regardless of the length

of the sequence, we just get a single vector. 
And that is that represent the input sequence, that's a dense vector representation. And then 
let's say we do some kind of a sentiment analysis, we run through a task-specific linear 
layer to map it to classification label positive or negative probabilities here.
And that's required for all these task-specific cases. And this is kind of popularized 
by Bert. And what this means is that, here at the time, 2018, when Bert came out 
we had the benchmark called glue, which was a language understanding task. You have a sequence 
in classification labels out for most cases. This was how the field really advanced at the time.
So when we care about such tasks, then there's an incentive to think about simplifying the problem, 
adding the structure to the problem so that we can make a progress. So this, the additional structure 
that was put into this particular architecture is

that we're going to give up on the generation.
If we do that, it becomes a lot simpler problem. Instead of sequence to sequence, we're talking
about sequence to classification labels, and that's just so much easier.
And so at some point, 2018, 2019, a lot of the papers are just research was like,
we sometimes call it Bert engineers, a little bit change of something get like 0.5% better on
glue, and you get a paper, and things like that. It was very chaotic error, and-- but if we look at
from this perspective, we are putting the sequence structure of not generating the sequence that puts
a lot of performance win, but in the long term, it's not really useful. So we're not going to look
at this encoder-only architecture going forward. Third architecture, decoder-only. This one is
my favorite personally. And it looks kind of

daunting because of this attention pattern, 
but it actually is very simple. So here, we only have a single stack, and it can actually 
generate stuff. And so there's misconception that some people think this decoder-only architecture 
is used for language modeling next to prediction, so it cannot be used for supervised 
learning. But here, we can actually do it. The trick is to have this input that is good, 
concatenate it with the target. And if you do that, then it just becomes simple to sequence in 
sequence out. So what we do is, the self-attention mechanism here is actually handling both the 
cross attention between target and the input, and self-attention sequence learning within 
each. So that's the causal attention. And then as I mentioned, the output is a 
sequence, and then the key design features are self-attention is serving both roles, and 
we are in some sense sharing the parameters

between input and target. So same set of 
parameters are applied to both input and the target sequences. So this is the decoder 
only. Now, we will go into the comparison. So I think there are many-- like they look 
very different, at least on the schematics. So how different are they actually? And I argue 
that they're actually quite similar. And so to illustrate that we're going to transform 
starting from this encoder-decoder which has more structures built in, and then 
into the decoder-only architecture and see what are some of the differences. And then 
interpret those differences, those additional structures, are they relevant nowadays now that 
we have more compute better algorithm and so on? So let's have this table, four differences we'll 
see each of them. And then as we go through, we will populate this table. So let's first 
look at this additional cross-attention. What

that means is that, this on the left is an 
encoder-decoder, which has this additional red block, the cross-attention compared 
to the simpler one that doesn't have that. So we want to make it, make the left closer to the 
right, so that means we need to either get rid of it or something. And attention mechanism 
has kind of the four projection matrices, and so self-attention and cross attention 
actually have the same number of parameters, same shape, so we can just share them. So 
that's the first step, share both of these, and then it becomes mostly the same mechanism.
And then so that's the first difference separate cross-attention or self-attention serving both 
roles. Second difference is the parameter sharing. So what that means is that, between the input and 
the target, encoder-decoder architecture uses a separate parameters. And decoder-only only has 
a single stack so it uses the shared parameter.

So if we want to make the left close to right, we want to share the encoder parameters, so let's do that. I just color this, so now, they share the parameters. Third difference is the target to input attention pattern. So we need to connect the target to the input. And how does that-- how is it done? In the encoder-decoder case, we had this cross attention. And then in the decoder-only, it's a per the self-attention doing everything. What the difference is that we have this every layer of the decoder attending to the final layer output of the encoder, whereas if you think about this decoder is actually per layer, within layer when we are decoding the say word thus, we are looking at the same layer representation of the encoder, and that's within layer, and I think this is the design feature. So if we want to make this close to that, we have

to bring back this attention to each layer.
So now, layer 1 will be attending to layer 1 of this. And that's then finally, the 
last difference is the input attention. I mentioned about this bidirectional attention. 
And because we have this decoder only typically, with the unidirectional attention, we need to make 
them matching, so that's the we can just get rid of it. I just got rid of the some of the arrows.
So then at this point, these two architectures are almost identical. A little bit 
of difference in the cross attention, but same number of parameters. And if you have 
in deep learning, if you just train these, these two architecture in the same task, same data, I 
think, you will get pretty much within the noise, probably closer than if you train the same 
thing twice. So I would say they are identical. And so these are the main differences. Now, 
we'll look at what are the additional structures.

what they mean, it means. So yeah. That's the populated table now, and then so we can say that encoder-decoder compared to the decoder-only architecture has these additional structures inductive biases built in, so let's go into each of them. The first one is what encoder-decoder tries added as a structure is that, input and the target sequences are sufficiently different that we-- it will be useful to use a separate parameters, that's the assumption. And so why is that a useful? When can that assumption be useful? And one example is machine translation. Back when the transformer was introduced in 2017, translation was really popular task, and it was difficult, considered difficult. And because it's a sequence-to-sequence and you can actually have a blue score which is heuristic based method that can give you a single number and then people can optimize that. So what

that in that task, we have this input and 
target in completely different languages. So,   if the goal is to learn translation only, 
then it kind of makes sense to have, OK,   this parameter in the encoder will take care of 
the English, and this parameter in the decoder   will take care of the German, that seems natural.
And what about now? Modern language models is   about learning knowledge. And it's not just 
about translation or not even about language,   languages comes up as a byproduct of doing 
this next token prediction and translation as   well. So, does it make sense to have a separate 
parameter for this kind of situation now? We   have some knowledge in German, some knowledge in 
English, and we-- if anything we want to combine   them. And if we represent them in a separate 
parameters, I don't think that's natural.  So, I would say with this much more general, 
larger models that can do a lot of things,

this assumption seems very unnatural to me. Second example is a little bit more modern. Two years ago when I was at Google and with Jason, we did this instruction fine-tuning work. And what this is, is you take the pre-trained model, and then just fine-tune on an academic data set, and so that it can understand the natural language instruction. So the detail doesn't matter, but here let's think about the performance gain by doing this fine-tuning on two different architectures we tried. So first five is the Flan-T5, which is T5 base, which is the encoder-decoder architecture. Last one, the latter five, decoder-only architecture based on PaLM. So we spent 99% of the time on PaLM, optimizing a lot of these, and then at the end we just spent like three days on T5. But the performance gain was a lot higher on this. And I was really confused about this.

and in a very good way. And after the paper 
was published, I wanted to take a little bit   deeper into why this might be the case. So my 
hypothesis is that, it's about the length. So   academic data sets we use, we use like 1,832 
tasks. And here they have this very distinctive   characteristic where we have a long input, 
long in order to make the task more difficult.  But then we cannot make the target long 
because if we do, there's no way to create it,   so there's fundamental challenge of that. So 
what happens is, you have a long text of input   and then short text of the target. And so this 
is kind of the length distribution of what it   went into the Flan fine tuning. So then you 
see this, you have a very different sequence   going into the encoder as an input, and the very 
different type of sequence going into the target.

So now, this encoder-decoder architecture has an assumption that they will be very different. That structure really shines because of this. It was a kind of an accident, but that was, I think, why this really architecture was just suitable for fine-tuning with the academic data sets. What about now? Do we care about this kind of assumption? And if you think about the general use cases of language models nowadays, if anything the more interesting cases involve longer generation, longer target. Just because we cannot grade them, doesn't mean that we are not interested in them. Actually, if anything we are more interested in that. So now we have this longer target situation, so this separate sequence length parameter doesn't seem to make much sense. And moreover, we think about this chat application like ChatGPT, we do multi-turn conversation. And then so what is a target of this turn

becomes the input of the next turn. And then, my question is, does that make sense to even think about a different parameters if next turn is going to be the same thing? So that was the first inductive bias we just mentioned. And then, the second structure is that target element can only attend to the fully encoded ones, the final output of the encoder. Let's look at this additional structure, what that means. So as I mentioned, we have this very top layer attending to it. And so in deep neural nets, typically, we see that the bottom layers and the top layers encode information at a very different level. Meaning that, for example, in computer vision, lower layer-- the bottom layers encode something like edges, top layers, higher levels combining the features, something like cat face. And so we call this deep learning a hierarchical representation learning method. And so now the question is, if decoder layer one attends to

encoder final layer, which probably has a very different level of information, is that some kind of an information bottleneck which actually motivated the original attention mechanism? And in practice, I would say in my experience, doesn't really make any difference, and that's because my experience was limited to say 25, 24 layers of encoder of T5. So layer one attended to 24, probably fine. But what if we have 10x or 1,000x more layers, would that be problematic? I'm not really comfortable with that. So I think this is also unnecessary design that maybe we need to revisit. Final structure we're going to talk about is the, when we do this, there's a bi-directional thing in the encoder-decoder. Let's think about that. So yeah. Bi-directional input attention, is that really necessary? So when we had this Bert

B in Bert stands for bi-directional. In 2018 when we were solving like question answering squad, actually it was a very difficult task. So if you have any additional trick, it can make a huge difference. Bidirectionality was really useful, I think, maybe boosting up the squad score by like 20, so it was really a huge thing. But at scale, I don't think this matters that much. This is my highly anecdotal experience, so we did in Flan-2, we tried both bi-directional and uni-directional fine-tuning, didn't really make much difference. So but I want to point out this bidirectionality actually brings in an engineering challenge for modern multi-turn chat application. So at every turn, the new input has to be encoded again, and for uni-directional attention it's much, much better, so here is what I mean by that. So let's think about this more modern

conversation between user and assistant,
how are you? Bad, and why? And so here, if we think about the bi-directional case, we will-- and when we generate bad, we need to encode this input with the bi-directional thing, which is fine. And then after the bad is generated, when we're trying to generate why, we'll need to encode how, again, because how can attend to bad, so we need to do everything from scratch again. In contrast, if we do uni-directional one, we can do much, much better because now when we are trying to generate why, we don't have to redo how because we cannot attend to the future tokens, so we don't have to do anything. So if you see the difference, this part can be cached and then this part is the only thing that can has to be encoded again. So this kind of makes a big difference when we think about multiple turns going on.

So I would say, bi-directional attention, that 
well, in 2018, which is mostly sold by scale,   and now because of this engineering challenge, 
we don't really need that. So to conclude,   we have looked into this driving force, 
dominant driving force covering this research,   and that was this exponentially cheaper 
compute and associated scaling effort.  And so to understand this driving force, we 
analyzed some of the additional structures   added to the encoder-decoder compared to the 
decoder only, and then thought about how--   what that means as a from the perspective of 
scaling. And I wanted to just conclude with   this remark. So we have looked at these kind 
of analysis, which are all-- one can say, this   is just historical artifacts and doesn't matter.
But if you do many of these, now you look at the   current events. You can hopefully think about 
those in a more unified manner and then see,

OK, what assumptions in my problem that I need to revisit? And are they relevant? And if not, why? And you have an answer to it. Can we do it with a more general thing and scale up? And so I hope you can go back and really think about these problems. And together we can really shape the future of AI in a really nice way, so that's it. Thanks.

Stanford CS25: V4 I Hyung Won Chung of OpenAI

You can gain access to a world of education through Stanford Online, the Stanford School of Engineering’s portal for academic and professional education offered by schools and units throughout Stanford University.  https://online.stanford.edu/

Our robust catalog of degree programs, credit-bearing education, professional certificate programs, and free and open content is developed by Stanford faculty, enabling you to expand your knowledge, advance your career, and enhance your life.

Stanford Online is operated and managed by the Stanford Center for Professional Development (SCPD), the global and online education unit within Stanford Engineering. SCPD works closely with Engineering departments, programs, and centers to design and deliver engaging, high quality online, in-person, and blended learning experiences to both matriculated students and a worldwide audience of learners. SCPD collaborates with many Stanford schools & centers to expand university-wide offerings available online.


Stanford Online

The History of Transformers and Predicting the Future of AI

Hyung Won, a research scientist at OpenAI, discusses the importance of understanding the history of transformers and their impact on the future of AI.

He emphasizes the need to study the driving forces behind the rapid changes in the field and the possibility of predicting future trajectories.

Hyung Won uses the example of predicting the future impact of narrow scientific domains, highlighting the significance of being right a few times despite the difficulty of perfect prediction.

The Impact of Dominant Driving Forces on Predicting the Future in AI Research

The complexity of aerodynamic interactions is challenging to model analytically, leading to the simplification of the dominant force as gravity in predicting the trajectory of a falling object.

The multitude of dominant driving forces and their interactions create a level of complexity that makes predicting the general future difficult.

The exponentially decreasing cost of computing power is the dominant driving force in AI research, significantly impacting its direction and progression.

The Impact of Structured Modeling on AI Research

AI research has been focused on developing more general methods with weaker modeling assumptions, relying on increased data and computing power for progress.

The decreasing cost of computing has driven AI research, allowing for the removal of imposed structures and biases as technology advances.

The current incentive structure in the AI research community favors adding structure to problems for short-term gain, but there is a need to focus on removing these shortcuts for long-term progress.

The Role of Exponentially Cheaper Compute in AI Research

Exponentially cheaper compute is the dominant driving force behind AI research, leading to a focus on understanding this force better and its historical impact on transformer architecture.

The transformer, as a sequence model, uses attention to model interactions between sequence elements, with the original transformer consisting of an encoder-decoder structure and two other variants: encoder-only and decoder-only.

The encoder-decoder variant, used in machine translation, involves encoding the input sequence into dense vectors, allowing for bidirectional attention and multiple layers of processing, while the decoder similarly consists of a stack of layers to generate the output sequence.

The Evolution of Sequence-to-Sequence Architectures

Sequence-to-sequence architecture involves a causal self-attention process during training to limit token attention to the current and previous time steps.

The architecture also includes a cross-attention mechanism between the encoder and decoder, with the decoder having layers that attend to the final layer output of the encoder.

Additionally, there are encoder-only and decoder-only architectures, with the decoder-only architecture being capable of supervised learning by concatenating the input with the target and using a self-attention mechanism to handle both the cross-attention and sequence learning within each.

Comparing Encoder-Decoder and Decoder-Only Architectures

The decoder-only architecture is being compared to the encoder-decoder architecture to illustrate their similarities and differences.

The differences between the two architectures include the presence of cross-attention, parameter sharing, and the pattern of target-to-input attention.

The additional structures in the encoder-decoder architecture provide inductive biases, assuming the usefulness of separate parameters for input and target sequences, which is particularly relevant in tasks like machine translation.

The Impact of Modern Language Models on Translation and Knowledge Learning

Language models are now focused on knowledge learning in addition to translation, raising the question of whether separate parameters for different languages are natural in these more general models.

Fine-tuning pre-trained models on academic datasets has shown that the performance gain is higher when the target sequence is shorter than the input sequence, challenging the assumption of the encoder-decoder architecture.

The use of separate parameters for different sequence lengths in language models is being questioned, especially in cases involving longer generation and multi-turn conversation, challenging the current inductive bias.

Revisiting the Design of the Encoder-Decoder in AI

The final layer of the encoder may not have a significant impact, even with a large number of layers, suggesting it may be an unnecessary design.

The need for bi-directional attention in the encoder-decoder is questioned, as it poses an engineering challenge for modern multi-turn chat applications, with uni-directional attention being more efficient.

The additional structures added to the encoder-decoder compared to the decoder only are analyzed in the context of scaling, with a call to revisit assumptions and shape the future of AI.