now we have Hyung Won give a talk. So he's
currently a research scientist on the OpenAI
ChatGPT team. He has worked on various aspects of
large language models. Things like pre-training, instruction fine-tuning, reinforcement learning
with human feedback, reasoning, and so forth.

And some of his notable works include the scaling
FLAN papers such as FLAN-T5, as well as FLAN-PALM, and T5X, the training framework used to train
the PALM language model.

And before OpenAI,
he was at Google Brain, and he received his
PhD from MIT. So give a hand for Hyung Won.

Do we switch the slides?
All right. My name
is Hyung Won, and I'm really happy to be here
today. And this week I was thinking about-- By the way, is mic working fine?

Yeah. So this
week I thought about-- okay, I'm giving a lecture on transformers at Stanford.

What should I
talk about? And I thought, okay, some of you in
this room and in Zoom will actually go shape the
future of AI, so maybe I should talk about that.

It's a really important goal and ambitious, and
we really have to get it right. So that could
be a good topic to think about. And when
we talk about something into the future, the best place to get advice is to
look into the history.

And in particular,
we'll look at the early history of transformer,
and try to learn many lessons from there.

And the goal will be to develop a unified
perspective in which we can look into many
seemingly disjoint events. And from
that we can probably hope to project into the future what might be coming.

And
so that will be the goal of this lecture,
and we'll look at some of the architectures
of the transformers, so let's get started.

Everyone I see is saying, AI is so advancing so
fast that it's so hard to keep up. And it doesn't matter if you have years of experience, there
are so many things are coming out every week
that it's just hard to keep up.

And I do see
many people spend a lot of time and energy catching up with the latest developments, the 
cutting edge and the newest thing, and then not   enough attention goes into old things because 
they become deprecated and no longer relevant.

But I think it's important, actually, to look into 
that because we really need to-- when things are   moving so fast beyond our ability to catch up, 
what we need to do is study the change itself.

And that means we can look back at the previous 
things, and then look at the current thing,   and try to map how we got here, and from which we 
can look into where we are heading towards.

So what does it mean to study the change itself? 
First, we need to identify the dominant driving   forces behind the change. So here, dominant 
is an important word because typically,   a change has many, many driving forces, and we 
only care about the dominant one because we're   not trying to get really accurate; we just 
want to have a sense of directionality.

Second, we need to understand the driving force 
really well, and then after that we can predict   the future trajectory by rolling out the driving 
force and so on. And you heard that right,   I mentioned about predicting the future.

This is 
a computer science class, not like an astrology or   something. But we do-- I think it's actually 
not that impossible to predict some future   trajectory of a very narrow scientific domain.

And that endeavor is really useful to do because   let's say you do all these and then make your 
prediction accuracy from 1% to 10%, and then   you make, say, 100 predictions, 10 of them will be 
correct, say one of them will be really, really   correct.

Meaning it will have an outsized impact 
that outweighs everything. And I think that is   kind of how many-- I've seen many general things in 
life that you really have to be right a few times.

So why-- so if we think about why predicting 
the future is difficult or maybe even think   about the extreme case where we can all 
do the prediction with perfect accuracy,   almost perfect accuracy.

So here, I'm going to 
do a very simple experiment of dropping this pen,   and follow this same three-step process. So we're 
going to identify the dominant driving force.

First of all, what are the driving forces 
acting on this pen? Gravity downwards,   and is that all? We also have, say, air friction, 
right? If I drop it, and that will cause what's   called a drag force acting upwards.

And 
actually, depending on how I drop this,   the orientation, the aerodynamic interaction 
will be so complicated that we don't currently   have any analytical way of modeling that.

We can do it with CFD, computational   fluid dynamics, but it will be non-trivial. So 
we can neglect that. This is heavy enough that   gravity is probably the only dominant force, so 
we simplify the problem.

Second, do we understand   this dominant driving force, which is gravity? And 
we do, because we have this Newtonian mechanics,   which provides a reasonably good model.

And then with that, we can predict the future   trajectory of this pen. And if we remember 
from this dynamics class, if we have this   initial velocity is 0, I'm not going to put any 
velocity.

And then let's say position is 0 here,   and then 1/2 GT squared will give a precise 
trajectory of this pen as I drop this.

So   if there is a driving force, a single driving 
force that we really understand, it's actually   possible to predict what's going to happen.

So then, why do we really fear about predicting   the future in the most general sense? And I argue 
that, among many reasons, the number of driving   forces, the sheer number of dominant driving forces 
acting on the general prediction is so complicated   and their interaction creates a complexity 
that we cannot predict in the most general sense.

So here is my cartoon way of thinking about the 
prediction of the future. X-axis, we have a number   of dominant driving forces; Y-axis, we have 
a prediction difficulty. So on the left-hand   side, we have a dropping of a pen.

It's a very 
simple case. Its difficulty is very small,   you just need to learn physics.

And then, as 
you add more stuff, it just becomes impossible.   So how does this fit into the AI research? And 
you might think, okay, I see all the time things   are coming in, we are bombarded by new things, and some people will come up with the new agent, new modality, new MMLU score or whatever.

We just see so many things, I'm not even able to catch up with the latest thing.

How can I even hope to predict the future of AI research? But I argue that it's actually simpler because there is a dominant driving force that is governing a lot, if not all, of the AI research.

And because of that, I would like to point out that it's actually closer to the left than to the right than we actually may perceive.

So what is the driving force? Maybe before that, I would like to caveat that when I do this kind of talk, I would like to not focus too much on the technical stuff which you can probably do better in your own time, but rather I want to share how I think.

And for that, I want to share how my opinion is, and so it will be very strongly opinionated.

And by no means I'm saying this is correct or not, I just wanted to share my perspective. So coming back to this driving force for AI, what is that dominant driving force? And here is a plot from Rich Sutton.

And on the y-axis, we have the calculations plot if you pay $100, and how much computing power do you get?

And it's in log scale. And then x-axis, we have a time of more than 100 years.

So this is actually more than exponential. And I don't know any trend that is as strong and as long-lasting as this one. So whenever I see this kind of thing, I should say, okay, I should not compete with this.

And better, I should try to leverage as much as possible. And so what this means is, you get 10x more compute every five years if you spend the same amount of dollar.

And so, in other words, you get the cost of compute is going down exponentially. And this associated scaling is really dominating the AI research, and that is somewhat hard to take, but that is, I think, really important to think about.

So, coming back to this AI research, how does this exponentially cheaper compute drive AI research? Let's think about the job of the AI researchers.

It is to teach machines how to think in a very general sense. And one somewhat unfortunately common approach is, we think about how we teach machines how we think we think.

So, meaning, we model how we think and then try to incorporate that into some kind of mathematical model and teach that. And now, the question is, do we understand how we think at the very low level?

I don't think we do. I have no idea what's going on.

So, it's fundamentally flawed in the sense that we try to model something that we have no idea about. And what happens if we go with this kind of approach is that it poses a structure that serves as a shortcut in the short term, and so you can maybe get a paper or something.

But then it becomes a bottleneck because we don't know how this will limit further scaling up.

More fundamentally, what this is doing is, we are limiting the degree of freedom we are giving to the machines, and that will backfire at some point.

And this has been going on for decades. And "Bitter Lesson" is, I think, the single most important piece of writing in AI.

And it says, this is my wording by the way, "Past 70 years of entire AI research can be summarized into developing progressively more general methods with weaker modeling assumptions or inductive biases, and add more data and compute," in other words, scale up.

And that has been the recipe of entire AI research, not fancy things. And if you think about this, the models of 2000 are a lot more difficult than what we use now. And so it's much easier to get into AI nowadays from a technical 
perspective.

So this is, I think, the really   the key information. We have this compute cost is 
going down exponentially, and it's getting cheaper faster than we are becoming a better researcher.

So don't compete with that and just try to   leverage that as much as possible, and that is 
the driving force that I wanted to identify.

And I'm not saying this is the only driving 
force, but this is the dominant driving force,   so we can probably neglect the other ones. So 
here's a graphical version of that. X-axis we have compute, Y-axis we have performance of some 
kind.

Let's think about some general intelligence.  And let's look at two different methods. One, 
with more structure, more modeling assumptions, fancier math, whatever.

And then the other one 
is less structure. What you see is typically, you start with better performance when you have 
a low compute regime, but it plateaus   because of some kind of structure backfiring.

And then with less structure, because we give a lot more freedom to the model, it doesn't work 
in the beginning.

But then as we add more compute, it starts working, and then it gets better.

We 
call this a more scalable method. So does that mean   we should just go with the least structure, most 
freedom to the model possible way from the get-go?

And the answer is, obviously, no. 
Let's think about an even less structured case. This   red one here, it will pick up a lot later and 
requires a lot more compute. So it really depends on where we are.

We cannot indefinitely wait 
for the most general case. And so let's think   about the case where our compute situation is 
at this dotted line. If we're here, we should choose this less structured one as opposed to 
this even less structured one because the other   one doesn't really work and the other one works.

But crucially, we need to remember that we are adding some structure because we don't have 
compute, so we need to remove that later.

And   so the difference between these two methods is 
that additional inductive biases or structure we impose, somewhat impose, that typically don't get removed.

So adding this. What that means is that, at the given level of compute, data, algorithmic development, and architecture that we have, there's like an optimal inductive bias or structure that we can add to the problem to make the progress.

And that has been really how we have made so much progress.

But these are like shortcuts that hinder further scaling later on, so we have to remove them later on when we have more compute, better algorithm, or whatever. And as a community, we do adding structure very well and because there's an incentive structure with like papers.

You add a nice one, then you get a paper. But removing that doesn't really get you much, so we don't really do that.

And I think we should do a lot more of those. So maybe another implication of this bitter lesson is that, because of this, what is better in the long-term almost necessarily looks worse now.

And this is quite unique to AI research because the AI research of current paradigm is learning-based method. Meaning that we are giving models freedom, the machines choose how they learn.

So because we need to give more freedom, it gives-- it's a more chaotic at the beginning, so it doesn't work. But then when it started working, we can put in more compute and then it can be better, so it's really important to have this in mind.

So to summarize, we have identified this dominant driving force behind the AI research, and that is this exponentially cheaper compute, and associated scaling up.

Now that we have identified, if you remember back from my initial slides, we-- the next step is to understand this driving force better.

And so that we're going to spend most of the time doing that. And for that, we need to go back to some history of Transformer because this is a Transformers class and analyze key structures and decisions that were made by the researchers at the time, and why they did that.

Whether that was an optimal structure that could have been added at the time, and why they might be irrelevant now, and should we remove that?

And we'll go through some of the practices of this. And hopefully, this will give you some flavor of what 
scaling research looks like.

So now,   we'll go into a little bit of the technical 
stuff, transformer architecture.

There are some variants. I'll talk about 
three of them. First, is the encoder-decoder,   which is the original transformer, 
which has a little bit more structure.

Second one is the encoder-only, which is 
popularized by BERT. And then third one   is decoder-only, which you can think of it as a 
current like GPT-3 or other language models. This has a lot less structure than the encoder-decoder.

So these are the three types we'll go into detail.

Second, the encoder-only is actually not that 
useful in the most general sense. It still has   some place, but we will just briefly go over 
that, and then spend most of the time comparing 1 and 3.

So one has more structure, what's the 
implication of that and so on? So first of all, let's think about what a transformer is, just 
at a very high level.

Or first principles,   what is a transformer? It's a sequence model.
And a sequence model has an input of a sequence.

So a sequence can be-- a sequence 
of elements can be words, or images,   or whatever. It's a very general concept. In this 
particular example I show you with the words, a sentence is a sequence of words.

And then 
the first step is to tokenize it because   we have to represent this word in computers, 
which requires some kind of encoding scheme, so we just do it with a fixed number of integers, 
and then we have now a sequence of integers.

And then the dominant paradigm nowadays is to 
represent each sequence element as a vector, a dense vector, because we know how to multiply them 
well.

And then so we have a sequence of vectors.   And finally, this sequence model will do the 
following: we just want to model the interaction between sequence elements.

And we do that by let 
them take the dot product of each other. And if   the dot product is high, we can say semantically 
they are more related than the dot products that is low, and that's kind of the sequence model.

And the transformer is a particular type of sequence model that uses what's called attention to model this interaction. So let's get into the details of this encoder-decoder, which was the original transformer.

It's quite many, many pieces. So let's go into a little bit a piece at a time. So starting with the encoder. So here I'm going to show you an example of machine translation, which used to be a very cool thing.

And so you are-- you have an English sentence that is good, and then we're going to translate it into German.

So first thing is to encode this into a dense vector. So here I'm representing it with this a vector of size three or something. And then we have to let them take the dot product.

So these lines represent which element can talk to which element, other elements. And here because it's an input, we take what is called a bidirectional attention, so any token can talk to any other token.

And then we have this MLP or feedforward layer which is per token, it doesn't have any interaction, we just do some multiplication just because we can do it.

And then that's one layer, and we repeat that n times, and that's just the transformer encoder.

And at the end, what you get is the sequence of vectors each representing the sequence element, in this case a word. So that's the output of this encoder. Now, let's look at the decoder, which is similarly shaped stack of layers.

So here we put in as an input what the answer should be. So here, das ist gut -- I don't know how to pronounce it, but that's the German translation of, that is good.

And so we kind of go through the similar process. Here we have a causal self-attention, meaning that the tokens of time step t can only attend to t and before because when we start generating it, we don't have the future tokens, so we cannot-- when we train it, we should limit that, and that way, this is done by like masking, but this is different from the encoder.

So after this, you can get-- after again, n layers, you get this sequence output, and you have this-- the output of sequence, to sequence, to sequence mapping, this is a general encoder-decoder architecture.

Stanford CS25: V4 I Hyung Won Chung of OpenAI

You can gain access to a world of education through Stanford Online, the Stanford School of Engineering’s portal for academic and professional education offered by schools and units throughout Stanford University.  https://online.stanford.edu/

Our robust catalog of degree programs, credit-bearing education, professional certificate programs, and free and open content is developed by Stanford faculty, enabling you to expand your knowledge, advance your career, and enhance your life.

Stanford Online is operated and managed by the Stanford Center for Professional Development (SCPD), the global and online education unit within Stanford Engineering. SCPD works closely with Engineering departments, programs, and centers to design and deliver engaging, high quality online, in-person, and blended learning experiences to both matriculated students and a worldwide audience of learners. SCPD collaborates with many Stanford schools & centers to expand university-wide offerings available online.


Stanford CS25: V4 I Hyung Won Chung of OpenAI

Scripts