yeah, I will talk about large language models in 2023. Why do we need that suffix in 2023? What we call large is misleading. In the large models of today, will be small models only in a few years. Along with the change in the perception of scale, many insights, observations, and conclusions we make along the way with the current large damage models will be outdated and, in some cases, invalidated. Fortunately, insights based on first principles or that are fundamental tend to stay relatively longer than the advanced ideas that look so fancy. In this talk, I would like to share such fundamental ideas that I have observed working in the field for the past four years, and mostly by observing some of the brightest minds that I'm fortunate to have interacted with. My hope is that whatever I talk about today is fundamental enough that it stays relevant at least for a few years. Let's get started. There's one unique aspect of large language models, and that is some abilities only emerge at a certain scale. And this makes it really important to, you know, just

have a different perspective in viewing all this field. So here's what I mean. Details don't matter here. The x-axis is some scale. You can think of it as a model parameter count, training data, training compute, whatever. You just think about some scale. The y-axis is also not that important. Some performance metrics of the task that you care about. And we have seen over and over this kind of pattern where small models just don't have any chance of solving it. Zero percent or random guessing. And at some scale, suddenly the model solves this test, and sometimes very well. And this kind of phenomenon we call it an emergence, a phase change, like sudden emergence of this capability. And this is very unique for these large English models. And I think this brings many important perspectives change that for researchers. First one is this perspective of yet. So let's say we have this new idea that I want to test out, and let's say some kind of a new reasoning technique. And I try out the experiment and say it doesn't work. Most of the time it doesn't work.

then instead of saying, this doesn't work forever, I want to say, this doesn't work yet, and this is really a fundamental change in the viewpoint. I'm saying this might not work for the current generation of the models, but in three years, five years, that might just work. And so we should really not conclude many things in a permanent sense. And if we think about why this perspective is not so obvious, we're so used to operating in an environment where the underlying axioms don't really change. So let's think about this as an experiment, an example where, let's say, you run some physics experiments or you study thermodynamics, which I did. And then you run some experiments, and then if you have some new idea and this experiment doesn't work out, then you know that if you run it again three years later, this will not work for that matter 30 years later. This doesn't work. The underlying physics, second law of thermodynamics, doesn't change. And we know this, and we're very used to this.

kind of situation and a lot of The Human Experience is like this. So, what for the language models? What is the equivalent concept of such axioms underlying the field? And that's more like the most capable model at the time. And if anything, this changes. So, hey, GPT-4 came out, and then researchers spilled on top of it a lot of research experiments and intuitions built on top of it. And then later on, some better model comes out, then all many of these ideas are outdated, in many cases contradictory. And it's really important to think about this, and this calls for the need to constantly unlearn the intuitions based on such invalidated ideas. And this is somewhat a new concept that I rarely see people practice. And I think this is, uh, we have seen this very competitive field of AI. There are many cases where people with one or two years of experience come in and have very impactful ideas and work done. And I think some of that has to do with this newcomers just come in with the first

perspective then try out the ideas that experienced people have tried in the past that did not work at UH with the model at the time. Now, it suddenly works. This, I think, is a very neutralizing force, and for people experienced in the field, I think this is something to think about. Another perspective I would like to talk about is we should go ahead of some of the scaling curve. So, here's what I mean. When I work on this day-to-day research, here's kind of the mental picture I have. Document all the experiments that fail, and because of this insufficient intelligence. So, this is a very vague term, but maybe it doesn't just have enough reasoning capability to solve this hard math problem or coding problem. Then, I kind of, you know, just document it somewhere and do not declare failure yet and make it easy to rerun in the future. And as soon as the new better model comes out, we run them and then learn what works and what doesn't. And this, with this, I update my intuition, unlearn necessary stuff, and

really build intuition around emerging abilities associated with the scale. I think this will become more and more important. So, here's a visual way of thinking about this. Let's again very simplify. Not all the tasks work like this, but the scale is just whatever. It's just some scale. And then I just put GPT-3 and GPT-4 for the illustration purposes of two models at different scales. And let's think about this graph on the left, ability one, some ability. And this one, GPT-4, cannot do it yet, but it's very close to the inflection point. That little bit better model than we will probably, certainly, we'll see this jump. And then the ability in the middle, we have even this powerful GPT-4 model, is so far away from it that any kind of approach is probably not going to work in a meaningful sense. And ability three on the right, this is GPT-3, is even though past the inflection point, and at this point, whatever I work on will probably be incremental changes. And that is kind of

the implication, uh, in reality, I cannot think of whatever problem that I'm solving falls under one, two, one, or two. But just having this mental framework and always thinking about it and going back to the previous slide, updating my intuition, really makes it easier to pin down which one I'm, which problem I'm solving. So, that's kind of the perspective I want to introduce. Let's, so the summary is that everything we do has to do with scale and skill. The first perspective is really critical, but how do we actually build the scaling? I'll do the scaling done in the field. So, in the next 20 minutes or so, I'll think about this problem. And so far, all the large language models use this Transformer architecture. You probably have seen this club many, many times. The details don't really matter for us, this talk. We will think about the fundamental ideas and then build up from first principles. And for that, we need to aggressively abstract out unnecessary details. For that, I would like to say the internals of the Transformers don't really matter.

as much and so we are going to take this functional viewpoint of the Transformer and just think about it as a sequence of sequence mapping with a bunch of map metrics multiplications with the following array transformation. The input is a sequence with batch D model, which is kind of the width of the Transformer, and length, the sequence length, and the output is the same size array. And this is for training, for inference, it's different, but scaling happens at pre-train time, and that's this is the same length. So, that's going to be the picture of the transform, very simple sequence to sequence mapping. So, let me illustrate with this functional viewpoint input and the output shapes, what happens from the beginning. So, typically, we have the sentence like this, I just took a random example, many words don't map to one token, colon indivisible. This is a string, so the shape of that is just a string. And then the first step is to tokenize it, and the tokenization is done with some DPE or sentence piece, which is

essentially an external model that tries to compress the text as much as possible. So here we get the integer list. So now the shape is length and then now we embed it into this hidden space which we call um is to call word embedding. Now the representation of each token is its vector of size D model and we have length of them. So shape is the model by length and this is where actual computation happens. We have M Transformer layers and like I said this is just a mapping from the same sequence to sequence and here everything is about scale and scale really needs to be um have as little assumption as possible here. What we want is each sequence token wants to interact with other tokens in sequence so we should just let them do it but not get in the way of how they do it. So the one way of doing this in Transformer is just let it let them be able to take the dot product. That's it and everything else is kind of details and model learns to do which one to take that product and so on. So this is where

major competition happens, and because it's a high-dimensional array, we're talking about matrix multiplication or array computation. So, at the end of this Transformer, we get the sequence, and then we just take this loss function, which I'll cover in the later section. This is just a maximum likelihood based on next token prediction, so just predict the next, and we get a single number. Now, we do the backward and update all the parameters and so on. In practice, we do it in batch, which only means we just have this batch dimension appended or prepended at the shape. The only interdependence across the batch is at the end when we take the loss; we take the average of that. That's all the process. And like I said, this is where the vast majority of the compute is happening. And so, when we say scaling the Transformer, this is where we're scaling, and we need to understand this part in depth. So, let's do that from first principle. Scaling transforming means doing that part very well, which means efficiently doing matrix multiplications with many, many machines. And this involves distributing or

allocating all these matrices involved in the Transformers to various machines. Which machine gets which part of the Matrix and so on. That allocation is the critical part, and we should do that while minimizing the communication between the machines. So, this is the scaling perspective at the very low level. So, that means we should think about this matrix multiplication. So, I'm going to talk about this trivial matrix multiplication. Probably everyone knows how to do it, but with the twist that we are using multiple machines and doing it in a very scalable manner. We're going to scale it up to Transformers later. So, one at a time. So, now let's say we have some eight machines. They might be CPUs or GPUs. Now, the first and then we're going to do this trivial A B equals to C 16 by 16 matrix multiplication. And for that, the first step is to think about some kind of abstraction over hardware. These eight GPUs or CPUs might be in the same room or same cluster or maybe in different rooms or different

buildings. Whatever we don't want to deal with such things. We define this mesh, which is like a two-by-four grid, and then this layout is in virtual space. We don't care what the physical layout is, and that's we call this label this x-axis and then y-axis or there's hardware mesh. And then the job is for every com arrays involved in this computation. We map this array axis to the hardware axis. That's what we're going to do. And so here I use little color notation where this first row dimension of a is a six red 16y means this dimension is mapped to Y and then yellow X means that dimension is mapped to X. So because there are two machines across y-axis and four machines are called x-axis, what that means is a column. This rows are split in two ways, columns are four ways. So that's what I have this labels machine one, two, three, four, and so on. This is the allocation for the matrices. Now let's think about what machine one does, and it

has to do, uh, and sorry this, uh. If you look at the C, the Alp Matrix, we want to specify that after this matrix multiplication, we want the machine one to have this Slice on the top left corner of eight by four. So that's a requirement. And then we see what needs to what it needs to do. So it has to do, if you think about this matrix multiplication, we do like column-wise and row-wise, and so it has to get some data. It doesn't have everything it needs to do this computation. So we do that with this all-gather operation, which is, uh, one of the MPI operations, and we will look at this. So communicate with the four machines, so, uh, one, two, three, four, each having their local data, and then after, like, it talks to two, three, and four, gets the data, and it has all the copies, and all of them does the same thing, which is why it's all gathered instead of just gather, and that's the process that happens after this machine one has all the copies it needs to do.

that computation, and that's what happens. And crucially, this part is done in parallel across all eight machines, and that's where the speed-up comes up. Um, and then the cost was that the communication has to happen, so that's kind of the trade-off between the two. Uh, we have to think about the speed of being larger than the communication cost. So, that's the matrix multiplication. Let's generalize a little bit. Maple, we can generalize to einsum, Einstein summation notation, where which is just a higher-level view of this array of computation. So, the two rules are if the letter appears in both input, multiply component-wise. I'll show you this. So, if you look at the first one, numpy and sum of I comma I becomes I. So, we have both letter I appears in the input. That means we just multiply component-wise to get I. So, this is equivalent to A times B in numpy. And then if the letter doesn't appear on the output, then sum over the dimension I comma I becoming nothing. So, we should do that component-wise multiplication and then sum it over and

if the letter appears um and the third one is i j comma J going to I, so what we do is just take the top product over uh J and then you get this comma. So that's the Matrix Factor multiplication if you well. So that's the view of Einstein who can do. We can add many more Dimensions. It doesn't have to be two or one. Can be as many dimensions as possible. So now matrix multiplication from this perspective can be expressed as an einstum of n m n comma MP with M being this Contracting Dimension. We take the product, sum it over to get MP of the AEP. So that's that. Why are we doing this? We go back to this matrix multiplication example with this einstall. So I put it here and then now because we all have labels for this array axis MP MN whatever. So now this mapping means that I'm just saying n is array axis mapped to Y order axis and map to x-axis. And let's think about this some magic decorator parallelized that has this

sticks in this mapping and does it for you. Uh, and I'm going to talk about how this might be actually implemented later, but the real implementation of some kind actually looks like this, so very similar to this, and we will just think about it as a magic. So, what it does is it inserts, it all gathers in two dimensions, and gives you back the parallelized version of this. So, that's that. So, um, Einstein really exposes the what we need to do. So, um, yeah, now generalized one. One more thing we were doing met malls, but we are interested in Transformers. So, the most complicated operation happening is the self-attention layer in the Transformer, which can be represented in einstomes. Everything except for the soft Max can be expressed in an example, so we don't have to go into details, but you can lay out like just cleanly like this, and you have some inputs X being this uh sequence of three dimension batch, the model by length, and then all these are model parameters that are learned and will be updated. So, when you have this

you have a bunch of Einstein operations with labeled array axes now. Uh, let's go to just parallelization of this thing. So, we have eight machines now. Instead of calling an X or Y by convention, we call it data and model, and that is data parallel dimension and model parallel dimension. We're doing both data and model parallelism, and this is kind of the mesh TensorFlow way old in like 2018, but now it's a little bit more general, but we still use this terminology. So, changing this code or very little change, we just put this paralyzed and just put B is mapped to Theta because B is like the best dimension we split across the machines in the data axis, and N is a sequence length. We never in any Transformer parallel version, we never paralyze across the sequence length because there are so many like things happening there. Uh, and D nothing H, the number of heads is the model. Actually, this is why multi-head attention, the inventors here, and he actually likes really paralyzing stuff. And so, how can we parallelize this?

attention mechanism. Uh, if we split into very separate heads, then you can do it. So, that's actually, uh, motivated by this kind of a strategy. So, now once you have this, you can use the same exact same code and then just put it in, and you will get back this parallelized version that you can run with eight machines or how many ever you want. So, that has been Pro example eight machines, but this exact framework actually works with any number of machines. And here, I got it from the Google blog post, and this is actually what happened. So, uh, in TPU V4 part, it has, it can utilize like 3072 chips. And in when we were training Palm model, we used two parts. So, it's like six thousand tips showing here, each one being very escapable as, uh, some of the Heart Like highest-end GPUs. So, now you have this um bunch of machines, you define a mesh, still the same thing, model parallel Dimension is 48, and data parallel Dimension is shown in 2D, but you can flatten out, and then you have a 64. in

Palm, we actually use different things, but actually, this is exactly the same framework, just if more machines. One last bit of detail is this DC and data parallelism. So when the two parts are really not connected in anything, but it's connected with the data center Network. I think it's like 25 GPS per second, so this is slow compared to what's connecting the GPU pods. So here, we should not do any model parallelism. What we do is, after the gradient computation, we sum the gradient over this data center Network, and that's done only once, and that takes, say, one second or something. I don't know the exact number, but this is fine because when we're training models like Palm 500 billion, it can actually look at the paper. Each train step takes like 17 seconds, so if you add in like a one second here and there or 0.5, it doesn't really matter. This is really different from small model training where we're thinking about step super time. We're talking about seconds per step.

so um I think also it's really cool to just experience wise. I type in some code and then enter in my laptop, and then it just runs on all these machines. I think is really cool, and I think scaling. If you understand this, that what's happening at the end is just matrix multiplication with many, many machines that I just explained. You'll probably feel a lot better about this process. So far, we have assumed that this parallelized decorator just works. How is it actually done? One approach is this GSP MD. I link the paper there, and it's a compiler-based approach. You write neural net code as if you have a machine with infinite memory, so you don't really have to paralyze anything, which is kind of the approach we took. We just wrote no communication Rhythm and then represent the core part of this neural net, say this train step, as a computation graph. And then you can just. The graph is defined by some input tensor coming in. I'll put already going.

out and so on. You just specify how those are mapped to the hardware axis, the same process with it, and give that to XLA. It automatically inserts all gather, any other communications that has to happen to use all that machine, and this is literally magic when it works. It doesn't always work, and it really requires a lot of iterations, and many people struggle. Now, I think it really works, and if you think about all the big models that came out of Google, U5, Palm, Switch Transformer, all these things use GSPMD as a back end, exactly using the process I described. Other approaches exist, I'm not as familiar with this one, but you could think about manually annotating and separating the compiler, the operations, but at the end, the process is just the same. All involves mapping the array axes to hardware, that's all, and manual approach is sometimes better because you have more control over. So, I have some concrete examples. So, this one is this process of doing GSPMD is probably complicated for most researchers, so JAX exposes this front-end.

called PJ. I think now is just called because it just becomes a so uh useful thing. And just the processes we Define this train step that has to be parallelized and that we wrap this with Jack said fidget which is a decorator thing you get back this partition train step that can run with many many machines. And this exact code path actually was used to train palm. And so you can take a look how this kind of thing is actually done. It's a little bit more boilerplate but now that you have this um you know framework understanding you can probably just uh have some ideas what's going on there. These codes are open source and this is based on this t5x framework. We have very short paper to illustrate the systems there. I like they're linked it at the bottom there so that's good um. It's all about engineering Hardware. It's a system level thing. What about this machine learning picture so here is a llama two we recently came out so uh all looks good after a successful run but one critical

thing when you're actually running pre-training is that it's really expensive to iterate on ideas. Maybe you want to do this kind of data split or you want to do this, and each prawn has to be done at some scale, and that's really expensive. So, this is the ideal picture. In reality, if you're running on this, so let's say this is a 2000 um two trillion tokens, right? I don't know how long it took. Let's say it's 50 days. Then let's think about in one day what the progress looks like. It's like this, and you have to make decisions. Oh, oh, looks like our 70 billion model is doing well, and that I don't think I can make that call, and this is already using one day of thousands of GPUs hooked up, and then you to do some experiment. This is very difficult. So, um, and this is even a good case, and sometimes it's not really clear what to do, and making decisions with this many, many resources, uh, waiting for me is just

quite stressful situation. So, this brings to the fundamental thing in pre-training. It's all about scaling laws, and the GPT-4 technical report has some, you know, ideas about this. OpenAI has published a scaling laws paper in the past that I think is a really critical concept. So, here we're talking about prediction extrapolation of the performance of the pre-trained model across orders of magnitude, like, you know, just GPT-4. This one is a prediction, and this scaling law is developed with small-scale models. So, now we can see, okay, if we were to scale up, then this might be what we get. And we kind of, uh, this is like the test loss on the unseen corpus, but this is a guiding principle. And so, this picture is a lot more satisfactory if you have underlying scaling loss, and this is really, uh, the critical part of the pre-training. So, um, I would like to say that many, uh, now scaling is, like, a lot easier than, say, two years ago, but still very, very hard, and it's just not going from some flag.

small to large because of the mentioned reasons, and maybe one example in Palm training. There were like lost bikes. This is actually from the papers; you can check it out. There were about 20 lost spikes that happened, and this really unnerves many people, like just lost spiking up from like two to six. It's really on looks, not good. So, we train three models, exact same data, but only one of them had. That's the largest one, and so it's hard to debug and really hard to reproduce this kind of thing at smaller scale. And this is not caused by bad data. Some researchers run these experiments to verify this hypothesis, and then when this happened every hour, we not making decision is like having this giant 6100 44 chips sitting idle, and that's really hard. And so this kind of makes things makes really scaling challenging. And also, but then llama 2 and all these things make much easier to train for a given size, like tensor buildings a model like.

llama. It's easier to train, but I would say that scale is increasing at a faster rate than the rate at which things become easier at the frontier. It's always challenging for many reasons, but most people will be not scaling up. I think you can just wait for them to do the hard work, get it, and do interesting research on top of it. So, that leads to this next section: scaling doesn't solve all the problems. We also need to do a lot more research on top of this massive engineering work, and many of those can be framed into this post-training. So, let me talk about some motivations why we might need post-training. We can all talk directly to this pre-trained model because the learning objective was just next to contradiction. So, here's one example I put: let's say the input was make up a word that means when two AI researchers go on a date, and then a pre-trained model will just continue generating because it doesn't know that it has to answer the question. It's

just predict whatever is the most likely and just go on forever, and whereas what we want is more like this, just crisp answer. Um, and this is kind of the lunch models that we know, uh, how they behave. So, that's one problem. This one, we have a remedy, more like a hack. We can frame the question so that the answer happens to be the next token. So, here's another question. It's a square, whatever, some question, and then we give a colon, and then the model just doing the next prediction. Now, it's like answering this question is probably the most natural thing, so we just do it. Um, if it doesn't, then we can put an example, a few examples with the answer so that it knows how to do the thing. This is the future prompting. It is quite a powerful technique, but it's not really General and probably not the most principled solution. So, probably a bigger problem than this formatting error is that pre-trained models always generate something that is a natural continuation of the problems, even if the problems are malicious.

harmful and so on, and in particular, it doesn't know how to refuse a prompt, and all these capabilities that we think are important, is taught at the post-training level. Sometimes people call it alignment to human values; I just call it pull training. Here are kind of the four stages of the current generation of large language models. Pre-training is the first stage, and then there are three remaining stages: instruction fine-tuning, or sometimes people call it SFT (supervised fine-tuning), and then reward model and policy model training. That's kind of the RL part or LHF. So, we'll go in these three orders: one, two, three. So, let's take a look at the instruction fine-tuning. The high-level idea is that we frame all the tasks in the form of natural language instruction to natural language response mapping. So, I think the best way to think about this is what we did before we had this idea. So, if we go back to 2018, first was the best model at the time and say we are solving this.

classification task. We just have again functional viewpoint. Input is the text, output is the label, classification label. To do this kind of task, we have to do birds maps sentence, and then you need to project to the classification space. In this case, we need a test-specific parameter, and this was not really fun to extend it up to multiple tasks. T5, a year after, came along with unification of the architecture, and that now the output text output is text, so you just map text to text. All the tasks look similar, same, and it was much better. But now, if you're training on multiple tasks, multitask learning, then there's no way to distinguish the task, so you kind of put in this meta-data. This is Cola task from Glue, and then some additional things for other tasks, so that at the inference time, when you prepare this, then the model knows that I'm solving this Cola task. But this is very unnatural. Instruction following goes one step further and explains the task in natural language. It's the following sentence acceptable. That's exactly what we mean.

by Cola test we don't really need Cola. That's just some random data, and this is what we mean. And then I explain the concept, and for the other task, we can also do the same thing, explain in natural language the model understands and does the task. So this um now looks looking back is so obvious. We should have done this. Why this was not obvious at the time of D5 2019 2018 was that at the time people did not really think that lunch models understand the instruction. So having this kind of metadata felt more natural and even this was like a stretch. So now we have larger models where kind of the emergence is this understanding, which in a very vague sense is possible. This kind of utilizing the rich space of natural language makes all the things unified. So that's the viewpoint of the instruction following, and maybe one more generation will be instead of instruction in response out. You can think about dialogue as a kind of ultimate most general form of two agents.

interacting, and some of them involve question answering or other things. So, this is the instruction fine-tuning idea. So, uh, last year, when I was still at Google, I did work on this paper where we pushed the limit of instruction fine-tuning. Okay, so instruction. Um, actually, let me just take one more thing. So, if you train on these instructions, say you see a new task at the inference time, then the task is unseen during training, but still, this is just framed as an instruction. So, you just follow the instruction. That's how the generalization is done, and I think it's a really critical concept. So, that means if we have more instructions in their training set, would that be a better model, better generalization happening? And that was the hypothesis we tested with this massive thing, tasks. So, at the time, we gathered pretty much all the academic tasks we could find. Then, there were like 1,836 of them, put it into one mixture, train it. Um, I won't go into detail. I linked this paper at the bottom. Um, and this is where like Flan T5 and

one pound. Those models came out, and so here we have a model X size and the x-axis. Whenever we think about some idea, always plot the scaling so that we understand the effect of the scale on this research idea: eight billion, 62 billion, 540 billion, and on the y-axis, some average score of the evaluation set. We carefully picked six of them, unseen during training and very hard, as evidenced by this 8 billion pound model, which was trained well but still getting less than 10. So, the takeaway is that as we go from this gray line to the darker and darker blue, we add more tasks, and the performance monotonically increases but with diminishing returns. And that is more probably clear, as shown in this cloud on the right, the number of fine-tuning tasks on the right, and then same data point, but you see like each model gets the benefit of like 10, but you know, adding more tasks doesn't really help. So, what matters is the task diversity, and but still this, even with this 1800.

cash massive benefits happen for any model size. We have tried any model family you have tried, and I think for this kind of method, it's hard to see any use cases that don't benefit from this. So, very effective, but it has inherent limitations. Again, let's think about fundamental reasons why this might be not so scalable or effective. And for that, we need to go to the learning objective in this learning phase. What are we teaching the model? And so, learning objective that is supervised learning with cross-entropy loss or maximum likelihood objective, which means for this given input, the target we have is the single correct answer. Everything else is wrong. Very strong statement in reinforcement learning literature. This is called the behavioral cloning. I kind of like this terminology better. So, the hope is that if we have enough of these variations, the model can generalize to different types, even though the training signal was quite prescriptive. So, crucially, this requires formalizing the correct behavior for a given input so that the model can clone, and this has been easy and straightforward in the

past increasingly. This is becoming more difficult, and the way I'm going to illustrate this is I'm going to show you a bunch of examples, and let's do a thought experiment where just think about the single correct answer. It has to be a single one. So, start with the trivial example, two plus three. Then, it's obvious that the answer is five. Maybe the answer is five might be a better one, but anyway, this one not ambiguous. Another one, a little bit more ambiguity here. Translate this Korean. I should have studied instead of watching this movie. And this one, I transit in myself. I know Korean, so this probably is correct, but this I don't. Yeah, it was a single correct answer, but there might be some variations in it. Now, this is kind of the problems that we're actually dealing with in large things models. Write a letter to a five-year-old boy from Santa Claus explaining that Santa Claus is not real conveniently so as not to break his heart. And this one, I cannot think of any good answer, let alone the best one that I will.

confidently give it to the model and say, This is a single correct answer. So, this kind of thing, I'm less confident and comfortable using this maximum likelihood as the learning objective for the largest model we have, and that's maybe a little concocted. So, let's think about some more practical use cases. This is actually one of the favorite prompts that I use whenever there's a new model comes out: Implement logistic regression with gradient descent in Python. And for this one, there's no clean answer. I mean, a single answer for this. You might like a functional programming style, you might like an object-oriented style, or you might, within that, your style might be different, and all those might be valid. So, is it okay to give one of the completions as the only correct answer? So, let's make some observations. Increasingly, we want to teach models more abstract and ambiguous behaviors, and the objective function of the instruction fine-tuning seems to become the bottleneck. I don't have hard evidence in practice this generalizes quite well, but if you think about, say, a

thousand times larger model than GPT-4. Well, are we comfortable teaching the model with this very narrow signal? At least I'm not so. Then can we use something? What is this maximum molecular objective? If I give you a model prediction and a target, then there's a predefined mathematical function that computes this loss or objective function. So, there's no learnable parameter. If this is the bottleneck, can we make this more expressive and actually learn and parameterize it? And that's kind of the key idea for the RLHF. And RL provides one of many ways of doing this using this learned objective function. So, in RL, we try to maximize this expected reward function shown here, very general formulation. So, let's think about, say, an agent solving, playing chess. At the end of the game, if it wins the game, we give the reward of one, reward of zero otherwise. This is a valid function we call a reward, but there's no learnable parameters, anything. But we can actually use a model to formulate the rewards for more

complicated situations, and we call that a reward model. So, from this perspective, um, how do we actually do that? We know how to do supervised learning with neural nets very well, so let's use neural nets to represent that reward model, and that brings it to the next stage of the reward model. So, the core idea here is that for a given input, we give two completions, and humans provide the preference instead of saying which one is the best answer among these two. This is better - the initial like this kind of thing doesn't really say why, but this is better, and human labels just. This is how models learn what humans prefer, um, and this crucially, the completion two in this case is preferred, but it doesn't have to be the best one. It's just better than the other one. Then the model can learn the axis of difference, and this is really critical behavior, um, and why is this? Why do we use comparison for things like this? Uh, two plus three, the fact that five is better than four is probably not useful because there's a single correct answer. Why not?

use the supervised learning with maximum likelihood objective function, but for the Santa Claus example like this. I can probably come up with two someone you know just good completions and then just say this one is better and that might be better. So, the core hypothesis is that it's easier to compare than to absolutely grade something. So, uh, mathematics of this I will probably skip it, but we just model it. I'll go very quickly. Two completions y i and y j and call probability p ij is the probability that completion i is better than j, and we get that label from humans, and we model that probability as a difference in reward, uh, more specifically log odds, but that's kind of the model from 1952. And this, we just rearrange it and maximize this probability because it's essentially a maximum likelihood. The reward model is trained with the maximum likelihood, just maximize this probability, but that's fine because it's going to be a rich function providing the signal to the policy model, which is the final stage of this large Linux model, and here the

objective function is what we just learned. Reward model with the parameter V that is fixed after the initial RL um. And you might remind this. This might remind you of the Gan formulation where you have this okay this community generator here. You know we were kind of fixing one part of it and just doing the other iteration. So for this we bring some prompts. This policy model which is typically initialized from the supervised instruction pontoon checkpoint. You just generate some conditions for that prompt. Give it to the reward model. It gives back the score. And based on that if it's low then I know okay this is not good. If it's high then okay I will do more of these. So it's a try on error formalization in mathematics. That's essentially the high level idea. And because we want to maximize it with the gradient based iterative method we just take the gradient hope to take the gradient. And that is done by some policy graded algorithms like PPO or anything that should work. So that's the idea iterates.

while satisfying this RM model, and in doing so, RM encodes human preference and then gives it to the policy model learned through RL. That's the um kind of the framework in practice. Many people don't like this, and I particularly hear people saying, Oh, we should just get rid of the RL. Many, many uh people say that, not just like I'm not talking about OpenAI in specific, but just many people in the field say that. So, why is it? It's really hard to get this right uh. RL is just so tricky, and um it can go wrong. One failure mode that is probably the most common I would like to explain is what's called reward hacking. So, let's say the human labelers label this um completions like comparison pairs that this is better with whatever they're thinking about. But it happens to be that they that can preferred completions are all longer. Then this policy model can exploit this fact that, Okay, if I just give it something longer, then the reward model likes it. And increasingly, it just gives

something like dumb but long, then it can get uh better and better rewards. The reward model is happy, and human preference goes down. This divergence is what we call reward hacking, and it's so hard to control. I don't think there's a great answer to this at this point; it's an open-ended research question. This kind of thing is really a good problem to solve because I don't think we have used this RLHF in any um way like this. This is just scratching the surface, so uh yeah, that's the kind of the failure mode that is most common um, and there are kind of many ways to uh do that, but I'll probably skip that for now. So even that, even with this um kind of a fundamental change challenge that it's hard to use it um, why should we keep studying RLHF? Here's just my take um, maximum likelihood is too strong of an inductive bias, and if we think about the very strong end of the bias, it will be problematic. If you think if we have much larger scale, so GPT-4 probably fine, but

thousand times up to even larger. This, I think, will be problematic. So, learning this objective function to relieve this bottleneck in this two bias in scaling is a different paradigm, and there's a lot of room for improvements. We have seen some success stories with our LHF model, like ChatGPT, but I think it's just a beginning. Also, here's my just personal take: if something is so principle, we should just keep at it until it works. So, I would like to summarize this kind of what I just said in the pool training section with a bird's eye view of the field of AI progress, starting with this rule-based system, which I'm not really familiar with, so I might make mistakes, but just high-level concept matters here. Input is mapped by this hand-designed program to output, and here nothing is learned. And the next generation is like a classical machine learning system where input is transformed to hand-designed feature, and then that gets mapped to an output by this vulnerable part that is, let's say, like an SVM or something. And this is the

a label learnable part with the blue box, and then the output is fed to the along with the input hand design loss function objective function, and then iterate so deep learning that change one thing. This is that hand design feature was changed to learn feature. In particular, hierarchical representation learning was the technique that defines deep learning, and this one change was really successful. And why is that? That is because it is a much weaker inductive bias, and that allows much more scalable system, and we have a lot more compute utilize it, and that's how the Deep learning success story can be summarized. So, if you think about classical machine learning versus deep learning, the closest comparison will be will just regression with the V4 neural net. Same thing, except for the fact that the representation, it has some hidden representation that is learned from the data, that is whatever the model thinks is a good thing to solve this binary scratch can test. So, at this point, I was wondering what would be the next Generation if just changing color in one box with blue was so successful. Why not

do another thing, and I only see one box left over, which is this cross function. So, this still is hand-designed, so maybe we should relieve that assumption and learn it. And that, I think, is the next generation. We have one or two success stories, scan and RLHS, and this is, I think, really powerful. And it allows the model to demonstrate behaviors that are so hard to think about if you have to formalize the correct behavior. So, um, but it can be really just one instance. It can be any other RL simulations or any types of other methods, not just RL. Um, so that, I think, is the next paradigm, and we should go for that. And if I have to think about this mental picture, uh, what if kind of the representative flagship model from each paradigm, maybe like IBM Deep Blue. I don't know what exactly is like some search with that, but let's call it just robot system. And then I don't actually have a good idea of any famous classical machine learning system. Let's call it some S3M-based ones.

and then from there to the next Paradigm was gpd3. So now this much change happens in every Paradigm, and I believe that learning this loss function or objective function is as big of a change from the previous steps. Then I think it's really exciting to think about what can come out of this. That's it. Thank you.

Large Language Models (in 2023)

Large Language Models (in 2023)

Scripts