I want to thank the organizers for choosing our paper for this award. It was very nice, and I also want to thank my

incredible co-authors and collaborators, Oral Vinyals and Quoc Le, who stood right before you a moment

ago. And what you have here is an image, a screenshot from a similar talk 10 years ago at NeurIPS

in 2014 in Montreal. And it was a much more innocent time. Here we are shown in the

photos—this is the before, here's the after, by the way—and now we've got more experienced, hopefully wiser,

but here I'd like to talk a little bit about the work itself and maybe a 10-year

retrospective on it, because a lot of the things in this work were correct, but some not so much,

and we can review them and we can see what happened and how it gently flowed to where we are

today. So let's begin by talking about what we did, and the way we'll do it is by showing

slides from the same talk 10 years ago. But the summary of what we did is the following three bullet points: It's

an autoregressive model trained on text; it's a large neural network; and it's a large data set. And that's it. Now let's

dive into the details a little bit more. So this was a slide 10 years ago. Not too bad. The deep learning

hypothesis. And what we said here is that if you have a large neural network with 10 layers, then it can do anything that a

human being can do in a fraction of a second. Like, why did we have this emphasis on things that human

beings can do in a fraction of a second? Why this thing specifically? Well, if you believe the deep learning dogma, so to

Say that artificial neurons and biological neurons are similar, or at least not too different, and you believe that real

neurons are slower than anything that we can do quickly. By "we," I mean human beings; I even mean just one human in the entire

world. If there is one human in the entire world that can do some task in a fraction of a second, then a 10-layer

neural network can do it too, right? It follows: you just take their connections and you embed them inside your neural net,

the artificial one. So this was the motivation: anything that a human being can do in a fraction of a second,

a 10-layer neural network can do too. We focused on 10-layer neural networks because this was the neural

network we knew how to train back in the day. If you could go beyond 10 layers somehow, then you could do more,

but back then we could only do 10 layers, which is why we emphasized: whatever human beings can do in a fraction of a

second. A different slide from the talk, a slide which says our main idea, and you may be able to recognize two things, or

at least one thing you might be able to recognize: that something autoregressive is going on here. What is it saying

really? What does this slide really say? This slide says that if you have an autoregressive model and it predicts the next token

well enough, then it will in fact grab, capture, and grasp the correct distribution over whatever sequences come next. And this was a

relatively new thing. It wasn't literally the first ever autoregressive neural network, but I would argue it was the first autoregressive

neural network where we really believed that if you train it really well, then you will get whatever

you want. In our case, back then, was the humble—today humble, then incredibly audacious—task of translation. Now I'm going to show you

Some ancient history that many of you might have never seen before. It's called the LSTM. To those unfamiliar, an LSTM is the

thing that poor deep learning researchers did before Transformers. And it's basically a ResNet, but rotated 90°. So that's an

LSTM, and it came before... it's like, it's kind of like a slightly more complicated ResNet. You can see there

is your integrator, which is now called the residual stream, but you've got some multiplication going on. It's a little

bit more complicated, but that's what we did. It was a ResNet rotated 90°. Another cool feature from that old

talk that I want to highlight is that we used parallelization, but not just any parallelization; we used

pipelining, as witnessed by this one layer per GPU. Was it wise to pipeline? As we now know, pipelining is not wise, but we were

not as wise back then, so we used that and we got a 3.5x speedup using eight

GPUs. And the conclusion slide—in some sense, the conclusion slide from the talk from back then is the most important slide—

because it spelled out what could arguably be the beginning of the scaling hypothesis: right, that if you have a very

big data set and you train a very big neural network, then success is guaranteed. And one can argue, if one is

charitable, that this indeed has been what's been happening. I want to mention one other idea, and this is—I claim—the idea that truly

stood the test of time. It's the core idea of deep learning itself. It's the idea of connectionism. It's the idea that if

you allow yourself to believe that an artificial neuron is kind of, sort of like a biological neuron—right, if you believe that one is

kind of sort of like the other—then it gives you the confidence to believe that very large neural networks; they don't need to be literally

human brain scale, they might be a little bit smaller, but you could configure them to do pretty much all the

things that we do, human beings. There's still a difference—oh, I forgot to add at the end—there is still a difference because

the human brain also figures out how to reconfigure itself, whereas we are using the best learning algorithms that we

have, which require as many data points as there are parameters. Human beings are still better in this regard, but what this led to—so I claim

arguably—to the age of pre-training. And the age of pre-training is what we might say the GPT-2 model, the GPT-3 model, the

scaling laws, and I want to specifically call out my former collaborators, Alec Radford, also Jared Kaplan, Dario

Amodei, for really making this work. But that led to the age of pre-training, and this is what's been the

driver of all of progress—all the progress that we see today. Extra-large neural networks, extraordinarily large neural

networks trained on huge data sets. But pre-training, as we know it, will unquestionably end. Pre-training will end. Why will it end?

Because while computers are growing through better hardware, better algorithms, and logic clusters—right, all those things keep increasing your

compute—all these things keep increasing your compute, the data is not growing because we have but one internet. We have but one

internet. You could even say—you can even go as far as to say—that data is the fossil fuel of

AI. It was like created somehow, and now we use it, and we've achieved peak data, and there'll be no more. We have to deal with

the data that we have now. It still—still lets us go quite far, but this is—there's only one

internet. So here I'll take um, a bit of liberty to speculate about what comes next. Actually, I don't need to speculate

because many people are speculating, too, and I'll mention their speculations. You may have heard the phrase "agents". It's common, and I'm sure

that eventually something will happen. But people feel like something agents is the future, more concretely, but also a little

bit vaguely, synthetic data. But what does synthetic data mean? Figuring this out is a big challenge, and I'm sure that different people have

all kinds of interesting progress there. And inference time compute, or maybe what's been most recently, most vividly

seen in the O1 model; these are all examples of things, of people trying to figure out what to do after

pre-training. And those are all very good things to do. I want to mention one other example from biology, which I think is really

cool. And the example is this: So, about many, many years ago at this conference, also, I saw a talk where someone presented this

graph. But the graph showed the relationship between the size of the body, the size of the body of a mammal, and

the size of their brain; in this case, it's in mass. And that talk, I remember vividly, they were saying, "Look,

it's in biology; everything is so messy. But here you have one rare example where there is a very tight relationship

between the size of the body of the animal and their brain." And totally randomly, I became curious at this graph,

and one of the early, one of the early, so I went to Google to do research, to look for this graph. And one of the

images in Google Images was this. And the interesting thing in this image is you see, like, I don't know, is

the mouse working? Oh yeah, the mouse is working great! So you've got this mammals, right? All the different

mammals. Then you've got non-human primates; it's basically the same thing. But then you've got the hominids, and to

My knowledge: hominids are close relatives to humans in evolution, like Neanderthals; there are many, like *Homo habilis*. There's a whole bunch, and what's interesting is that they have a

different slope on their brain-to-body scaling exponent. So that's pretty cool. What that means is that there is a precedent; there's an example of biology figuring out some kind of different scaling; something clearly is different. So I think that is cool. And by

the way, I want to highlight that this x-axis is a log scale. You see, this is 100, this is 1000, 10,000, 100,000, and

likewise in grams: 1 g, 10 g, 100 g, 1000 g. So it is possible for things to be

different. The things that we are doing, the things that we've been scaling so far, is actually the first thing that we

figured out how to scale. And without doubt, the field—everyone who's working here—will figure out

what to do. But I want to talk here; I want to take a few minutes and speculate about the longer term. The longer term:

where are we all headed? Right? We're making all this progress; it's astounding progress. It's really—I

mean, those of you who've been in the field 10 years ago, you remember just how incapable everything has

been. Like, yes, you can say—even if you kind of say, 'Of course'—learning to still see it is just

unbelievable. It's completely—I can't convey that feeling to you. You know, if you joined the field in

the last two years, then of course you speak to computers and they talk back to you, and they disagree, and that's what

computers are. But it hasn't always been the case. But I want to talk a little bit about superintelligence, just a bit.

because that is obviously where this field is headed; this is obviously what's being built here. And the thing about super

intelligence is that it will be different qualitatively from what we have. And my goal in the next minute is to

try to give you some concrete intuition of how it will be different, so that you yourself could reason about

it. So right now we have our incredible language models and the unbelievable chatbots, and they can even do things, but

they're also kind of strangely unreliable, and they get confused while also having dramatically superhuman

performance on evaluations. So it's really unclear how to reconcile this, but eventually, sooner or later, the following

will be achieved: those systems are actually going to be agentic in a real way, whereas right now the systems are

not agents in any meaningful sense; just very—that might be too strong—they're very, very slightly agentic, just

beginning. It will actually reason. And by the way, I want to mention something about reasoning: a system that

reasons, the more it reasons, the more unpredictable it becomes. The more it reasons, the more unpredictable it

becomes. All the deep learning that we've been used to is very predictable because if you've been working on replicating

human intuition—essentially, it's like the gut feeling. If you come back to the 0.1-second reaction time, what kind of

Ilya Sutskever NeurIPS 2024 full talk

Ilya Sutskever NeurIPS 2024 full talk

Scripts