want to thank the organizers for choosing a paper for this award, it was very nice, and I also want to thank my incredible co-authors and collaborators.

Ollie, Vineel, and Quoc Le, who stood right before you a moment ago. And what you have here is an image, a screenshot from a similar talk 10 years ago at NeurIPS in 2014 in Montreal, and it was a much more innocent time.

Here we are shown in the photos, this is the before, here's the after, by the way.

And now we've got more experienced, hopefully wiser, but here I'd like to talk a little bit about the work itself and maybe a 10-year retrospective on it because a lot of the things in this work were correct, but some not so much, and we can review them, and we can see what happened and how it gently flowed to where we are today.

So let's begin by talking about what we did.

And the way we'll do it is by showing slides from the same talk 10 years ago. But the summary of what we did is the following three bullet points: it's an auto-regressive model trained on text, it's a large neural network, and it's a large data set.

And that's it. Now let's dive in into the details a little bit more. So this was a slide 10 years ago.

Not too bad, the Deep load hypothesis. And what we said here is that if you have a large neural network with 10 layers, then it can do anything that a human being can do in a fraction of a second.

Like why did we have this emphasis, emphasis on things that human beings can do in a fraction of a second?

Why this thing specifically? Well, if you believe the Deep Learning Dogma, so to say that artificial neurons and biological neurons are similar or at least not too different, and you believe that real neurons are slower than anything that we can do quickly.

By we, I mean human beings. I even mean just one human in the entire world. If there is one human in the entire world that can do some task in a fraction of a second, then a 10 layer neural network can do it too, right?

You just take their connections and you embed them inside your neuronet, the artificial one. So this was the motivation: anything that a human being can do in a fraction of a second, a big 10, 10 layer neural network can do too.

We focused on 10 layer neural networks because this was the neural networks we knew how to train back in the day.

If you could go beyond in your layers somehow, then you could do more, but back then we could only do 10 layers, which is why we emphasized whatever human beings can do in a fraction of a second.

A different slide from the talk, a slide which says our main idea, and you may be able to recognize two things, or at least one thing. You might be able to recognize that something auto regressive is going on here.

What is it saying really? What does this slide really say? This slide says that if you have an auto regressive model and it predicts the next token well enough, then it will in fact grab and capture and grasp the correct distribution over whatever, over sequences that come next.

And this was a relatively new thing. It wasn't literally the first ever auto regressive neural network, but I would argue it was the first auto regressive neural network where we really believed that if you train it really well, then you will get whatever you want in our case back then was the humble, today humble then incredibly audacious task of translation.

Now I'm going to show you some ancient history that many of you might have never seen before.

It's called the LSTM. To those unfamiliar, an LSTM is the things that poor deep learning researchers did before Transformers, and it's basically a ResNet, but rotated 90°.

It's like, it's kind of like a slightly more complicated reset. You can see there is your integrator, which is now called the residual stream, but you've got some multiplication going on.

It's a little bit more complicated, but that's what we did. It was a ResNet Ro 90°.

Another cool feature from that old talk that I want to highlight is that we used parallelization, but not just any parallelization, we used pipelining, as witnessed by this one layer per GPU.

Was it wise to pipeline? As we now know, pipelining is not wise, but we were not as wise back then.

So we used that and we got a 3.5x speed up using eight GPUs and the conclusion slide, in some sense, the conclusion slide from the talk from back then is the most important slide because it spelled out what could arguably be the beginning of the scaling hypothesis, right?

That if you have a very big data set and you train a very big neural network, then success is guaranteed.

And one can argue, if one is charitable, that this indeed has been what's been happening. I want to mention one other idea and this is, I claim, the idea that truly stood the test of time.

It's the core idea of deploying itself. It's the idea of connectionism.

It's the idea that if you allow yourself to believe that an artificial neuron is kind of sort of like a biological neuron, right?

If you believe that one is kind of sort of like the other, then it gives you the confidence to believe that very large neural networks, they don't need to be literally human brain scale, they might be a little bit smaller, but you could configure them to do pretty much all the things that we do human beings.

There's still a difference. Oh, I forgot to the end. There is still a difference because the human brain also figures out how to reconfigure itself, whereas we are using the best learning algorithms that we have which require as many data points as there are parameters.

Human beings are still better in this regard.

But what this led, so I claim arguably to the age of pre-training. And the age of pre-training is what we might say the GPT-2 model, the GPT-3 model, the scaling laws, and I want to specifically call out my former collaborators Alec Radford, also Jared Kaplan, Dario Amodei, for really making this work.

But that led to the age of pre-training, and this is what's been the driver of all of progress, all the progress that we see today, extra large neural networks, extraordinarily large neural networks trained on huge data sets.

But pre-training as we know it will unquestionably end. Pre-training will end. Why will it end? Because while computers growing through better hardware, better algorithms and logic clusters, right?

All those things keep increasing your compute. All these things keep increasing your compute. The data is not growing because we have but one internet.

We have but one internet. You could even say, you can even go as far as to say that data is the fossil fuel of AI, it was like created somehow and now we use it and we've achieved peak data and there'll be no more.

We have to deal with the data that we have now.

It still still let us go quite far, but this is, there's only one internet, so here I'll take um a bit of liberty to speculate about what comes next.

Actually, I don't need to speculate because many people are speculating too and I'll mention their speculations.

You may have heard the phrase agents, it's common and I'm sure that eventually something will happen, but people feel like something agents is the future.

More concretely, but also a little bit vaguely, synthetic data. But what does synthetic data mean? Figuring this out is a big challenge and I'm sure that different people have all kinds of interesting progress there and an inference time compute or maybe what's been most recently, most vividly seen in 01, the O1 model.

These are all examples of things of people trying to figure out what to do after pre-training and those are all very good things to do.

I want to mention one other example from biology which I think is really cool and the example is this so about many many years ago at this conference also I saw a talk where someone presented this graph, but the graph showed the relationship between the size of the body of the size of the body of a mammal and the size of their brain, in this case it's in mass and the that talk I remember vividly they were saying look it's in biology everything is so messy but here you have one rare example where there is a very tight relationship between the size of the body of the animal and their brain and totally randomly I became curious at this graph and one of the early one of the early so I went to Google to do research, to look for this graph, and one of the images in Google Images was this, and the interesting thing in this image is you see, like, I don't know, is the mouse working?

Oh yeah, the mouse is working great. So you've got these mammals, right?

All the different mammals. Then you've got non-human primates. It's basically the same thing. But then you've got the hominids, and to my knowledge, hominids are like close relatives to the humans in evolution, like the Neanderthals.

There's a bunch of them, like it's called Homo habilis, maybe there are a whole bunch, and they're all here.

And what's interesting is that they have a different slope on their brain-to-body scaling exponent.

So that's pretty cool. What that means is that there is a precedent, there is an example of biology figuring out some kind of different scaling.

Something clearly is different. So I think that is cool. And by the way, I want to highlight highlight this x-axis is log scale.

You see this is 100, this is 1,000, 10,000, 100,000, and likewise in grams, 1 g, 10 g, 100 g, 1,000 g.

So it is possible for things to be different. The things that we are doing, the things that we've been scaling so far is actually the first thing that we figured out how to scale, and without doubt the field, everyone who's working here will figure out what to do.

But I want to talk here, I want to take a few minutes and speculate about the longer term, the longer term, where are we all headed, right?

We're making all this progress. It's an it's astounding progress. It's really, I mean, those of you who've been in the field 10 years ago and you remember just how incapable everything has been, like, yes, you can say, even if you kind of say, of course learning still to see it is just unbelievable.

It's completely, I can't convey that feeling to you. You know, if you joined the field in the last two years, then of course, you speak to computers and they talk back to you and they disagree and that's what computers are, but it hasn't always been the case.

But I want to talk a little bit about super intelligence, just a bit, 'cause that is obviously where this field is headed.

This is obviously what's being built here and the thing about super intelligence is that it will be different qualitatively from what we have and my goal in the next minute to try to give you some concrete intuition of how it will be different so that you yourself could reason about it.

So right now we have our incredible language models and the unbelievable chat bot and they can even do things but they're also kind of strangely unreliable and they get confused when, while also having dramatically superhuman performance on evals, so it's really unclear how to reconcile this.

But eventually sooner or later the following will be achieved, those systems are actually going to be agentic in a real ways, whereas right now the systems are not agents in any meaningful sense, just very, that might be too strong, they're very, very slightly agentic, just beginning.

It will actually reason and by the way, I want to mention something about reasoning is that a system that reasons, the more it reasons, the more unpredictable it becomes.

The more it reasons, the more unpredictable it becomes. All the deep learning that we've been used to is very predictable because if you've been working on replicating human intuition, essentially it's like the gut feeling.

If you come back to the 0.1 second reaction time, what kind of processing we do in our brains, well, it's our intuition.

So we've endowed our AIs with some of that intuition, but reasoning, you're seeing some early signs of that.

Reasoning is unpredictable, and one reason to see that is because the chess AIs, the really good ones, are unpredictable to the best human chess players.

So we will have to be dealing with AI systems that are incredibly unpredictable. They will understand things from limited data.

They will not get confused, all the things which are really big limitations. I'm not saying how, by the way, and I'm not saying when.

I'm saying that it will, and when all those things will happen together with self-awareness, because why not? Self-awareness is useful, it is part your ourselves, are parts of our own world models.

When all those things come together, we will have systems of radically different qualities and properties that exist today.

And of course, they will have incredible and amazing capabilities, but the kind of issues that come up with systems like this, and I'll just leave it as an exercise to imagine, it's very different from what we used to, and I would say that it's definitely also impossible to predict the future.

Really, all kinds of stuff is possible, but on this uplifting note, I will conclude. Thank you so much. Um, thank you.

Um, now in 2024, are there other biological structures that are part of human cognition that you think are worth exploring in a similar way or that you're interested in anyway?

So the way I'd answer this question is that if you are or someone is a person who has a specific insight about, hey, we are all being extremely silly because clearly the brain does something and we are not and that's something that can be done, they should pursue it.

I personally don't, well, depends on the level of abstraction you're looking at.

Maybe I'll answer it this way, like there's been a lot of desire to make biologically inspired AI, and you could argue on some level that biologically inspired AI is incredibly successful, which is all of the learning biologically inspired AI, but on the other hand, the biological inspiration was very, very, very modest.

It's like, let's use neurons, this is the full extent of the biological inspiration.

Let's use neurons and more detailed biological inspiration has been very hard to come by, but I wouldn't rule it out. I think if someone has a special insight, they might be able to to see something and that would be useful.

I have a question for you, um, about sort of autocorrect.

Um, so here is, here's the question. You mentioned reasoning as being um one of the core aspects of maybe the modeling in the future and maybe a differentiator.

Um, what we saw in some of the poster sessions is that hallucinations in today's models are the way we're analyzing, I mean, maybe you correct me, you're the expert on this, but the way we're analyzing whether a model is hallucinating today without because we know of the dangers of models not being able to reason, that we're using a statistical analysis, let's say some amount of standard deviations or whatever away from the mean.

In the future, wouldn't it, would you think that a model given reasoning will be able to correct itself sort of autocorrect itself and that will be a core feature of future models so that there won't be as many hallucinations because the model will recognize when I— maybe that's too esoteric of a question, but the model will be able to reason and understand when a hallucination is occurring.

Does the question make sense? Yes, and the answer is also yes. I think what you described is extremely highly plausible. Yeah, I mean, you should check. I mean, for yeah, it's— I wouldn't— I wouldn't rule out that it might already be happening with some of the, you know, early reasoning models of today.

I don't know, but longer term, why not? Yeah, I mean, it's part— part of like Microsoft Word like autocorrect.

It's a— you know, it's a— it's a core feature. Yeah, I just— I mean, I think calling it autocorrect is really doing it a disservice.

I think you are— when you say autocorrect, you evoke— like, it's far grander than autocorrect, but other than— but, you know, this point aside, the answer is yes.

Thank you. Ilya, I loved the ending, uh, mysteriously, uh, leaving out, do they replace us or are they, you know, superior?

Do they need rights? You know, it's a new species of homo sapien spawned intelligence, so maybe they need— I mean, uh, I think the RL guy, uh, thinks they think, uh, you know, we need rights for these things.

I have a UNR question to that. How do you— how do you create the right incentive mechanisms for humanity to actually create it in a way that gives it the freedoms that we have as Homo sapiens?

You know, I feel like this in some— in some— in some sense, those are— those are the kind of questions that people should be, uh, reflecting on more.

But to your question about what incentive structure should we create?

I I don't feel that I know. I don't feel confident answering questions like this because, uh, it's like you're talking about creating some kind of a top-down structure, government thing.

I don't know, it could be a cryptocurrency too. Yeah, I mean there's Bittensor, you know, those things.

I don't feel like I am the right person to comment on cryptocurrency, but, but, you know, there is a chance, by the way, what what you're describing will happen, that indeed we will have, you know, in some sense, it's it's it's not a bad end result if you have AIs and all they want is to coexist with us and also just to have rights, maybe that will be fine.

It's but I don't know. I mean, I think things are so incredibly unpredictable. I I hesitate to comment, but I encourage the speculation. Thank you. Uh, and, uh, yeah, thank you for the talk.

It's really awesome. Hi there. Thank you for the great talk. My name is Shalev Liit from University of Toronto, working with Sheila.

Thanks for all the work you've done. I wanted to ask, do you think LLMs generalize multi-hop reasoning out of distribution?

So okay, the question assumes that the answer is yes or no, but the question should not be answered with yes or no because what does it mean out of distribution generalization?

What does it mean? What does it mean in distribution and what does it mean out of distribution? Because it's a test of time talk.

I'll say that long, long ago, before people were using deep learning, they were using things like string matching engrams for machine translation.

People were using statistical phrase tables. Can you imagine? They had tens of thousands of code of complexity, which was, I mean, it's it was truly unfathomable, and back then, generalization meant, is it literally not in the same phrasing as in the data?

Ilya Sutskever: "Sequence to sequence learning with neural networks: what a decade"

seremot

Acknowledgements and Introduction of the Topic

Expresses gratitude to the organizers and co-authors.

Shows a before-and-after comparison of a past presentation (NeurIPS 2014).

Plans a 10-year retrospective on the presented work, highlighting both correct and incorrect aspects.

The work involved an autoregressive model trained on text.

Discussion of the 'deep learning hypothesis' and its implications.

Focus on tasks achievable by humans in a fraction of a second.

Explanation of the autoregressive model's function and its novelty.

Description of LSTMs and their role in the research.

Discussion of pipelining parallelization and its impact on speed.

The conclusion slide introduced the scaling hypothesis.

Long-Term Impact and the Idea of Connectionism

The enduring idea of connectionism and its implications for large neural networks.

Comparison between artificial and biological neurons.

The transition to the age of pre-training.

Acknowledgement of collaborators and their contributions.

The limitations of pre-training due to the finite nature of data.

Data is compared to fossil fuel, suggesting a peak and subsequent limitations.

Speculation on future directions, including agents and synthetic data.

Discussion of inference-time compute and the O1 model.

Biological Analogy: Mammal Brain-to-Body Size Scaling

Presentation of a graph showing the relationship between mammal body and brain size.

Observation of a different scaling exponent for hominids.

Discussion of the implications of different scaling in biology.

The possibility of different scaling in AI development.

Speculation on the Long-Term Future of AI

Discussion of the astounding progress in AI and its implications.

Focus on superintelligence and its qualitative differences from current AI.

Comparison of current language models and chatbots, highlighting their unreliability.

Discussion of agentic systems and their unpredictability due to reasoning.

Prediction of future AI systems possessing self-awareness and advanced capabilities.

Q&A: Biological Inspiration and Future of AI Models

Response to a question about other biological structures worth exploring in AI.

Discussion on the potential for future AI models to self-correct hallucinations through reasoning.

Confirmation that self-correction through reasoning is plausible.

Discussion on the ethical implications of advanced AI and the need for appropriate incentive mechanisms.

Discussion about whether LLMs generalize multi-hop reasoning out of distribution.

The speaker emphasizes the evolving definition of generalization in AI.

Comparison of generalization standards in older AI approaches and current deep learning models.

Conclusion that human beings still generalize better than current AI models, but AI models do generalize to some degree.

Ilya Sutskever: "Sequence to sequence learning with neural networks: what a decade"

Ilya Sutskever: "Sequence to sequence learning with neural networks: what a decade"

Scripts