AI Progress: Reflections on Deep Learning and Future Challenges

Large neural networks, trained on massive datasets, guarantee AI success.

### Reflecting on a Decade of AI Progress and Future Speculations
* 👏 I want to express my gratitude to the organizers for selecting our paper for this award, and to my co-authors Oral and Vineel for their **incredible collaboration**.
* 🖼 A screenshot from my NeurIPS 2014 talk highlights how much has changed over the last decade, as we now possess **more experience and hopefully wisdom** in our field.
* 📊 Our original work focused on an **autoregressive model** trained on text, utilizing a **large neural network and dataset**, which was **groundbreaking at the time**.
* 🔍 We emphasized that a **10-layer neural network** could perform tasks that a human can do in a **fraction of a second**, based on the belief that artificial and biological neurons share similarities.
* 📈 The scaling hypothesis emerged from our conclusion that **training large neural networks on vast datasets would guarantee success**, a principle that has proven accurate in subsequent advancements.
* 💡 I also highlighted the importance of connectionism, which posits that **large neural networks can replicate many human functions**, leading to the **age of pre-training** that has driven **significant progress in AI**.
* 🔮 However, I warned that the era of pre-training may come to an end due to the **limitations of data availability**, as there is **only one internet**, and we must **adapt to this reality for future advancements**.
* 🤖 As we look ahead, I speculated on the potential for AI systems to become **truly agentic and capable of reasoning**, which would fundamentally change their **reliability and capabilities**, marking a **new chapter in artificial intelligence**.

I want to thank the organizers for choosing our paper for this award. It was very nice. I also want to thank my incredible co-authors and collaborators, Oral and Vineel, who spoke just a moment ago.

This image is a screenshot from a similar talk I gave 10 years ago at NeurIPS 2014 in Montreal. It was a much more innocent time.  Here we are shown in the photos – this is the 'before,' this is the 'after,' and now we've got more...

experience, hopefully wiser. But here I'd like to talk a little bit about the work itself and maybe do a 10-year retrospective on it, because a lot of the things in this work were correct, but some not so much. We can review them and see...

what happened and how it gently flowed to where we are today. So let's begin by talking about what we did.  We'll do this by showing slides from that same talk 10 years ago. But the summary of what we did is the following three bullet points: It's...

an autoregressive model trained on text; it's a large neural network; and it's a large dataset. And that's it. Now let's dive into the details a little bit more. So this was a slide from 10 years ago. Not too bad. The deep learning...

hypothesis. And what we said here is that if you have a large neural network with 10 layers, then it can do anything a human being can do in a fraction of a second.  Why did we have this emphasis on things that human beings can do in a fraction of a second?

Why this thing specifically? Well, if you believe the deep learning dogma, so to say, that artificial neurons and biological neurons are similar, or at least not too different, and you believe that real neurons are slower than anything that we can do quickly—by 'we', I mean human beings—

I even mean just one human in the entire world. If there is one human in the entire world that can do some task in a fraction of a second, then a 10-layer neural network can do it too, right? It follows; you just take their connections

and you embed them inside your artificial neuronet. So this was the motivation: anything that a human being can do in a fraction of a second, a big 10-layer neural network can do too. We focused on 10-layer neural

networks because these were the neural networks we knew how to train back in the day. If you could go beyond 10 layers somehow, then you could do more, but back then we could only do 10 layers, which is why we emphasized: whatever human beings can do in a fraction of a

second... A different slide from the talk, a slide which says 'Our main idea,' and you may be able to recognize two things, or at least one thing you might be able to recognize—that something autoregressive is going on here. What is it saying, really? What does this slide really say?

This slide says that if you have an autoregressive model and it predicts the next token well enough, then it will, in fact, grab, capture, and grasp the correct distribution over whatever sequences come next. And this was a relatively new thing; it wasn't literally

the first ever autoregressive neural network, but I would argue it was the first autoregressive neural network where we really believed that if you train it really well, then you will get whatever you want. In our case back then, it was the humble—today humble, then incredibly

audacious task of translation. Now I'm going to show you some ancient history that many of you might have never seen before. It's called the LSTM. To those unfamiliar, an LSTM is the thing that poor deep learning researchers did before Transformers, and it's basically a recurrent

net, but rotated 90°. So that's an LSTM, and it came before... it's like... it's like kind of like a slightly more complicated recurrent net. You can see there is your integrator, which is now called the residual stream, but you've got some

multiplication going on. It's a little bit more complicated, but that's what we did. It was a recurrent net rotated 90°. Another cool feature from that old talk that I want to highlight is that we used parallelization, but not just any parallelization; we used

pipelining, as witnessed by this: one layer per GPU. Was it wise to pipeline? As we now know, pipelining is not wise, but we were not as wise back then, so we used that and we got a 3.5x speedup using eight

GPUs. And the conclusion slide—in some sense, the conclusion slide from the talk from back then is the most important slide—because it spelled out what could arguably be the beginning of the scaling hypothesis: right, that if you have a very big dataset and you train a very big

Neural networks, then success is guaranteed, and one can argue, if one is charitable, that this indeed has been what's been happening. I want to mention one other idea, and this is, I claim, the idea that truly stood the test of time; it's the core

idea of deploying itself. It's the idea of connectionism; it's the idea that if you allow yourself to believe that an artificial neuron is kind of, sort of like a biological neuron—right?—if you believe that one is kind of sort of like the other, then it gives you the confidence

to believe that very large neural networks—they don't need to be literally human brain scale; they might be a little bit smaller—but you could configure them to do pretty much all the things that we do, human beings. There's still a difference...

Oh, I forgot to add: there is still a difference because the human brain also figures out how to reconfigure itself, whereas we are using the best learning algorithms that we have, which require as many data points as there are parameters. Human beings are still better in this regard, but what this led to, I claim,

arguably, to the age of pre-training. And the age of pre-training is what we might say the GPT-2 model, the GPT-3 model, the scaling laws... and I want to specifically call out my former collaborators, Alec Radford, also Jared Kaplan, Dario

Amodei, for really making this work. But that led to the age of pre-training, and this is what's been the driver of all of the progress—all the progress that we see today—extraordinarily large neural networks, extraordinarily large neural networks trained on huge data.

Sets, but pre-training, as we know it, will unquestionably end. Pre-training will end. Why will it end? Because while computers are growing through better hardware, better algorithms, and logic clusters—right, all those things keep increasing your compute—all these things keep increasing your compute, the data is not.

growing because we have but one internet. We have but one internet. You could even say—you can even go as far as to say—that data is the fossil fuel of AI. It was like created somehow, and now we use it, and we've achieved peak data, and

there'll be no more. We have to deal with the data that we have now. It still—still lets us go quite far, but this is—there's only one internet. So here I'll take um, a bit of liberty to speculate about what comes next. Actually, I don't need to speculate

because many people are speculating too, and I'll mention their speculations. You may have heard the phrase 'agents.' It's common, and I'm sure that eventually something will happen, but people feel like something—agents—is the future. More concretely, but also a little

bit vaguely, synthetic data. But what does synthetic data mean? Figuring this out is a big challenge, and I'm sure that different people have all kinds of interesting progress there, and inference-time compute, or maybe what's been most recently—most vividly—seen in the O1 model. These are all

examples of things—of people trying to figure out what to do after pre-training. And those are all very good things to do. I want to mention one other example from biology, which I think is really cool, and the example is this: So about many, many years ago at this conference, also I

I saw a talk where someone presented this graph showing the relationship between the size of a mammal's body and the size of its brain—in this case, measured by mass.  I remember vividly them saying, "Look,

in biology, everything is so messy, but here you have a rare example where there's a very tight relationship between the size of an animal's body and its brain." Totally randomly, I became curious about this graph, and so

I went to Google to research it and look for the graph. One of the images in Google Images was this.  The interesting thing in this image is—I don't know if the mouse is working... oh yeah, the mouse is working great—so you've got these

mammals, right? All the different mammals. Then you've got non-human primates; it's basically the same thing. But then you've got the hominids, and to my knowledge, hominids are close human relatives in evolution, like the

Neanderthals—there are a bunch of them, like *Homo habilis*; maybe there are a whole bunch—and they're all here. What's interesting is that they have a different slope on their brain-to-body scaling exponent. So that's pretty cool. What that

means is that there is a precedent; there's an example of biology figuring out some kind of different scaling; something clearly is different. So I think that is cool. And by the way, I want to highlight that this x-axis is a log scale; you see this is

100, this is a thousand, 10,000, 100,000, and likewise in grams: 1g, 10g, 100g, 1000g. So it is possible for things to be different. The things that we are doing, the things that we've been scaling so

far is actually the first thing that we figured out how to scale. And without doubt, the field—everyone who's working here will figure out what to do—but I want to talk here, I want to take a few minutes and speculate about the longer term; the longer term,

where are we all headed? Right? We're making all this progress; it's astounding progress. It's really—I mean, those of you who've been in the field 10 years ago and you remember just how incapable everything was—like, yes, you can say even if you

kind of say, 'Of course,' learning is still—to see it is just unbelievable. It's completely—I can't convey that feeling to you. You know, if you joined the field in the last two years, then of course you speak to computers and they talk back to

you, and they disagree, and that's what computers are. But it hasn't always been the case. But I want to talk a little bit about superintelligence, just a bit, because that is obviously where this field is headed. This is obviously what's being

built here. And the thing about superintelligence is that it will be different qualitatively from what we have. And my goal in the next minute is to try to give you some concrete intuition of how it will be different, so that you yourself could reason about it. So right now we have our incredible

Language models and the unbelievable chatbots can do amazing things, but they're also strangely unreliable and get confused, while also having dramatically superhuman performance on evaluations. So it's really unclear how to reconcile this, but eventually, sooner or later, the following will be achieved:

Those systems are actually going to be agentic in a real way, whereas right now the systems are not agents in any meaningful sense.  To say they're 'slightly agentic' might be too strong; they're just beginning.  They will actually reason and...

By the way, I want to mention something about reasoning: a system that reasons, the more it reasons, the more unpredictable it becomes. All the deep learning we've been used to is very predictable because...

If you've been working on replicating human intuition—essentially, it's like gut feeling—if you come back to the 0.1-second reaction time, what kind of processing do we do in our brains? Well, it's our intuition. So we've endowed our AI with some of that intuition, but...

Reasoning... you're seeing some early signs of that. Reasoning is unpredictable, and one reason to see that is because the really good chess AIs are unpredictable to the best human chess players. So we will have to be dealing with AI systems that are incredibly...

unpredictable. They will understand things from limited data; they will not get confused—all things which are really big limitations right now.  I'm not saying *how*, by the way, and I'm not saying *when*; I'm saying that *it will*, and when all those things will happen together...

self-awareness, because why not? Self-awareness is useful; it's part of our own world models. When all those things come together, we will have systems of radically different qualities and properties that exist today, and of course, they will have incredible

and amazing capabilities. But the kind of issues that come up with systems like this—and I'll just leave it as an exercise to imagine—it's very different from what we're used to, and I would say that it's definitely also impossible to predict the future.

Really, all kinds of stuff is possible. But on this uplifting note, I will conclude. Thank you so much. [Applause] [Music] [Applause] Thank you. Um, now, in 2024, are there other biological structures that are part of human

cognition that you think are worth exploring in a similar way, or that you're interested in anyway? So the way I'd answer this question is that if you are, or someone is, a person who has a specific insight about, 'Hey, we are all being extremely

silly because clearly the brain does something and we are not,' and that's something that can be done, they should pursue it. I personally don't—well, it depends on the level of abstraction you're looking at. Maybe I'll answer it this way: like there's been a lot of

desire to make biologically inspired AI, and you could argue on some level that biologically inspired AI is incredibly successful—which is all of the learning biologically inspired AI—but on the other hand, the biological inspiration was very, very, very modest. It's like, 'Let's use neurons.' This is the full

The extent of biological inspiration—let's use neurons and more detailed biological inspiration—has been very hard to come by, but I wouldn't rule it out. I think if someone has a special insight, they might be able to see something that would be useful.

I have a question for you about autocorrect. You mentioned reasoning as one of the core aspects of maybe future modeling and maybe a differentiator. What we saw in some of the poster sessions is that hallucinations in today's models are analyzed—maybe you can correct me, you're the expert—by analyzing whether a model is hallucinating. Because we know of the dangers of models not being able to reason, we're using statistical analysis, let's say some number of standard deviations from the mean.

In the future, wouldn't a model, given reasoning, be able to correct itself—sort of autocorrect itself—and that will be a core feature of future models so that there won't be as many hallucinations? Because the model will recognize when...

Maybe that's too esoteric a question, but will the model be able to reason and understand when a hallucination is occurring? Does the question make sense?

Yes, and the answer is also yes. I think what you described is extremely highly plausible.  You should check—for yeah, it's—I wouldn't rule out that it might already be happening with some of the...

Early reasoning models of today, I don't know, but longer term, why not? Yeah, I mean it's part of, like, Microsoft Word's autocorrect. It's a, you know, it's a core feature. Yeah, I just... I mean, I think calling it 'autocorrect' is really

doing it a disservice. I think you are, when you say 'autocorrect,' you evoke something far grander than autocorrect. But other than that point aside, the answer is yes. Thank you. I loved the ending, mysteriously leaving out...

Do they replace us, or are they, you know, superior? Do they need rights? You know, it's a new species of homo sapien, spawned intelligence, so maybe they need... I mean, uh, I think the RL guy thinks they think, uh, you know, we need rights for these things. I have a UN-related question to

that: How do you create the right incentive mechanisms for humanity to actually create it in a way that gives it the freedoms that we have as homo sapiens? You know, I feel like, in some sense, those are

those are the kind of questions that people should be reflecting on more. But to your question about what incentive structure should we create, I don't feel that I know. I don't feel confident answering questions like this because, uh, it's like you're talking

about creating some kind of top-down structure, a government thing. I don't know. It could be a cryptocurrency, too. Yeah, I mean, there's BitTorrent, you know, those things. I don't feel like I am the right person to comment on cryptocurrency, but, but you know, there is a chance by the

What you're describing will happen; indeed, we will have, in some sense, it's not a bad end result if you have AIS and all they want is to coexist with us and also just to have rights.

That will be fine, but I don't know. I mean, I think things are so incredibly unpredictable, I hesitate to comment, but I encourage the speculation. Thank you. Uh, and uh, yeah, thank you for the talk. It's really awesome.

Hi there. Thank you for the great talk. My name is Shalev Liit from the University of Toronto, working with Sheila. Thanks for all the work you've done. I wanted to ask, do you think LLMs generalize multi-hop reasoning out of distribution?

So, okay, the question assumes that the answer is yes or no, but the question should not be answered with yes or no because what does it mean, out-of-distribution generalization? What does it mean in-distribution, and what does it mean out-of-distribution?

Because it's a test of time. I'll say that long, long ago, before people were using deep learning, they were using things like string matching, n-grams for machine translation. People were using statistical phrase tables. Can you imagine? They had tens of thousands of lines of code of complexity, which was truly unfathomable.

And back then, generalization meant is it literally not in the same phrasing as in the dataset? Now we may say, well, my model achieves this high score on, um, I don't know, math competitions, but maybe the math, maybe some discussion in some forum on the internet was about the same ideas.

And therefore, it's memorized well. Okay, you could say maybe it's in-distribution, maybe it's memorization, but I also think that our standards for what counts as generalization have increased really quite substantially, dramatically, unimaginably. If you keep track, and so I think the answer is to some

degree, probably not as well as human beings. I think it is true that human beings generalize much better, but at the same time, they definitely generalize out of distribution to some degree. I hope it's a useful topological answer. Thank you, and unfortunately we're out of time

for this session. I have a feeling we could go on for the next six hours, uh, but thank you so much, Ilia, for the talk. Thank you. Wonderful. [Applause] [Music]

Ilya Sutskever: "Sequence to sequence learning with neural networks: what a decade"

seremot

Acknowledgements and Introduction of the Topic

Expresses gratitude to the organizers and co-authors.

Shows a before-and-after comparison of a past presentation (NeurIPS 2014).

Plans a 10-year retrospective on the presented work, highlighting both correct and incorrect aspects.

The work involved an autoregressive model trained on text.

Discussion of the 'deep learning hypothesis' and its implications.

Focus on tasks achievable by humans in a fraction of a second.

Explanation of the autoregressive model's function and its novelty.

Description of LSTMs and their role in the research.

Discussion of pipelining parallelization and its impact on speed.

The conclusion slide introduced the scaling hypothesis.

Long-Term Impact and the Idea of Connectionism

The enduring idea of connectionism and its implications for large neural networks.

Comparison between artificial and biological neurons.

The transition to the age of pre-training.

Acknowledgement of collaborators and their contributions.

The limitations of pre-training due to the finite nature of data.

Data is compared to fossil fuel, suggesting a peak and subsequent limitations.

Speculation on future directions, including agents and synthetic data.

Discussion of inference-time compute and the O1 model.

Biological Analogy: Mammal Brain-to-Body Size Scaling

Presentation of a graph showing the relationship between mammal body and brain size.

Observation of a different scaling exponent for hominids.

Discussion of the implications of different scaling in biology.

The possibility of different scaling in AI development.

Speculation on the Long-Term Future of AI

Discussion of the astounding progress in AI and its implications.

Focus on superintelligence and its qualitative differences from current AI.

Comparison of current language models and chatbots, highlighting their unreliability.

Discussion of agentic systems and their unpredictability due to reasoning.

Prediction of future AI systems possessing self-awareness and advanced capabilities.

Q&A: Biological Inspiration and Future of AI Models

Response to a question about other biological structures worth exploring in AI.

Discussion on the potential for future AI models to self-correct hallucinations through reasoning.

Confirmation that self-correction through reasoning is plausible.

Discussion on the ethical implications of advanced AI and the need for appropriate incentive mechanisms.

Discussion about whether LLMs generalize multi-hop reasoning out of distribution.

The speaker emphasizes the evolving definition of generalization in AI.

Comparison of generalization standards in older AI approaches and current deep learning models.

Conclusion that human beings still generalize better than current AI models, but AI models do generalize to some degree.