I want to thank the organizers for choosing our paper for this award. It was very nice. I also want to thank my incredible co-authors and collaborators, Oral and Vineel, who spoke just a moment ago.

This image is a screenshot from a similar talk I gave 10 years ago at NeurIPS 2014 in Montreal. It was a much more innocent time.  Here we are shown in the photos – this is the 'before,' this is the 'after,' and now we've got more...

experience, hopefully wiser. But here I'd like to talk a little bit about the work itself and maybe do a 10-year retrospective on it, because a lot of the things in this work were correct, but some not so much. We can review them and see...

what happened and how it gently flowed to where we are today. So let's begin by talking about what we did.  We'll do this by showing slides from that same talk 10 years ago. But the summary of what we did is the following three bullet points: It's...

an autoregressive model trained on text; it's a large neural network; and it's a large dataset. And that's it. Now let's dive into the details a little bit more. So this was a slide from 10 years ago. Not too bad. The deep learning...

hypothesis. And what we said here is that if you have a large neural network with 10 layers, then it can do anything a human being can do in a fraction of a second.  Why did we have this emphasis on things that human beings can do in a fraction of a second?

Why this thing specifically? Well, if you believe the deep learning dogma, so to say, that artificial neurons and biological neurons are similar, or at least not too different, and you believe that real neurons are slower than anything that we can do quickly—by 'we', I mean human beings—

I even mean just one human in the entire world. If there is one human in the entire world that can do some task in a fraction of a second, then a 10-layer neural network can do it too, right? It follows; you just take their connections

and you embed them inside your artificial neuronet. So this was the motivation: anything that a human being can do in a fraction of a second, a big 10-layer neural network can do too. We focused on 10-layer neural

networks because these were the neural networks we knew how to train back in the day. If you could go beyond 10 layers somehow, then you could do more, but back then we could only do 10 layers, which is why we emphasized: whatever human beings can do in a fraction of a

second... A different slide from the talk, a slide which says 'Our main idea,' and you may be able to recognize two things, or at least one thing you might be able to recognize—that something autoregressive is going on here. What is it saying, really? What does this slide really say?

This slide says that if you have an autoregressive model and it predicts the next token well enough, then it will, in fact, grab, capture, and grasp the correct distribution over whatever sequences come next. And this was a relatively new thing; it wasn't literally

the first ever autoregressive neural network, but I would argue it was the first autoregressive neural network where we really believed that if you train it really well, then you will get whatever you want. In our case back then, it was the humble—today humble, then incredibly

audacious task of translation. Now I'm going to show you some ancient history that many of you might have never seen before. It's called the LSTM. To those unfamiliar, an LSTM is the thing that poor deep learning researchers did before Transformers, and it's basically a recurrent

net, but rotated 90°. So that's an LSTM, and it came before... it's like... it's like kind of like a slightly more complicated recurrent net. You can see there is your integrator, which is now called the residual stream, but you've got some

multiplication going on. It's a little bit more complicated, but that's what we did. It was a recurrent net rotated 90°. Another cool feature from that old talk that I want to highlight is that we used parallelization, but not just any parallelization; we used

pipelining, as witnessed by this: one layer per GPU. Was it wise to pipeline? As we now know, pipelining is not wise, but we were not as wise back then, so we used that and we got a 3.5x speedup using eight

GPUs. And the conclusion slide—in some sense, the conclusion slide from the talk from back then is the most important slide—because it spelled out what could arguably be the beginning of the scaling hypothesis: right, that if you have a very big dataset and you train a very big

Neural networks, then success is guaranteed, and one can argue, if one is charitable, that this indeed has been what's been happening. I want to mention one other idea, and this is, I claim, the idea that truly stood the test of time; it's the core

idea of deploying itself. It's the idea of connectionism; it's the idea that if you allow yourself to believe that an artificial neuron is kind of, sort of like a biological neuron—right?—if you believe that one is kind of sort of like the other, then it gives you the confidence

to believe that very large neural networks—they don't need to be literally human brain scale; they might be a little bit smaller—but you could configure them to do pretty much all the things that we do, human beings. There's still a difference...

Oh, I forgot to add: there is still a difference because the human brain also figures out how to reconfigure itself, whereas we are using the best learning algorithms that we have, which require as many data points as there are parameters. Human beings are still better in this regard, but what this led to, I claim,

arguably, to the age of pre-training. And the age of pre-training is what we might say the GPT-2 model, the GPT-3 model, the scaling laws... and I want to specifically call out my former collaborators, Alec Radford, also Jared Kaplan, Dario

Amodei, for really making this work. But that led to the age of pre-training, and this is what's been the driver of all of the progress—all the progress that we see today—extraordinarily large neural networks, extraordinarily large neural networks trained on huge data.

Sets, but pre-training, as we know it, will unquestionably end. Pre-training will end. Why will it end? Because while computers are growing through better hardware, better algorithms, and logic clusters—right, all those things keep increasing your compute—all these things keep increasing your compute, the data is not.

growing because we have but one internet. We have but one internet. You could even say—you can even go as far as to say—that data is the fossil fuel of AI. It was like created somehow, and now we use it, and we've achieved peak data, and

there'll be no more. We have to deal with the data that we have now. It still—still lets us go quite far, but this is—there's only one internet. So here I'll take um, a bit of liberty to speculate about what comes next. Actually, I don't need to speculate

because many people are speculating too, and I'll mention their speculations. You may have heard the phrase 'agents.' It's common, and I'm sure that eventually something will happen, but people feel like something—agents—is the future. More concretely, but also a little

bit vaguely, synthetic data. But what does synthetic data mean? Figuring this out is a big challenge, and I'm sure that different people have all kinds of interesting progress there, and inference-time compute, or maybe what's been most recently—most vividly—seen in the O1 model. These are all

examples of things—of people trying to figure out what to do after pre-training. And those are all very good things to do. I want to mention one other example from biology, which I think is really cool, and the example is this: So about many, many years ago at this conference, also I

I saw a talk where someone presented this graph showing the relationship between the size of a mammal's body and the size of its brain—in this case, measured by mass.  I remember vividly them saying, "Look,

in biology, everything is so messy, but here you have a rare example where there's a very tight relationship between the size of an animal's body and its brain." Totally randomly, I became curious about this graph, and so

I went to Google to research it and look for the graph. One of the images in Google Images was this.  The interesting thing in this image is—I don't know if the mouse is working... oh yeah, the mouse is working great—so you've got these

mammals, right? All the different mammals. Then you've got non-human primates; it's basically the same thing. But then you've got the hominids, and to my knowledge, hominids are close human relatives in evolution, like the

Neanderthals—there are a bunch of them, like *Homo habilis*; maybe there are a whole bunch—and they're all here. What's interesting is that they have a different slope on their brain-to-body scaling exponent. So that's pretty cool. What that

means is that there is a precedent; there's an example of biology figuring out some kind of different scaling; something clearly is different. So I think that is cool. And by the way, I want to highlight that this x-axis is a log scale; you see this is

100, this is a thousand, 10,000, 100,000, and likewise in grams: 1g, 10g, 100g, 1000g. So it is possible for things to be different. The things that we are doing, the things that we've been scaling so

far is actually the first thing that we figured out how to scale. And without doubt, the field—everyone who's working here will figure out what to do—but I want to talk here, I want to take a few minutes and speculate about the longer term; the longer term,

where are we all headed? Right? We're making all this progress; it's astounding progress. It's really—I mean, those of you who've been in the field 10 years ago and you remember just how incapable everything was—like, yes, you can say even if you

kind of say, 'Of course,' learning is still—to see it is just unbelievable. It's completely—I can't convey that feeling to you. You know, if you joined the field in the last two years, then of course you speak to computers and they talk back to

you, and they disagree, and that's what computers are. But it hasn't always been the case. But I want to talk a little bit about superintelligence, just a bit, because that is obviously where this field is headed. This is obviously what's being

built here. And the thing about superintelligence is that it will be different qualitatively from what we have. And my goal in the next minute is to try to give you some concrete intuition of how it will be different, so that you yourself could reason about it. So right now we have our incredible

Language models and the unbelievable chatbots can do amazing things, but they're also strangely unreliable and get confused, while also having dramatically superhuman performance on evaluations. So it's really unclear how to reconcile this, but eventually, sooner or later, the following will be achieved:

Those systems are actually going to be agentic in a real way, whereas right now the systems are not agents in any meaningful sense.  To say they're 'slightly agentic' might be too strong; they're just beginning.  They will actually reason and...

By the way, I want to mention something about reasoning: a system that reasons, the more it reasons, the more unpredictable it becomes. All the deep learning we've been used to is very predictable because...

If you've been working on replicating human intuition—essentially, it's like gut feeling—if you come back to the 0.1-second reaction time, what kind of processing do we do in our brains? Well, it's our intuition. So we've endowed our AI with some of that intuition, but...

Reasoning... you're seeing some early signs of that. Reasoning is unpredictable, and one reason to see that is because the really good chess AIs are unpredictable to the best human chess players. So we will have to be dealing with AI systems that are incredibly...

unpredictable. They will understand things from limited data; they will not get confused—all things which are really big limitations right now.  I'm not saying *how*, by the way, and I'm not saying *when*; I'm saying that *it will*, and when all those things will happen together...

self-awareness, because why not? Self-awareness is useful; it's part of our own world models. When all those things come together, we will have systems of radically different qualities and properties that exist today, and of course, they will have incredible

and amazing capabilities. But the kind of issues that come up with systems like this—and I'll just leave it as an exercise to imagine—it's very different from what we're used to, and I would say that it's definitely also impossible to predict the future.

Really, all kinds of stuff is possible. But on this uplifting note, I will conclude. Thank you so much. [Applause] [Music] [Applause] Thank you. Um, now, in 2024, are there other biological structures that are part of human

cognition that you think are worth exploring in a similar way, or that you're interested in anyway? So the way I'd answer this question is that if you are, or someone is, a person who has a specific insight about, 'Hey, we are all being extremely

silly because clearly the brain does something and we are not,' and that's something that can be done, they should pursue it. I personally don't—well, it depends on the level of abstraction you're looking at. Maybe I'll answer it this way: like there's been a lot of

desire to make biologically inspired AI, and you could argue on some level that biologically inspired AI is incredibly successful—which is all of the learning biologically inspired AI—but on the other hand, the biological inspiration was very, very, very modest. It's like, 'Let's use neurons.' This is the full

The extent of biological inspiration—let's use neurons and more detailed biological inspiration—has been very hard to come by, but I wouldn't rule it out. I think if someone has a special insight, they might be able to see something that would be useful.

I have a question for you about autocorrect. You mentioned reasoning as one of the core aspects of maybe future modeling and maybe a differentiator. What we saw in some of the poster sessions is that hallucinations in today's models are analyzed—maybe you can correct me, you're the expert—by analyzing whether a model is hallucinating. Because we know of the dangers of models not being able to reason, we're using statistical analysis, let's say some number of standard deviations from the mean.

In the future, wouldn't a model, given reasoning, be able to correct itself—sort of autocorrect itself—and that will be a core feature of future models so that there won't be as many hallucinations? Because the model will recognize when...

Maybe that's too esoteric a question, but will the model be able to reason and understand when a hallucination is occurring? Does the question make sense?

Yes, and the answer is also yes. I think what you described is extremely highly plausible.  You should check—for yeah, it's—I wouldn't rule out that it might already be happening with some of the...

Early reasoning models of today, I don't know, but longer term, why not? Yeah, I mean it's part of, like, Microsoft Word's autocorrect. It's a, you know, it's a core feature. Yeah, I just... I mean, I think calling it 'autocorrect' is really

doing it a disservice. I think you are, when you say 'autocorrect,' you evoke something far grander than autocorrect. But other than that point aside, the answer is yes. Thank you. I loved the ending, mysteriously leaving out...

Do they replace us, or are they, you know, superior? Do they need rights? You know, it's a new species of homo sapien, spawned intelligence, so maybe they need... I mean, uh, I think the RL guy thinks they think, uh, you know, we need rights for these things. I have a UN-related question to

that: How do you create the right incentive mechanisms for humanity to actually create it in a way that gives it the freedoms that we have as homo sapiens? You know, I feel like, in some sense, those are

those are the kind of questions that people should be reflecting on more. But to your question about what incentive structure should we create, I don't feel that I know. I don't feel confident answering questions like this because, uh, it's like you're talking

about creating some kind of top-down structure, a government thing. I don't know. It could be a cryptocurrency, too. Yeah, I mean, there's BitTorrent, you know, those things. I don't feel like I am the right person to comment on cryptocurrency, but, but you know, there is a chance by the

What you're describing will happen; indeed, we will have, in some sense, it's not a bad end result if you have AIS and all they want is to coexist with us and also just to have rights.

That will be fine, but I don't know. I mean, I think things are so incredibly unpredictable, I hesitate to comment, but I encourage the speculation. Thank you. Uh, and uh, yeah, thank you for the talk. It's really awesome.

Hi there. Thank you for the great talk. My name is Shalev Liit from the University of Toronto, working with Sheila. Thanks for all the work you've done. I wanted to ask, do you think LLMs generalize multi-hop reasoning out of distribution?

So, okay, the question assumes that the answer is yes or no, but the question should not be answered with yes or no because what does it mean, out-of-distribution generalization? What does it mean in-distribution, and what does it mean out-of-distribution?

Because it's a test of time. I'll say that long, long ago, before people were using deep learning, they were using things like string matching, n-grams for machine translation. People were using statistical phrase tables. Can you imagine? They had tens of thousands of lines of code of complexity, which was truly unfathomable.

And back then, generalization meant is it literally not in the same phrasing as in the dataset? Now we may say, well, my model achieves this high score on, um, I don't know, math competitions, but maybe the math, maybe some discussion in some forum on the internet was about the same ideas.

And therefore, it's memorized well. Okay, you could say maybe it's in-distribution, maybe it's memorization, but I also think that our standards for what counts as generalization have increased really quite substantially, dramatically, unimaginably. If you keep track, and so I think the answer is to some

degree, probably not as well as human beings. I think it is true that human beings generalize much better, but at the same time, they definitely generalize out of distribution to some degree. I hope it's a useful topological answer. Thank you, and unfortunately we're out of time

for this session. I have a feeling we could go on for the next six hours, uh, but thank you so much, Ilia, for the talk. Thank you. Wonderful. [Applause] [Music]

Ilya Sutskever: "Sequence to sequence learning with neural networks: what a decade"

seremot

주최자와 공동 저자에게 감사를 표합니다.

과거 프레젠테이션의 전후 비교를 표시합니다(NeurIPS 2014).

발표된 작품에 대한 10년 회고전을 기획하여 올바른 측면과 잘못된 측면을 모두 강조합니다.

이 작업에는 텍스트에 대해 학습된 자동 회귀 모델이 사용되었습니다.

대규모 신경망을 활용했습니다.

대규모 데이터 세트에 대해 학습되었습니다.

'딥 러닝 가설'과 그 의미에 대해 논의합니다.

사람이 단 몇 초 만에 달성할 수 있는 작업에 집중하세요.

자동 회귀 모델의 기능과 새로운 기능에 대한 설명입니다.

LSTM에 대한 설명과 연구에서의 역할.

파이프라이닝 병렬화와 속도에 미치는 영향에 대해 논의합니다.

결론 슬라이드에서는 스케일링 가설을 소개했습니다.

장기적인 영향과 연결주의의 이념

연결주의에 대한 지속적인 아이디어와 대규모 신경망에 대한 의미.

인공 뉴런과 생물학적 뉴런의 비교.

공동 작업자 및 그들의 공헌에 대한 감사.

데이터의 유한한 특성으로 인한 사전 학습의 한계.

데이터를 화석 연료와 비교하여 정점과 그 이후의 한계를 제시합니다.

에이전트 및 합성 데이터를 포함한 향후 방향에 대한 추측.

추론 시간 계산과 O1 모델에 대해 설명합니다.

생물학적 비유: 포유류의 뇌 대 신체 크기 스케일링

포유류의 신체와 뇌 크기 사이의 관계를 보여주는 그래프를 제시합니다.

호모 사피엔스에 대해 다른 스케일링 지수를 관찰합니다.

생물학에서 다양한 스케일링의 의미에 대해 논의합니다.

AI 개발에서 다양한 확장의 가능성.

AI의 장기적인 미래에 대한 추측

AI의 놀라운 발전과 그 의미에 대해 논의합니다.

초지능과 기존 AI와의 질적 차이에 집중하세요.

현재 언어 모델과 챗봇의 비교를 통해 신뢰성이 낮음을 강조합니다.

에이전트 시스템과 추론으로 인한 예측 불가능성에 대해 논의합니다.

자기 인식 및 고급 기능을 갖춘 미래 AI 시스템 예측.

Q&A: 생물학적 영감과 AI 모델의 미래

AI에서 탐구할 가치가 있는 다른 생물학적 구조에 대한 질문에 대한 답변입니다.

추론을 통해 환각을 스스로 교정하는 미래 AI 모델의 잠재력에 대해 논의합니다.

추론을 통한 자기 수정이 그럴듯하다는 것을 확인합니다.

고급 AI의 윤리적 함의와 적절한 인센티브 메커니즘의 필요성에 대한 논의.

LLM이 멀티홉 추론을 배포에서 일반화할 수 있는지에 대한 논의입니다.

연사는 AI의 일반화에 대한 정의가 진화하고 있음을 강조합니다.

구형 AI 접근 방식과 최신 딥러닝 모델의 일반화 표준을 비교합니다.

인간은 여전히 현재의 AI 모델보다 더 잘 일반화하지만, AI 모델은 어느 정도 일반화한다는 결론을 내렸습니다.