초지능 AI 안전의 과제

### AI의 미래: AGI, 스케일링 법칙, AI 안전
* 🧠 "AGI는 지적 노동의 대부분을 자동화할 수 있는 컴퓨터 시스템으로, 본질적으로 **사람만큼 똑똑한 컴퓨터**입니다.
* 📈 트랜스포머와 같은 현재 모델의 기능은 규모에 따라 지속적으로 개선되고 있으며, **더 나은 아키텍처가 있더라도 여전히 극한까지 발전할 가능성**이 높습니다.
* 🤖 LSTM이나 트랜스포머와 같이 사용되는 특정 알고리즘은 **적절하게 수정하고 확장**하기만 하면 크게 중요하지 않을 수 있습니다.
* 📏 입력 크기와 성능 사이의 확장 법칙은 강력하지만, **다음 단어 예측 정확도와 같은 단순한 지표**를 넘어서는 돌발 행동을 예측하는 것은 여전히 어렵습니다.
* 😲 일리아에게 확장된 모델의 가장 놀라운 점은 신경망의 초기 기능이 제한적이라는 점을 감안할 때 **신경망이 전혀 작동하지 않는다**는 점입니다.
* ⚠ 초지능 AI를 통한 AI 안전의 주요 과제는 **정렬, 인간의 통제, 아이디어와 조직의 자연스러운 선택**입니다.
* 💡 대규모 언어 모델을 기반으로 구축하는 기업가에게는 **가치 있는 고유 데이터와 향후 모델 기능을 예측**하는 것이 핵심 고려 사항입니다.

OpenAI에서 정의하는 AGI는 지적 노동을 자동화하는 컴퓨터 시스템으로, 일반성과 유능함을 목표로 합니다.

Hi, it's interesting. Let's start with, so what's your definition of AGI? What's your mental picture? Yeah, so AGI. At OpenAI, we have a document which we call the OpenAI Charter, which outlines the goal of OpenAI. And there we offer a definition of AGI. And we say that an AGI is a computer system which can automate the great majority of intellectual labor. That's one useful definition. In some sense, an AGI would be the intuition. There is, it's a computer that's as smart as a person. So you might, for example, have a coworker that's a computer. So that would be a definition of AGI, which I think is intuitively satisfying. The term is a bit ambiguous because AGI, the G means general. So it's a generality that we want, that we care about in the AGI. But it's actually a bit more than generality. We care about generality and competence. Needs to be general in a sense that it

can respond sensibly when you throw things at it, but it needs to be competent so that when you when it does something, you ask it a question or ask you to do something, it will do it. Yeah, I like the sort of very practical definition at the end of the day because it gives you some measurement where you can figure out how close are you. Do you think we have all the ingredients to get to AGI? Um, if not, what's missing kind of in the deack? It's a complicated stack already. Um, a Transformer is really all we need, kind of paying homage to the famous attention paper. Yeah, you know, I won't be overly specific in my answer to this question, but I will say that I think that no. I'll comment on the second part of the question. Is Transformers all we need? And I think that the question is a bit wrong because it implies something binary. It implies Transformers are either good enough or not good enough, but I think it's better to

think about it in terms of tax. Where we have Transformers and they're pretty good. Mhm. Maybe we could have something better that would be maybe more efficient. Or maybe you'll be faster. But we as we know when you make the Transformers large they still become better. They might just become might be becoming better more slowly. So while I am totally sure that it will be possible to improve very significantly on the on the current architectures that we have. Even if we didn't we would be able to go extremely far. Do you think it matters what the algorithm is? So so for example an lstm versus a Transformer just scaled up sufficiently. Maybe that's an efficiency Delta or something like that. But don't we end up in the same same place at the end? So I would say almost entirely yes with a caveat. So there are two caveats Alis. So I'm just thinking of how what level of detail to go here you know maybe I will I will I will skip the details how many people in the audience.

know what an lstm is. Oh, see, it's quite CR around here so I think we're mostly okay. Let's dig, let's dig in then. So I would argue that with a few, if we made a few simple modifications to the lstm, their hidden states are quite small. If you somehow made it larger and then we were to go through the trouble of figuring out how to train them. CU lstms are recurrent neural networks and we kind of forgot about them. We haven't put in the effort to cuz you know how neural training works, you have the hyper parameters. Well, how do you set them? It's like you don't know how do you set your learning rates. If it doesn't learn, can you explain why? And so this kind of work has not been done for lstms so that's why our ability to train them is more reduced but had we done that work so that we were able to train the lstms and we just did some simple things to increase their hidden State size. I think they would be worse.

than Transformers, but we would still be able to go extremely far with them. Also, okay, um, how good is our understanding of scaling laws? Like, if we scale these models up, how confident are you in being able to predict capabilities of these particular models? How good is that science? So, that's a very good question. The answer is so-so. I was hoping for a more definitive answer. Well, go for it. So, so is a very definitive answer. It means we are not great, but we are not absolutely terrible either, but we are not great, definitely not great. So, what the scaling law tells you, it relates, it's a relationship between the inputs that you put into the neural network and some kind of a simple-to-measure performance, simple to evaluate performance measure like your next word prediction accuracy, MH, and that relationship is very strong. But what is challenging is that we don't really care about next word prediction, we care about it indirectly, we care about the other incidental benefits that we get out of it and our and so our. For example, you

all know that if you predict the next word accurately enough, you get all kinds of interesting emerging properties. Those have been quite hard to predict, or at least I'll say I'm not aware of such work. And if anyone is looking for interesting research work pro s to work on, that would be one. I will say I will mention one example, something that we've done at OpenAI in our run-up to GPT-4, where we tried to do a scaling law for a more interesting task, which is predicting accuracy at solving coding problems. We were able to do that accurately, very accurately, and that's a pretty good thing because this is a more tangible metric. It's still an improvement over next step next word prediction accuracy as far as things that are relevant to us. So in other words, it's more relevant to us to know what the coding accuracy is going to be, ability to solve coding problems, compared to just ability to predict and exp word. It still doesn't answer the really important question of

can you predict some emergent behavior that you haven't seen before? Okay, um, speaking of these capabilities that are kind of emerging capabilities, which one surprised you the most as these models scaled? What was the thing where you said, like, well, I'm kind of astonished these models can do this? It's a very difficult question to answer because it's too easy to get used to where things are, so there definitely have been times when I was surprised, but you adapt so fast, it's kind of crazy. I think maybe the big surprise for me is, you know, it may sound a little odd probably to most people in this audience, but the big surprise for me is that neural networks work at all. Because when I was starting my work in this area, they didn't work, or it was like, let's define what it means to work at all. It means they could do, they could work a little bit, but not really, not in any serious way, not in a way that anyone except for the most intense enthusiasts would care.

about and so now we see yeah like those neural nets work so I guess the artificial neuron really is at least somewhat related to the biological neuron or at least that basic assumption has been validated to some degree. What about like an emergent property was the one that sticks out to you like for example I don't know code generation or did you maybe it was different in your mind maybe you just once you saw like hey neural nets can work and they can scale yeah of course all these sort of properties will emerge because you know at the limit point we're building a human brain and humans know how to code and humans know how to reason about tasks and so on um was that did you just expect all of that or did I've definitely been surprised and I'll mention why because the human brain can do those things it's true but does it follow that our training process will produce something similar so so it was definitely very amazing I think yeah seeing seeing the coding ability improve.

quickly that was quite quite a sight to be seen and for coding in particular because you know it went from no one has ever seen a computer code anything at all ever there was a little area of computer science called program synthesis mhm which maybe it was very Niche and it was very Niche because they couldn't have any accomplishments it was a very they had a very difficult experience and then these neural Nets came in and said oh yeah code synthesis like we're going to do we're going to accomplish what you hope were hoping to achieve one day like tomorrow so that was yeah deep learning just just out of curiosity when you write code how much of your code is yours how much of your code is I mean like collaboration but I I I do enjoy I do enjoy it when the neural net writes most of it all right let's let's switch TCT here a little bit um as this models get more and more powerful um it's worthwhile to to also talk about AI safety and uh uh and open

AI has released a document just, uh, just recently. That where you're one of the undersigners, um, uh, Sam has testified in front of Congress. What worries you most about AI safety? Yeah, I can talk about that. So let's take a step back and talk about the state of the world. So, you know, you've had this AI research happening and it was exciting. And now you have the GPT models. And now you all get to play with all the different chatbot and assistants and, you know, bar and chat GPT. And you say, okay, that's pretty cool. It can do things. And indeed, there already are. You can start, perhaps, worrying about the implications of the tools that we have today. And I think that it is a very valid thing to do. But that's not where I allocate my concern. The place where things get really tricky is when you imagine fast forward in some number of years, a decade, let's say, how powerful will AI be? Of course, with this incredible future power of AI, which I think will be difficult to

Imagine, frankly, with an AI this powerful, you could do incredible, amazing things that are perhaps even outside of our dreams. Like, if you can really have a dramatically powerful AI. But the place where things get challenging are directly connected to the power of the AI. It is powerful. It is going to be extremely unbelievable, unbelievably powerful. And it is because of this power that's where the safety issues come up. And I'll mention three. I, I personally see three. You know, when you get so, you alluded to the letter mhm that, uh, we posted at OpenAI a few days ago, actually yesterday, about what we, about some ideas that we think would be good to implement to navigate the challenges of super intelligence. Now, what is super intelligence? Why did we choose to use the term super intelligence? The reason is that super intelligence is meant to convey something that's not just like an AGI. With AGI, we said, well, you have something. It's kind of like a person, kind of like a coworker. Super intelligence is meant to

convey something far more capable than that. When you have such a capability, it's like, can we even imagine how it will be? But without question, it's going to be unbelievably powerful. It could be used to solve incomprehensibly hard problems if it is used well. If we navigate the challenges that super intelligence poses, we could radically improve the quality of life. But the power of super intelligence is so vast, so the concerns, the concern number one, has been expressed a lot, and this is the scientific problem of alignment. You might want to think of it as an analog to nuclear safety. You build a nuclear reactor, you want to get the energy, you need to make sure that it won't melt down even if there's an earthquake, and even if someone tries to, I don't know, smash a truck into it. So this is the super intelligence safety, and it must be addressed in order to contain the vast power of super intelligence. This called the alignment problem, one of the

suggestions that we had in our in the post was an approach that an international organization could do to create various standards at this very high level of capability, and I want to make this other point, you know, about the post and also about um our CEO mman Congressional testimony where he advocated for regulation of AI. The intention is primarily to put rules and standards of various kinds on the very high level of capability. You know, you could maybe start looking at GPT-4, but that's not really what is interesting, what is relevant here, but something which is vastly more powerful than that. When you have a technology that is so powerful, it becomes obvious that you need to do something about this power. That's the first concern, the first challenge to overcome. The second challenge to overcome is that, of course, we are people, we are humans, humans of interests, and if you have super intelligences controlled by people, well, who knows what's going to happen. I do hope that at this point we will have the super intelligence itself try to help us.

solve the challenge in world that it creates. This is not no longer an unreasonable thing to say. Like if you imagine a super intelligence that indeed sees things more deeply than we do, much more deeply to understand reality better than us, we could use it to help us solve the challenges that it creates. Then there is the third challenge, which is the challenge, maybe, of natural selection. You know what the Buddhists say, that change is the only constant. So even if you do have your super intelligences in the world, and they are all we managed to solve alignment, we managed to solve no one wants to use them in very destructive ways, we managed to create a life of unbelievable abundance, which really like not just not just material abundance but health, longevity, like all the things we don't even try dreaming about because they're so obviously impossible. If you've got to this point, then there is the third challenge of natural selection. Things change, you know, you know that natural selection applies to ideas, to organizations, and that's a challenge as.

well maybe the neural link solution of people becoming part AI will be one way we will choose to address this. I don't know, but I would say that this kind of describes my concern. And specifically, just as the concerns are big, if you manage, man, it is so worthwhile to overcome them because then we could create truly unbelievable lives for ourselves that are completely even unimaginable. So it is, it is like a challenge that's really, really worth overcoming. I very much like the idea that there needs to be the sort of threshold above which we really, really should pay attention because, you know, speaking as a German, if it's like European style regulation, often from people that don't really know very much about the field, you can also completely kill innovation, um, which would be a little bit of a pity. But let's change TCT here a little bit. So this is a room mostly filled with entrepreneurs, um, uh, lots of which are actually using tools from open

AI, so just practically speaking, um, what are the main things or the main pieces of advice you would give folks that are building on top of large language models? Like, what is the, let's say, canonical set of, uh, things they should read, they should, uh, think about in using these models? Well, yeah, advice, advice, practical. With a few minutes to spare, few minutes to spare. I'll point out that I am, with the caveat that I am not in similar shoes. I think that two things are value, two things are worth keeping in mind. One is obvious, some kind of special data that cannot be found anywhere else that can be extremely helpful. And I think the second one is to always keep in mind, mind, not just about where things are right now, but where things will be in 2 years, in four years, and try to plan for that. I think those two things are very helpful. The data is helpful today, but even a little bit, kind of trying to get an intuitive sense for yourself of where do you imagine things being, say, in 3

years and how will it affect some of the basic assumptions of what what the product is trying to do. I think that can be a helpful thing. So what's the sort of thing that so when I think about this I I think about oh we used to be in a world with really small context Windows right and then you know I have embeddings. I page things into context Windows like all the classic stuff but maybe that just goes away maybe context Windows become really large or something like that um. I so I'm trying to extrapolate from these sort of past things is that what you mean something like this I think it's wor trying. I'll give you another example yeah like say you're playing with a model and you can see that the model can do something really cool and really maybe amazing if it was reliable but it's so unreliable so you kind of like forget it it's not there's no point using it so that's the kind of thing which can change something which is like if you can for example something which is

unreliable can become reliable enough, and so if you're just kind of experiencing those models, you paying attention to what people are sharing, and you say, Oh, like look at this cool thing, which works once in a while, but if it worked, what would happen. So these kind of thought experiments, I would argue, can help prepare for the kind of near to medium-term future. That's super good advice. I think we are, we are unfortunately at time. We could, we could do this forever. Please join me in, uh, uh, thanking Ilia. [Applause]

Ilya Sutskever | The birth of AGI will subvert everything |AI can help humans but also cause trouble

Welcome to our AI and GPT channel! Here, we are dedicated to exploring and sharing the latest developments in AI, discussing the harmonious coexistence of AI and humans, and contributing to the advancement of AI technology. Our goal is to help humanity achieve the dream of Artificial General Intelligence (AGI) as soon as possible!

If you enjoy our content, make sure to follow us! Additionally, you can become an exclusive fan to enjoy special perks like priority comments and exclusive stickers. We are excited to have you join us and witness the amazing future of AI together!

Follow us, become a pioneer in the AI field, and embrace the endless possibilities of the future with us!


Me&ChatGPT

OpenAI에서 AGI는 대부분의 지적 노동을 자동화할 수 있는 컴퓨터 시스템으로 정의되며, 본질적으로 사람만큼 똑똑합니다.

AGI의 목표는 일반성뿐만 아니라 능력의 유능함입니다.

AGI의 발전 가능성은 트랜스포머에만 의존하지 않으며, 알고리즘은 전반적인 효율성보다 덜 중요하기 때문에 크게 개선될 수 있습니다.

LSTM 수정 및 스케일링 법칙이 신경망에 미치는 영향

LSTM의 숨겨진 상태를 수정하여 잠재적으로 성능을 향상시킬 수 있지만, 훈련 과정과 하이퍼파라미터에 대한 이해는 더 많은 연구가 필요합니다.

현재 LSTM의 훈련 능력은 제한적이지만, 숨겨진 상태 크기를 향상시키면 트랜스포머를 능가하지는 못하더라도 상당한 발전을 이룰 수 있을 것입니다.

신경망 모델의 스케일링 법칙에 대한 이해는 입력과 성능 측정 사이의 강력한 관계로 인해 어느 정도 확신할 수 있지만, 새로운 속성과 부수적인 이점을 예측하는 데는 어려움이 있습니다.

코딩 문제 정확도 예측을 위한 스케일링 법칙에 대한 OpenAI의 연구는 단어 예측 정확도보다 더 관련성이 높은 지표를 제공하면서 유망한 결과를 보여줍니다.

신경망의 작동 능력은 초기에 효과적으로 작동하는 데 어려움을 겪었던 것을 고려하면 매우 놀라운 일이었습니다.

특히 틈새 영역에서 인간의 기대치를 뛰어넘는 코딩 능력의 급속한 향상은 놀라웠습니다.

신경망의 코드 생성 능력은 기존 컴퓨터 과학 분야의 프로그램 합성 능력을 훨씬 뛰어넘음에 따라 발전했습니다.

OpenAI의 샘은 인공지능과 초지능의 미래 파워에 초점을 맞춰 인공지능 안전의 우려스러운 측면에 대해 증언했습니다.

가장 큰 우려는 초지능의 막대하고 잠재적으로 통제할 수 없는 힘으로 인해 심각한 안전 문제가 발생할 수 있다는 것입니다.

가장 큰 우려 중 하나는 초지능의 막강한 힘을 억제하기 위해 해결해야 하는 원자력 안전과 유사한 정렬의 과학적 문제입니다.

초지능 AI를 위한 표준 수립의 중요성

강력한 인공지능에 대한 높은 수준의 표준을 만들어 규제와 규칙의 필요성을 다루는 국제기구가 제안되었습니다.

초지능 AI의 도전 과제에는 인류에 미칠 잠재적 영향과 자연선택을 극복해야 할 필요성이 포함됩니다.

이러한 표준을 수립하는 것은 도전을 극복하고 상상할 수 없는 미래를 만드는 데 매우 중요하지만, 그 과정에서 혁신을 억압하지 않도록 하는 것이 중요합니다.

대규모 언어 모델 구축을 위한 실용적인 조언

다른 곳에서는 쉽게 구할 수 없는 전문 데이터는 대규모 언어 모델을 구축하는 데 매우 유용할 수 있습니다.

현재 상황뿐만 아니라 향후 몇 년 동안 기술이 어떻게 발전할지 예측하고 계획하는 것도 중요합니다.

모델의 잠재적 신뢰성에 주의를 기울이고 사고 실험을 수행하면 단기 및 중기 미래를 대비하는 데 도움이 될 수 있습니다.