Challenges of Superintelligent AI Safety

At OpenAI, the definition of AGI is a computer system automating intellectual labor, aiming for generality and competence.

### The Future of AI: AGI, Scaling Laws, and AI Safety
* 🧠 "AGI is a computer system which can **automate the great majority of intellectual labor**," essentially a computer as smart as a person.
* 📈 The capability of current models like Transformers is continuously improving with scale, and even if there are better architectures, they will likely still go **extremely far**.
* 🤖 The specific algorithm used, such as LSTM or Transformer, may not matter much as long as it's **modified and scaled appropriately**.
* 📏 While scaling laws relating input size to performance are strong, predicting emergent behaviors beyond simple metrics like **next word prediction accuracy** remains challenging.
* 😲 The most surprising aspect of scaled models for Ilia was that **neural networks work at all**, given their limited early capabilities.
* ⚠ The main challenges of AI safety with superintelligent AI are **alignment, human control, and natural selection of ideas and organizations**.
* 💡 For entrepreneurs building on large language models, **valuable unique data** and **anticipating future model capabilities** are key considerations.

Hi, it's interesting. Let's start with, so what's your definition of AGI? What's your mental picture? Yeah, so AGI. At OpenAI, we have a document which we call the OpenAI Charter, which outlines the goal of OpenAI. And there we offer a definition of AGI. And we say that an AGI is a computer system which can automate the great majority of intellectual labor. That's one useful definition. In some sense, an AGI would be the intuition. There is, it's a computer that's as smart as a person. So you might, for example, have a coworker that's a computer. So that would be a definition of AGI, which I think is intuitively satisfying. The term is a bit ambiguous because AGI, the G means general. So it's a generality that we want, that we care about in the AGI. But it's actually a bit more than generality. We care about generality and competence. Needs to be general in a sense that it

can respond sensibly when you throw things at it, but it needs to be competent so that when you when it does something, you ask it a question or ask you to do something, it will do it. Yeah, I like the sort of very practical definition at the end of the day because it gives you some measurement where you can figure out how close are you. Do you think we have all the ingredients to get to AGI? Um, if not, what's missing kind of in the deack? It's a complicated stack already. Um, a Transformer is really all we need, kind of paying homage to the famous attention paper. Yeah, you know, I won't be overly specific in my answer to this question, but I will say that I think that no. I'll comment on the second part of the question. Is Transformers all we need? And I think that the question is a bit wrong because it implies something binary. It implies Transformers are either good enough or not good enough, but I think it's better to

think about it in terms of tax. Where we have Transformers and they're pretty good. Mhm. Maybe we could have something better that would be maybe more efficient. Or maybe you'll be faster. But we as we know when you make the Transformers large they still become better. They might just become might be becoming better more slowly. So while I am totally sure that it will be possible to improve very significantly on the on the current architectures that we have. Even if we didn't we would be able to go extremely far. Do you think it matters what the algorithm is? So so for example an lstm versus a Transformer just scaled up sufficiently. Maybe that's an efficiency Delta or something like that. But don't we end up in the same same place at the end? So I would say almost entirely yes with a caveat. So there are two caveats Alis. So I'm just thinking of how what level of detail to go here you know maybe I will I will I will skip the details how many people in the audience.

know what an lstm is. Oh, see, it's quite CR around here so I think we're mostly okay. Let's dig, let's dig in then. So I would argue that with a few, if we made a few simple modifications to the lstm, their hidden states are quite small. If you somehow made it larger and then we were to go through the trouble of figuring out how to train them. CU lstms are recurrent neural networks and we kind of forgot about them. We haven't put in the effort to cuz you know how neural training works, you have the hyper parameters. Well, how do you set them? It's like you don't know how do you set your learning rates. If it doesn't learn, can you explain why? And so this kind of work has not been done for lstms so that's why our ability to train them is more reduced but had we done that work so that we were able to train the lstms and we just did some simple things to increase their hidden State size. I think they would be worse.

than Transformers, but we would still be able to go extremely far with them. Also, okay, um, how good is our understanding of scaling laws? Like, if we scale these models up, how confident are you in being able to predict capabilities of these particular models? How good is that science? So, that's a very good question. The answer is so-so. I was hoping for a more definitive answer. Well, go for it. So, so is a very definitive answer. It means we are not great, but we are not absolutely terrible either, but we are not great, definitely not great. So, what the scaling law tells you, it relates, it's a relationship between the inputs that you put into the neural network and some kind of a simple-to-measure performance, simple to evaluate performance measure like your next word prediction accuracy, MH, and that relationship is very strong. But what is challenging is that we don't really care about next word prediction, we care about it indirectly, we care about the other incidental benefits that we get out of it and our and so our. For example, you

all know that if you predict the next word accurately enough, you get all kinds of interesting emerging properties. Those have been quite hard to predict, or at least I'll say I'm not aware of such work. And if anyone is looking for interesting research work pro s to work on, that would be one. I will say I will mention one example, something that we've done at OpenAI in our run-up to GPT-4, where we tried to do a scaling law for a more interesting task, which is predicting accuracy at solving coding problems. We were able to do that accurately, very accurately, and that's a pretty good thing because this is a more tangible metric. It's still an improvement over next step next word prediction accuracy as far as things that are relevant to us. So in other words, it's more relevant to us to know what the coding accuracy is going to be, ability to solve coding problems, compared to just ability to predict and exp word. It still doesn't answer the really important question of

can you predict some emergent behavior that you haven't seen before? Okay, um, speaking of these capabilities that are kind of emerging capabilities, which one surprised you the most as these models scaled? What was the thing where you said, like, well, I'm kind of astonished these models can do this? It's a very difficult question to answer because it's too easy to get used to where things are, so there definitely have been times when I was surprised, but you adapt so fast, it's kind of crazy. I think maybe the big surprise for me is, you know, it may sound a little odd probably to most people in this audience, but the big surprise for me is that neural networks work at all. Because when I was starting my work in this area, they didn't work, or it was like, let's define what it means to work at all. It means they could do, they could work a little bit, but not really, not in any serious way, not in a way that anyone except for the most intense enthusiasts would care.

about and so now we see yeah like those neural nets work so I guess the artificial neuron really is at least somewhat related to the biological neuron or at least that basic assumption has been validated to some degree. What about like an emergent property was the one that sticks out to you like for example I don't know code generation or did you maybe it was different in your mind maybe you just once you saw like hey neural nets can work and they can scale yeah of course all these sort of properties will emerge because you know at the limit point we're building a human brain and humans know how to code and humans know how to reason about tasks and so on um was that did you just expect all of that or did I've definitely been surprised and I'll mention why because the human brain can do those things it's true but does it follow that our training process will produce something similar so so it was definitely very amazing I think yeah seeing seeing the coding ability improve.

quickly that was quite quite a sight to be seen and for coding in particular because you know it went from no one has ever seen a computer code anything at all ever there was a little area of computer science called program synthesis mhm which maybe it was very Niche and it was very Niche because they couldn't have any accomplishments it was a very they had a very difficult experience and then these neural Nets came in and said oh yeah code synthesis like we're going to do we're going to accomplish what you hope were hoping to achieve one day like tomorrow so that was yeah deep learning just just out of curiosity when you write code how much of your code is yours how much of your code is I mean like collaboration but I I I do enjoy I do enjoy it when the neural net writes most of it all right let's let's switch TCT here a little bit um as this models get more and more powerful um it's worthwhile to to also talk about AI safety and uh uh and open

AI has released a document just, uh, just recently. That where you're one of the undersigners, um, uh, Sam has testified in front of Congress. What worries you most about AI safety? Yeah, I can talk about that. So let's take a step back and talk about the state of the world. So, you know, you've had this AI research happening and it was exciting. And now you have the GPT models. And now you all get to play with all the different chatbot and assistants and, you know, bar and chat GPT. And you say, okay, that's pretty cool. It can do things. And indeed, there already are. You can start, perhaps, worrying about the implications of the tools that we have today. And I think that it is a very valid thing to do. But that's not where I allocate my concern. The place where things get really tricky is when you imagine fast forward in some number of years, a decade, let's say, how powerful will AI be? Of course, with this incredible future power of AI, which I think will be difficult to

Imagine, frankly, with an AI this powerful, you could do incredible, amazing things that are perhaps even outside of our dreams. Like, if you can really have a dramatically powerful AI. But the place where things get challenging are directly connected to the power of the AI. It is powerful. It is going to be extremely unbelievable, unbelievably powerful. And it is because of this power that's where the safety issues come up. And I'll mention three. I, I personally see three. You know, when you get so, you alluded to the letter mhm that, uh, we posted at OpenAI a few days ago, actually yesterday, about what we, about some ideas that we think would be good to implement to navigate the challenges of super intelligence. Now, what is super intelligence? Why did we choose to use the term super intelligence? The reason is that super intelligence is meant to convey something that's not just like an AGI. With AGI, we said, well, you have something. It's kind of like a person, kind of like a coworker. Super intelligence is meant to

convey something far more capable than that. When you have such a capability, it's like, can we even imagine how it will be? But without question, it's going to be unbelievably powerful. It could be used to solve incomprehensibly hard problems if it is used well. If we navigate the challenges that super intelligence poses, we could radically improve the quality of life. But the power of super intelligence is so vast, so the concerns, the concern number one, has been expressed a lot, and this is the scientific problem of alignment. You might want to think of it as an analog to nuclear safety. You build a nuclear reactor, you want to get the energy, you need to make sure that it won't melt down even if there's an earthquake, and even if someone tries to, I don't know, smash a truck into it. So this is the super intelligence safety, and it must be addressed in order to contain the vast power of super intelligence. This called the alignment problem, one of the

suggestions that we had in our in the post was an approach that an international organization could do to create various standards at this very high level of capability, and I want to make this other point, you know, about the post and also about um our CEO mman Congressional testimony where he advocated for regulation of AI. The intention is primarily to put rules and standards of various kinds on the very high level of capability. You know, you could maybe start looking at GPT-4, but that's not really what is interesting, what is relevant here, but something which is vastly more powerful than that. When you have a technology that is so powerful, it becomes obvious that you need to do something about this power. That's the first concern, the first challenge to overcome. The second challenge to overcome is that, of course, we are people, we are humans, humans of interests, and if you have super intelligences controlled by people, well, who knows what's going to happen. I do hope that at this point we will have the super intelligence itself try to help us.

solve the challenge in world that it creates. This is not no longer an unreasonable thing to say. Like if you imagine a super intelligence that indeed sees things more deeply than we do, much more deeply to understand reality better than us, we could use it to help us solve the challenges that it creates. Then there is the third challenge, which is the challenge, maybe, of natural selection. You know what the Buddhists say, that change is the only constant. So even if you do have your super intelligences in the world, and they are all we managed to solve alignment, we managed to solve no one wants to use them in very destructive ways, we managed to create a life of unbelievable abundance, which really like not just not just material abundance but health, longevity, like all the things we don't even try dreaming about because they're so obviously impossible. If you've got to this point, then there is the third challenge of natural selection. Things change, you know, you know that natural selection applies to ideas, to organizations, and that's a challenge as.

well maybe the neural link solution of people becoming part AI will be one way we will choose to address this. I don't know, but I would say that this kind of describes my concern. And specifically, just as the concerns are big, if you manage, man, it is so worthwhile to overcome them because then we could create truly unbelievable lives for ourselves that are completely even unimaginable. So it is, it is like a challenge that's really, really worth overcoming. I very much like the idea that there needs to be the sort of threshold above which we really, really should pay attention because, you know, speaking as a German, if it's like European style regulation, often from people that don't really know very much about the field, you can also completely kill innovation, um, which would be a little bit of a pity. But let's change TCT here a little bit. So this is a room mostly filled with entrepreneurs, um, uh, lots of which are actually using tools from open

AI, so just practically speaking, um, what are the main things or the main pieces of advice you would give folks that are building on top of large language models? Like, what is the, let's say, canonical set of, uh, things they should read, they should, uh, think about in using these models? Well, yeah, advice, advice, practical. With a few minutes to spare, few minutes to spare. I'll point out that I am, with the caveat that I am not in similar shoes. I think that two things are value, two things are worth keeping in mind. One is obvious, some kind of special data that cannot be found anywhere else that can be extremely helpful. And I think the second one is to always keep in mind, mind, not just about where things are right now, but where things will be in 2 years, in four years, and try to plan for that. I think those two things are very helpful. The data is helpful today, but even a little bit, kind of trying to get an intuitive sense for yourself of where do you imagine things being, say, in 3

years and how will it affect some of the basic assumptions of what what the product is trying to do. I think that can be a helpful thing. So what's the sort of thing that so when I think about this I I think about oh we used to be in a world with really small context Windows right and then you know I have embeddings. I page things into context Windows like all the classic stuff but maybe that just goes away maybe context Windows become really large or something like that um. I so I'm trying to extrapolate from these sort of past things is that what you mean something like this I think it's wor trying. I'll give you another example yeah like say you're playing with a model and you can see that the model can do something really cool and really maybe amazing if it was reliable but it's so unreliable so you kind of like forget it it's not there's no point using it so that's the kind of thing which can change something which is like if you can for example something which is

unreliable can become reliable enough, and so if you're just kind of experiencing those models, you paying attention to what people are sharing, and you say, Oh, like look at this cool thing, which works once in a while, but if it worked, what would happen. So these kind of thought experiments, I would argue, can help prepare for the kind of near to medium-term future. That's super good advice. I think we are, we are unfortunately at time. We could, we could do this forever. Please join me in, uh, uh, thanking Ilia. [Applause]

Ilya Sutskever | The birth of AGI will subvert everything |AI can help humans but also cause trouble

Welcome to our AI and GPT channel! Here, we are dedicated to exploring and sharing the latest developments in AI, discussing the harmonious coexistence of AI and humans, and contributing to the advancement of AI technology. Our goal is to help humanity achieve the dream of Artificial General Intelligence (AGI) as soon as possible!

If you enjoy our content, make sure to follow us! Additionally, you can become an exclusive fan to enjoy special perks like priority comments and exclusive stickers. We are excited to have you join us and witness the amazing future of AI together!

Follow us, become a pioneer in the AI field, and embrace the endless possibilities of the future with us!


Me&ChatGPT

At OpenAI, AGI is defined as a computer system that can automate the majority of intellectual labor, essentially being as smart as a person.

The goal for AGI is not only generality but also competence in its abilities.

The potential for AGI's development is not solely reliant on Transformers and could be significantly improved upon, with the algorithm being less important than its overall efficiency.

The Impact of LSTM Modifications and Scaling Laws on Neural Networks

LSTM's hidden states can be modified to potentially improve their performance, but the training process and understanding of hyperparameters need further exploration.

The current ability to train LSTMs is limited, but enhancing their hidden state size could still allow them to achieve significant progress, though likely not surpassing Transformers.

The understanding of scaling laws for neural network models is only moderately confident, with a strong relationship between inputs and performance measures, but challenges in predicting emerging properties and incidental benefits.

OpenAI's work on scaling laws for predicting coding problem accuracy shows promising results, offering a more relevant metric than word prediction accuracy.

Surprising Capabilities of Neural Networks

Neural networks' ability to work was a big surprise, considering their initial struggle to function effectively.

The rapid improvement in coding abilities, from a niche area to outperforming human expectations, was particularly astonishing.

The neural network's capacity for code generation far exceeded the capabilities of the existing computer science field of program synthesis.

The Concerns About AI Safety and Super Intelligence

Sam, from OpenAI, testified about the worrying aspects of AI safety, focusing on the future power of AI and super intelligence.

The main concern is the immense and potentially uncontrollable power of super intelligence, which could lead to significant safety issues.

One of the primary worries is the scientific problem of alignment, similar to nuclear safety, which needs to be addressed to contain the vast power of super intelligence.

The Importance of Establishing Standards for Super Intelligent AI

An international organization is suggested to create high-level standards for powerful AI, addressing the need for regulation and rules.

The challenges of super intelligent AI include the potential impact on humanity and the need to overcome natural selection.

Establishing these standards is crucial for overcoming challenges and creating an unimaginable future, but it's important to avoid stifling innovation in the process.

Practical Advice for Building on Large Language Models

Specialized data that is not readily available elsewhere can be incredibly valuable for building on large language models.

It is important to consider not only the current state of things but also to anticipate and plan for how the technology will evolve in the next few years.

Paying attention to the potential reliability of the models and conducting thought experiments can help prepare for the near to medium-term future.