OpenAI Researcher Departs: Concerns Over AGI Readiness

Miles Brundage believes neither OpenAI nor the world is ready for AGI, emphasizing the need for more independent research and policy advocacy.

### The Departure of Miles Brundage: Insights on AGI Readiness and Future Directions
* 👨 Miles Brundage, a top-level researcher at OpenAI for six years, has announced his departure, emphasizing that neither OpenAI nor the world is currently **ready for AGI**.
* 📝 In his blog post, Brundage reflects on his time at OpenAI, highlighting his **desire for greater independence** and the opportunity to work on **broader AI industry issues** outside the organization.
* 📉 He notes that as OpenAI transitioned to a for-profit model, the focus shifted towards **product development**, limiting opportunities for **research and publication**, which he finds **concerning**.
* 🔍 Brundage introduces the concept of "**AGI Readiness**," which he believes involves both OpenAI's capability to **manage powerful AI** and society's preparedness to **handle its implications**.
* 📊 He rates various aspects of AI governance, giving low scores for societal resilience (F) and legitimacy (D+), underscoring the **urgent need for improved frameworks and public understanding of AI risks**.
* ⚖ He argues that regulation must evolve alongside technological advancements, cautioning that **current policies may not effectively address the rapid changes in AI capabilities and their societal impacts**.
* 💡 Brundage plans to start or join a nonprofit focused on AI policy and advocacy, seeking to **influence AI development positively from the outside** and foster **more open discussions on the benefits and risks of AI**.
* 🌍 He concludes by stressing that while the potential for AI to **enhance human life** is significant, it requires **deliberate actions from various stakeholders** to ensure **equitable benefits and effective risk management**.

neither OpenAI nor any other frontier lab, and the world is also not ready for AGI. That is a quote from a departing top-level researcher at OpenAI, Miles Brundage, who has worked at OpenAI for 6 years. He gave his notice, is leaving, and wrote a fascinating blog post about his time at OpenAI, what to expect from OpenAI, and how we're not ready for AGI currently. So, I'm going to break down his whole blog post. I read it in full, I highlighted the most interesting bits, let's read it together. So first, this is Miles Brundage. He is head of policy research at OpenAI. He has been there for over 6 years, which is a long time for that company because they've been growing so rapidly. And before that, he was an AI policy research fellow at Oxford, where he studied. So, if anybody is knowledgeable about AI policy and safety, it is Miles. So, he starts by saying, Around 12 years ago, I decided to devote my life to something in the rough.

vicinity of OpenAI's mission ensuring AGI benefits all of humanity, and that is OpenAI's mission. Whether you believe it or not, whether you believe in open source or not, that's their mission, and that is what Miles dedicated his life to. So, leaving what is essentially my dream job advising OpenAI executives and the OpenAI board on AGI readiness was not an easy decision. So, he put together a TL;DR for the entire post. I'll read it quickly, but it's really the interesting bits in the details that I want to go over with you. So, first, he wants to spend more time working on issues that cut across the whole industry, the whole AI industry, and to have more freedom to publish, which he'll talk about why he doesn't feel like he has enough freedom to publish, and to be more independent, which is really the theme of this entire post. He's going to be starting a nonprofit or joining an existing one, which is interesting because OpenAI was a nonprofit. That is maybe one of the

many reasons he decided to join them, and of course, we now know that they're converting into a for-profit. He still says OpenAI is a very exciting place to work, and if you do want to work there, that's great, and it's probably a good fit if you like that culture. Again, I'll touch on that in a little bit. So first, who are you, and what did you do at OpenAI? So he's going to talk about that. So first, he's a researcher and manager at OpenAI. He's been there for six years. He started as a research scientist on the policy team, then he became head of policy research, and I'm currently adviser for AGI Readiness. Why are you leaving? I decided that I want to impact and influence AI's development from outside the industry rather than inside, and that again, the independence theme you will see run throughout this entire post. And so why specifically is he leaving? He details it here. The opportunity costs have become very high. I don't have time to work on various research topics that

I think are important, and that seems to be a theme. As OpenAI blew up, they were making a ton of money. They started to convert into a for-profit, and they became focused on shipping products that made money rather than experimentation and research. And I'm sure they're doing tons of that, but from all of the folks who have departed and the notes that they've left upon departure, it seems that is the theme. They are focused on shipping products that make them money. To be clear, while I wouldn't say I've always agreed with OpenAI's stance on publication review, I do think it's reasonable for there to be some publishing constraints in industry. And I have helped write several iterations of OpenAI policies, but for me, the constraints have become too much. So he's saying the research that he has done and the rest of the team have done, they just don't get to publish. And that is kind of why they're called closed AI now. And I'm guessing that's also why a lot of the researchers there have been leaving.

recently. They joined when the company was a research lab, when it was an open-source company, when it was an open company, a nonprofit, and then all of a sudden they clamped it down. They're making money, they're not releasing research papers, they're not talking about the details of the research that went into building 01 or any of their other innovations as of late. And he also wants to be less biased, again just an extension of wanting to be independent. It is difficult to be impartial about an organization when you are part of it. He seemingly wants to be able to criticize OpenAI, not in a bad way; every company deserves criticism, but he wants to be able to do so without people pointing to him and saying, Well hey, don't you work there? Aren't you biased? So that's probably what he's thinking here. And he also just states, Working at OpenAI affects how people perceive my statements as well as those from others in the industry. He also says he's basically accomplished everything he set

out to do when he joined OpenAI, and now he wants to focus on two types of AGI Readiness. I've begun to think more explicitly about two kinds of AGI Readiness: OpenAI Readiness to steward increasingly powerful AI capabilities and the world's readiness to effectively manage those capabilities. And based on the very first quote in this video, you can tell he does not think we are there. And he says he told executives and the board a fair amount about what I think OpenAI needs to do and what the gaps are. And on the latter, I think I can be more effective externally. So, he's already been talking to the board about what he thinks OpenAI's gaps are in terms of stewardship of cutting-edge AI. And then he also wants to be able to talk about it more publicly from the outside. Now, of course, is a trade-off. Once you're on the outside, you don't have as much information, which he'll touch on. But from the outside, you can also be much more critical. And once

again let's read the most stunning quote in the entire post. So how are open AI in the world doing on AGI Readiness? In short, neither open AI nor any other Frontier lab is ready, and the world is also not ready. And in this quote, he links to a post that he published a while ago, probably about a year ago, a little over a year, not ready. So here is his post scoring Humanity's progress on AI governance. And I'm not going to go too deep into this paper; you can find the link in the post that I'll share in the description below. But really quickly, as of a year ago, I want to give you the scores that he gave on each of the major pillars in kind of scoring AGI Readiness. So first, shared understanding, understanding the upsides and downsides of AI. He gives the grade of a B minus. The general public agrees on the risk being significant and supports regulation. Technical tooling, a fair amount of people are working on this, but many more people should be working on it.

regulatory infrastructure C+. As of the past few months, policymakers have started paying serious attention to this issue. Legitimacy D+. There's been relatively little effort on this to date. Open AI has invested a lot in it at the moment, and I am excited for that to continue, but I am setting a high bar for what counts as success here. Societal resilience F. An F, a failure. Again, this was a year ago, but probably not much has changed in this category. We already have deep fakes, TTS claims, etc., running amok, and there doesn't yet seem to be an appreciation of the fact that Providence, watermarking, etc., are all whack-a-mole type situations with regards to both modalities and providers. So, he thinks the problems are really bad, and we are not ready as a society to deal with them. Differential technological development D+. Basically, everything good happening with AI is happening because of the market, meaning if the money's not there, it's not being done. Safety. The money's

not as much there now. Back to the article, and he doesn't even think that this is a controversial statement. The statement that we're not ready for AGI. To be clear, I don't think this is a controversial statement among OpenAI leadership, and notably, that's a different question from whether the company and the world are on track to be ready at the relevant time. So, although we're not ready today, he says that the question of whether we're ready today is very different, a completely different question from whether we will be ready when it is absolutely required to be ready. AGI. He also talks about AGI, the definition of AGI, how there's really no clear definition. He also points to the levels of AGI framework, which he was influential in developing. And quickly, I'll just go over what those different levels are. So, level one, conversational AI. We have that. That's ChatGPT. Level two, reasoning AI. That's kind of where we're at right now with the 01 model. I'd even say some of the frontier models are

there as well, but really when we have test time, inference time, compute, and we're able to give the models the ability to think long-term and reason long-term, that is really level two, level three Aon autonomous AI agents that can operate completely autonomously. We're starting to have a little bit of that, but we're still far away from True level three autonomy, level four innovating AI. So, this is AI that can develop and iterate on itself automatically. This is the point at which, if you remember Leopold Ashen Brenner's situational awareness paper, this is the requirement for the inflection to get the intelligence explosion. And then level five organizational AI, this final stage of super AI, involves artificial intelligence that is capable of performing the work of an entire organization, so a company, a society. That is what they're talking about, essentially AI that can replicate what we do as a group of people. So, back again to the article, should people work at open AI? Given everything he said so far, working at open AI is one of the most

impactful things that most people could hope to do. So, in many cases, yes, so he still believes OpenAI is a phenomenal organization to work at. He says if you have the right skills, the right passion, and kind of aligned with the mission of OpenAI, that is still a great place to work. But maybe he doesn't think that everybody at OpenAI is taking their role as seriously as they need to. Not that they're not working hard enough or doing the important things, but understanding how impactful their work truly is. Look at this statement: anyone working at OpenAI should take seriously the fact that their actions and statements contribute to the culture of the organization and may create positive or negative path dependencies as the organization begins to steward extremely advanced capabilities. So, the gist is he thinks he can do much more outside of OpenAI. But what is he going to be doing? And so, that's what he's going to talk about next. What are you doing next? He plans to start a new nonprofit or join an existing one and will work on AI policy.

research and advocacy. I will probably do some mix of research and advocacy, but the details and ratios are TBD. And so, he's just starting to think about what the next steps are. He even talks about wanting to find co-founders or collaborators to work with. So, if you're interested, he drops a form at the bottom of this post where people can get in touch with him and really ask if they can collaborate with him. So, definitely check it out if you're interested and the right type of person. Next, what do you think about AI policy? And he had a lot of important things to say here. As someone who has worked in this field for longer than most, it has been very sad to see the increasing polarization along the lines of whether people focus on one side of the cost-benefit ledger or the other or have different risk priorities. So, what does he mean by that? He's basically saying the accelerationist versus the deceleration has become such a binary opinion that it really doesn't make any sense. You should both see the

benefits of AI and understand that it could be incredible but also temper that with the risks that are apparent. And I try to do that, you know, I tend to be very optimistic, but at the same time, I try to add some realism in there that yes, things could go wrong. My view is that there is a lot to worry about and a lot to be excited about. We don't have to choose one thing to care about, and we should find common ground where it exists. So, I'm going to call a couple of people out. You shouldn't be like Gary Marcus as much, where you're just saying everything negative about AI all the time and pointing out every little folly, every little problem. And you probably shouldn't be like Beth Jizos, who is just gung-ho completely, Let's accelerate everything all the time. Now, if you were to ask me where I fall on that spectrum, I'm definitely much more of an accelerationist because I'm an optimist. I think that these tools, these technologies, these innovations are going

to help humanity greatly, like incredibly, and that's why I am spending so much of my time dedicated to educating the world about AI and driving my personal mission and my company's mission of bringing the value of AI to the most amount of people possible. He goes on to say, I think AI and AGI benefiting all of humanity is not automatic and requires deliberate choices to be made by decision-makers and governments, nonprofits, civil society, and industry, and this needs to be informed by robust public discussion. Notably, this is true not just for risk mitigation but also for ensuring equitable distribution of the benefits, as is the case with, for example, electricity and modern medicine as well. So, he's saying even if we get all of the great benefits of AI and we mitigate all of the risks of AI, we still need to make sure that the benefits are distributed equitably, which of course makes sense, especially as the frontier AI becomes more expensive. And he actually has some thoughts about that, which I'm going to touch on, and I don't necessarily agree.

but he's definitely more knowledgeable than I am. Let's read and let's discuss. He goes on to say that AI capabilities are improving very quickly, and policymakers need to act more urgently. He does not think the government is acting quickly enough, and he gives a clear example of that. And so, I'm going to skip down in the page a bit, then I'm going to go back up, but what he says here was fascinating to me and really made a lot of sense. Now, if you remember just a few weeks ago, the California AI bill was vetoed by Governor Gavin Newsom, and I was pretty happy about that. But why? Why did I not think that regulation was right? Well, without getting too deep into the details, they were regulating based on fixed numbers. For example, if you have above this many parameters in your model, or if you use this much compute to train your model, you need to go register, and you need to do all this stuff. So, that doesn't really make sense because the innovation and the technology are evolving so quickly that it almost

doesn't matter as much anymore, and the perfect example is inference time compute. Imagine this: we had this benchmark where if you exceeded the number of parameters, if you exceeded the amount of compute necessary to train a model, you'd have to go register. But then all of a sudden, we have test time compute. And so you can take a smaller model and just scale up the test time compute, and all of a sudden, you would have an extremely capable model without passing those original benchmarks that we just talked about. So in that case, the innovation basically negated the policy, and that is what he talks about here. To the extent that there is serious analysis of compute governance happening in the academic literature, it generally lags behind developments in the industry by a fair amount. To those within Frontier AI companies, it has become increasingly clear in recent years that scaling up inference, not just training, can enable higher performance. So fascinating, a few things there. One, obviously, he is extremely bullish on test time compute. I am as well. I don't think we've necessarily seen what test time compute

is really capable of. But we do have this completely new dimension in which we can scale up not just training, not just data, but the actual post-training inference time compute. And so again, that really would have completely negated any regulation that was based on training time. He goes on to talk about regulation. For example, Congress should robustly fund the US AI Safety Institute so that the government has more capacity to think clearly about AI policy, as well as the Bureau of Industry and Security, so that someone in the government will have some idea of what happens to advanced AI chips after they are exported. He goes on to say, Given that, that is not much time to set up entirely new institutions. I'm particularly interested in opportunities for action under existing legal authorities, as well as shaping the implementation of already approved legislation such as the EU AI Act. So I hope if he truly believes in the EU AI Act, which really has hamstrung the EU in terms of AI innovation, he's really thinking about changing it drastically because it has not worked so far at

least from my perspective, and it is obvious that he really believes government needs to come in and regulate, and his definition of regulation is not really clear. Based on this, I'm sure he has a lot of documentation, a lot of posts about it. I haven't read much of those yet, but he definitely believes that regulation is needed right now. He continues by saying companies and governments will not necessarily give AI safety and security the attention it deserves by default. And why is that? Well, because there's no money in it. If there's money in selling inference and there's money in shipping products that help enterprise and consumers, that is where the money is going to go. But the safety, if there's not a lot of money in it, that means that it's going to be hard to fund. It's going to be hard to get the eyeballs and the researchers in the room necessary to really do what they need to do. Misalignment between private and societal interests, which regulation can help reduce. There are also difficulties

around credible commitments to and verification of safety levels, which further incentivize corner-cutting. People assume others are going to cut corners to gain an advantage and can't tell what the ground truth is or think they will change their minds later. I'm excited to understand how companies can credibly demonstrate safety while protecting valuable and potentially misused IP. So basically, how do we show off that a model is safe or some piece of research is safe without giving away the secrets of that research? That is an incredibly hard problem. The difficulty of demonstrating compliance without compromising sensitive information is a major barrier to arms control agreements, which require innovation to address. Now let's move on to the economic impacts of AI. He continues with, I think it's likely that in the coming years, not decades, years, AI could enable sufficient economic growth that an early retirement at a high standard of living is easily achievable. That is kind of nuts to think about. In years, not decades, AI is going to increase productivity and value so much that people are going to be able to

retire at a much younger age at a high standard of living. That is really cool to think about. Before that, there will likely be a period in which it is easier to automate tasks that can be done remotely. Great in the near term. I worry a lot about AI disrupting opportunities for people who desperately want work, but I think it's simultaneously true that humanity should eventually remove the obligation to work for a living. Wow, it's crazy to think about. And of course, I've talked about this on the channel before. I just talked about this with Sat, the World Coin founder. And by the way, if you haven't seen that interview, I really enjoyed it. Check it out. And it is just so cool to think about that people might not have to work in the future. People might not have to work. And there's a lot of implications with that statement, which I'm not going to get into right now, but it's very interesting. And that doing so is one of the strongest arguments for building AI and AGI in the

first place doing so being allowing people to not work. He also goes on to talk about, and this is where I start to disagree with him, the growing gap between paid and free AI. Now, before I get into it, I want to remind you of a quote. This is from Sam Altman just a couple of months ago when they released GPT-4. Too cheap to meter. So, the CEO of OpenAI believes we are heading to a point where intelligence is going to be essentially free. But that is not what Miles thinks, or at least he thinks that the paid AI is going to be so substantially better than the free AI that it actually doesn't make sense to use the free AI. Let's read a little bit of what he said. It's also likely that there will by default be a growing gap between free and paid AI capabilities. There was a brief period where these were exactly equivalent other than rate limits, namely when GPT-4 was the best available paid model and also available to users for free. But that era has passed and will

soon be a distant memory as companies compete to leverage test time compute for those willing to pay more. Okay, so a few things to unpack here. So first of all, if we're just looking at the raw intelligence of models, I actually think that intelligence is heading to zero, especially with incredible models coming from Meta that are free, they're open source. But the test time compute is actually really interesting, and I hadn't thought of that. If test time compute is the new lever for reaching higher levels of intelligence, then that is just going to cost money. There is no amount of open-source models that can reach test time compute. You need the GPUs, you need the hardware, you need to actually run the compute, and that is always going to cost money because there is the built-in cost of electricity and the hardware. So I hadn't really thought about that. If he is right that test time compute is the future of AI, then we have to pay for it. He goes on to talk about the AI for good landscape, seems very well-intentioned but inefficient to me at the moment.

largely due to insufficient technical expertise in civil society and governments. Basically, if there's no money there, you're not going to have the best people there. Then he goes on to talk about compute governance, basically hardware GPUs. How do you govern? How do you regulate GPUs? So, compared to software, data, and talent, compute-shooting hardware has unique properties that make it an important focal point for AI policy. It is detectable, excludable, and quantifiable, and is produced via an extremely concentrated supply chain. So, hardware has a very different profile from software, from the models, from the weights, and it's because it is just so much more measurable. This makes it worrying that the part of the US government responsible for overseeing what happens when that compute is shipped overseas is severely understaffed and underfunded, and that more generally, there is little serious policy discussion of what the endgame is here. The government lags behind developments in industry by a fair amount, and we already touched on this, but I'll read it again to those

within Frontier AI companies, it has become increasingly clear in recent years that scaling up inference, not just training, can enable higher performance. He then goes on to talk about a grand AI strategy, and let me just briefly talk about the different pillars of what is included in that. What do we think about when we talk about our grand plan for AI? How can we resolve or alleviate the trade-off between the benefits of decentralized AI development and the downsides of it? Now, he doesn't really touch on open source or decentralized AI much in this entire article. This is really one of the only places he does so. I'm glad to see it here, though. What policy actions by companies and countries make sense in a world where a treaty or grand bargain on AI is possible? Or, if it's impossible, what are the various options on the Spectrum from nationalization or internationalization of AI research to a laissez-faire approach that are worth considering? What does a serious dialogue about the ideal pace of AI progress look like Beyond EAC vs. pause AI in domains like

energy and climate policy. There are contested quantitative frameworks for distilling the overall picture of impacts into a single number. So, he finishes up with how will he remain independent post OpenAI. So, apparently OpenAI has offered him funding, API credits, and access to early models as an external researcher. He hasn't accepted the offer yet, but I actually find that really nice, and I applaud OpenAI for doing that, and I think they're more heading in that right direction. And another example of that is being able to be critical about OpenAI from within OpenAI. Now, obviously, you are beholden to them because they hold your paycheck, but they've done a really good job after you leave of essentially negating the NDAs that they had in place. So, that's it. That's the end of his post. At the bottom, again, you can find a link to a form where, if you're interested in working with him, you can fill that out. And I found this interesting. I want to know your thoughts. Let me know your thoughts.

In the comments below, if you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

OpenAI Researcher Leaves and Shares Chilling Message "We're Not Ready For AGI"

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Miles Brundage, a top-level researcher at OpenAI for six years, is leaving the company to pursue work on AI issues from outside the industry.

He cites high opportunity costs and a shift in OpenAI's focus towards profit-making products as reasons for his departure.

Brundage expresses a desire for more independence and freedom to publish his research, which he feels has been constrained at OpenAI.

Concerns About OpenAI's Shift and AGI Readiness

The speaker highlights OpenAI's transition from an open, nonprofit organization to a profit-driven company, leading to a lack of transparency in their research.

They express a desire to be less biased and critical of OpenAI from an external perspective, aiming to address the organization's gaps in stewarding powerful AI capabilities.

The speaker's assessment of AGI readiness reveals a general lack of preparedness from both OpenAI and the world, with low scores in key areas such as regulatory infrastructure, legitimacy, and societal resilience.

The OpenAI leadership believes we are not ready for AGI, and the question of readiness is different from the timing of when we will need to be ready.

The levels of AGI framework, developed by the speaker, outlines five levels: conversational AI, reasoning AI, autonomous AI agents, innovating AI, and organizational AI.

Despite the impact of working at OpenAI, the speaker feels they can make a greater difference outside the organization and plans to start or join a nonprofit focused on AI policy.

The speaker is exploring a mix of research and advocacy in AI policy and is looking for co-founders or collaborators.

They emphasize the need to balance the benefits and risks of AI, criticizing the binary polarization between accelerationists and decelerationists.

The speaker advocates for deliberate decision-making and robust public discussion to ensure equitable distribution of AI's benefits, highlighting the importance of addressing both risk mitigation and equitable access.

AI capabilities are rapidly improving, and the government's response is lagging, as seen in the recent veto of the California AI bill.

The bill's fixed-number regulations are ineffective due to the fast-evolving technology, with test time compute scaling up to enable higher model performance.

The speaker advocates for robust funding of the US AI Safety Institute and other agencies to better address AI policy and governance.

The Need for AI Regulation and Its Economic Impact

The speaker emphasizes the necessity of government regulation for AI safety and security, driven by the misalignment of private and societal interests.

They highlight the challenge of demonstrating safety without compromising sensitive information and the potential for AI to enable early retirement at a high standard of living within a few years.

The speaker expresses concern about AI disrupting job opportunities while also suggesting that humanity should eventually remove the obligation to work for a living.

The CEO of OpenAI, Sam Altman, believes that intelligence will become essentially free, while Miles argues that paid AI will significantly outperform free AI.

Test time compute is identified as a key factor in achieving higher levels of intelligence, and its cost will always be a barrier.

The unique properties of hardware compute make it a crucial focus for AI policy, but the US government's oversight and policy discussion are lagging behind industry developments.

Scaling Inference in AI and a Grand Strategy

Scaling up inference has become crucial for higher AI performance, alongside training.

The article discusses the trade-offs of decentralized AI development and explores policy actions for a potential AI treaty.

The author reflects on maintaining independence post-OpenAI and appreciates the company's efforts to support external researchers and negate NDAs after departure.