AI Progress Plateau: Concerns Over Future Developments

AI progress may be plateauing, raising concerns about the sustainability of investments in model development.

### The Future of AI: Are We Approaching a Performance Plateau?
* 🤖 AI breakthroughs are no longer a question of "**if**" but "**when**," with Google revealing new details about its large language model, **Gemini**, and Claude 3 emerging as a leading AI model.
* 📉 Concerns are growing in Silicon Valley about a potential slowdown in AI progress, questioning whether the assumption that models will **continuously improve** is beginning to **fizzle**.
* 💰 The massive investments by tech giants like Amazon, Google, and Microsoft are now under scrutiny as the **expected returns on their AI spending may not materialize**, leading to speculation about a **peak in AI advancements**.
* 🔍 Reports indicate that the anticipated generational leap with OpenAI's next model, Orion, may not meet expectations, with employees noting that its performance improvements are **less significant than prior versions**.
* 📊 The challenges of scaling laws are becoming evident, as companies may face a "**data wall**," limiting the effectiveness of adding more computational power and data to improve AI models.
* 🧠 Despite the plateauing of large language model improvements, experts believe that innovation can still thrive in the post-training phase, focusing on **refining existing models** and **exploring new applications**.
* 🌐 The next phase of AI development will likely involve a **surge in AI agents** across various industries, potentially leading to a landscape where **AI agents outnumber humans**, fundamentally transforming business operations.
* 📈 As companies prepare to release new models, the sustainability of the AI trade will depend on their ability to **navigate these challenges and redefine the future of artificial intelligence**.

AI breakthroughs have been a question of when, not if. Google unveiled long-awaited new details about its large language model Gemini. Claude 3 is arguably now one of the most powerful AI models out there, if not the most powerful.

Preview, if you will, for us ChatGPT 5. I expect it to be a significant leap forward. But what if that core assumption that models can only keep getting bigger and better is now fizzling? Is there really a slowing in progress because that wasn't

expected? It could spell cracks in the Nvidia bull story. We're increasing GPUs at the same rate, but we're not getting the intelligence improvements out of it. Calling into question the gigantic ramp in spending from

Amazon, Google, Microsoft; a rush for tangible use cases and a killer app. I'm Deirdre Bosa with the TechCheck take: Has AI progress peaked? Call it performance anxiety. The growing concern in Silicon Valley is that AI's rapid

progression is losing steam. We've really slowed down in terms of the amount of improvement.  Reached a ceiling and is now slowing down. In the pure model competition, the question is, when do we start seeing an asymptote to scale?

Hitting walls that even the biggest players, from OpenAI to Google, can't seem to overcome? Progress didn't come cheap. Billions of dollars invested to keep pace, banking on the idea that the returns would be outsized, too.

But no gold rush is guaranteed to last, and early signs of struggle are now bubbling up at major AI players. The first indication that things are turning: the lack of progression between models. I expect that the delta between 5 and 4 will be the

same as between 4 and 3. Each new generation of OpenAI's flagship GPT models, the ones that power ChatGPT, they have been exponentially more advanced than the last in terms of their ability to understand, generate, and reason.

But according to reports, that's not happening anymore. There was talk prior to now that these companies were just going to train on bigger and bigger systems. If it's true that it's topped out, that's not going to happen

anymore. OpenAI has led the pack in terms of advancements; its highly anticipated next model, called Orion, was expected to be a groundbreaking system that would represent a generational leap in bringing us closer to AGI, or artificial

general intelligence. But that initial vision is now being scaled back. Employees who have used or tested Orion told The Information that the increase in quality was far smaller than the jump between GPT three and four, and that they believed Orion isn't reliably better than its predecessor at

handling certain tasks like coding. To put it in perspective, remember ChatGPT came out at the end of 2022. So now it's been, you know, close to two years. And so you had, initially, a huge ramp-up in terms of what

All these new models can do. And what's happening now is you've really trained all these models, and so the performance increases are kind of leveling off. The same thing may be happening at other leading AI developers, the startup Anthropic.

It could be hitting roadblocks to improving its most powerful model, the Opus, quietly removing wording from its website that promised a new version of Opus later this year, and sources telling Bloomberg that the model didn't perform better than previous versions

as much as it should, given the size of the model and how costly it was to build and run. These are startups focused on one thing: the development of large language models with billions of dollars in backing from names like Microsoft and Amazon and venture capital.

But even Google, which has enough cash on hand to buy an entire country, may also be seeing progress plateau. The current generation of LLMs are roughly—a few companies have converged at the top, but I think we're all

working on our next versions too. I think the progress is going to get harder. When I look at 2025, the low-hanging fruit is gone. You know, the curve—the hill is steeper. Its principal AI model, Gemini, is already playing

catch-up to OpenAI and Anthropic. Now, Bloomberg reports, quoting sources, that an upcoming version is not living up to internal expectations. It has to make you think, okay, are we going to go through a period here where we're going to need to digest?

All this hundreds of billions of dollars we've spent on AI over the last couple of years? Especially if revenue forecasts are getting cut or not changing, even though you're increasing spending on AI. The trend has even been confirmed by one of the most

widely respected and pioneering AI researchers, Ilya Sutskever, who co-founded OpenAI and raised a $1 billion seed round for his new AI startup. As you scale up pre-training,

a lot of the low-hanging fruit was plucked. And so it makes sense to me that you're seeing a deceleration in the rate of improvement, but not everyone agrees the rate of progress has peaked. Foundation model pre-training scaling is intact and continuing.

You know, as you know, this is an empirical law, not a fundamental physical law. But the evidence is that it continues to scale. Nothing I've seen in the field is out of

character with what I've seen over the last ten years or leads me to expect that things will slow down. There's no evidence that the scaling laws, as they're called, have begun to stop. They will eventually stop.

But we're not there yet. And even Sam Altman posting simply, "There is no wall." OpenAI and Anthropic did not respond to requests for comment. Google says it's pleased with its progress on Gemini and has

We've seen meaningful performance gains in capabilities like reasoning and coding. Let's get to the why. If progress is, in fact, plateauing, it has to do with scaling laws; the idea that adding more compute power and more data guarantees better models

to an infinite degree. In recent years, Silicon Valley has treated this as religion. One of the properties of machine learning, of course, is that the larger the brain, the more data we can teach it, the smarter it becomes.

We call it the scaling law. There's every evidence that as we scale up the size of the models, the amount of training data, the effectiveness, the quality, the performance of the intelligence improves. In other words, all you need to do is buy more Nvidia GPUs,

find more articles or YouTube videos or research papers to feed the models, and it's guaranteed to get smarter. But recent developments suggest that may be more theory than law. People call them scaling laws; that's a misnomer. Like Moore's Law, it's a misnomer.

Moore's Law, scaling laws—they're not laws of the universe, they're empirical regularities. I am going to bet in favor of them continuing, but I'm not certain of that. The hitch may be data.

It's a key component of that scaling equation, but there's only so much of it in the world. And experts have long speculated that companies would eventually hit what is called the data wall—that is, run out of it.

If we do nothing, and if you know, at scale, we don't continue innovating, we're likely to face similar bottlenecks in data like the ones we see in computational capability and chip production, or power, and

data center build-outs. So AI companies have been turning to so-called synthetic data; data created by AI, fed back into AI, but that could create its own problem. AI is an industry which is garbage in, garbage out.

So if you feed into these models a lot of AI gobbledygook, then the models are just going to spit out more AI gobbledygook. The information reports that Orion was trained in part on AI-generated data produced by other OpenAI models, and that

Google has found duplicates of some data in the sets used to train Gemini. The problem? Low-quality data, low-quality performance. This is what a lot of the research that's focused on synthetic data is focused on. Right. So if you don't do this well, you

don't get much more than you started with. But even if the rate of progress for large language models is plateauing, some argue that the next phase—post-training or inference—will require just as much compute power. Databricks CEO Ali Ghodsi says there's plenty to build

on top of the existing models. I think lots and lots of innovation is still left on the AI side. Maybe those who expected all of the ROI to happen in 2023, 2024, maybe they, you know, they should

readjust their horizons. The place where the industry is squeezing to get progress is shifted from pre-training, which is, you know, lots of internet data, maybe using synthetic data on huge clusters of GPUs, towards post-training and testing

compute, which is more about, you know, small amounts of data but is very high quality and very specific. Feeding data, testing different types of data, adding more compute. That all happens during the pre-training phase when models

are still being built before it's released to the world. So now companies are trying to improve models in the post-training phase. That means making adjustments and tweaks to how it generates responses to try and boost its performance. And it also means a whole new crop of AI models designed to

be smarter in this post-training phase. OpenAI just announced an improved version of their AI model. They say it has better reasoning. This has been reportedly called 'Strawberry', so there's been a lot of buzz around it.

They're called reasoning models, able to think before they answer. And the newest leg in the AI race. We know that thinking is oftentimes more than just one shot, and thinking requires us to maybe develop multiple plans, multiple potential answers from which we choose the best one

from. Just like when we're thinking, we might reflect on the answer before we deliver the answer. Reflection; we might take a problem and break it down into a step-by-step chain of thought. If AI acceleration is tapped out, what's next?

The search for use cases becomes urgent. Just in the last few weeks, there's been a lot of debate. Have we hit a wall with scaling laws? It's actually good to have some skepticism, some debate,

because that, I think, will motivate more innovation. We've barely scratched the surface of what existing models can do. The models are so powerful today, and we haven't really utilized them to anywhere near their full potential

to bring about true business transformation. OpenAI, Anthropic, and Google are making some of the most compelling use cases yet. OpenAI is getting into the search business. Anthropic is unveiling a new AI tool that can analyze your

computer screen and act on your behalf. One of my favorite applications is NotebookLM. It's a Google application that I used extensively—it's just fun.

But the next phase, the development and deployment of AI agents, is expected to be another game changer for users. I think we're going to live in a world with hundreds of millions, or even billions, of different AI

agents—eventually, probably more AI agents than people. I spoke with Nvidia after the call. They said, 'Jim, you better start thinking about how to use the term "agentic"'

Because "agentic" is the term. Benioff's been using it for a while. He's very agent-focused. You can have health agents, banking agents, product agents, ops agents, sales agents, support agents, marketing agents, and customer experience agents.

and analytics agents, finance agents, and HR agents. And it's all built on this Salesforce platform, meaning it's all powered by software. Everybody's talking about when AI is going to kick in for software? It's happening now.

Well, it has to be. It's not a future thing. It's now; it's something the stock market is already taking note of. Software stocks are seeing their biggest outperformance versus semiconductors in years. And it's key for Nvidia,

which has become the most valuable company in the world and has powered broader market gains. It's hard for me to imagine that Nvidia can grow as fast as people are modeling, and I see that probably as a problem at some point when you get into next year and Nvidia

starts shipping Blackwell in volume, which is their latest chip, and then the vendors can say, okay, we're getting what we need, and now we just need to digest all this money that we've spent because it's not scaling as fast as we thought.

In terms of the improvements, the sustainability of the AI trade hinges on this debate. OpenAI, xAI, Meta, Anthropic, and Google—they're all set to release new models over the next 18 months. Their rate of progress, or lack of it, could redefine the

Are AI advancements already slowing down?

Welcome to the official YouTube channel for CNBC. From 'Wall Street' to 'Main Street' to award winning original documentaries and Reality TV series, CNBC has you covered. Experience special sneak peeks of your favorite shows, exclusive video and more.

For real-time news updates from around the world, visit: http://www.cnbc.com

For CNBC's primetime TV lineup, visit http://tv.cnbc.com

For CNBC's International YouTube channel, visit: http://www.youtube.com/cnbcinternational

CNBC

Recent AI breakthroughs from Google (Gemini) and Anthropic (Claude 3) are mentioned.

Questions arise about the future of ChatGPT and whether AI model improvements are slowing down.

Concerns are raised about the massive investments in AI by major tech companies, questioning the return on investment.

The growing concern in Silicon Valley is that AI's rapid progression is losing steam.

The question of whether AI progress is hitting an asymptote in terms of scale is posed.

Evidence of slowing progress in major AI companies

Billions of dollars have been invested, but early signs of struggle are emerging.

The lack of significant progress between newer AI models is noted, particularly concerning OpenAI's GPT models.

OpenAI's Orion model, initially expected to be groundbreaking, is reported to have shown smaller-than-expected improvements.

Anthropic's Opus model also shows signs of hitting roadblocks, with a lack of significant performance improvement compared to previous versions.

Even Google's Gemini model is reported to be falling short of internal expectations.

Debate on the reasons for slowing progress

Ilya Sutskever, co-founder of OpenAI, suggests that the low-hanging fruit in pre-training has been plucked.

Others argue that scaling laws are still intact and that progress will continue.

The debate centers on scaling laws—the idea that adding more compute power and data guarantees better models—and whether this is still true.

The potential limitation of available data is discussed, with the concept of a 'data wall' being mentioned.

The use of synthetic data is discussed as a potential solution, but also as a potential source of problems due to the 'garbage in, garbage out' nature of AI.

Shifting focus to post-training and inference

Some argue that the next phase of AI development will focus on post-training and inference, which will require significant compute power.

The industry is shifting its focus from pre-training to post-training and testing compute.

Companies are now trying to improve models in the post-training phase by making adjustments and tweaks.

New AI models designed to be smarter in the post-training phase are emerging, such as OpenAI's 'Strawberry' model.

Focus on AI use cases and future development

The search for compelling use cases for existing AI models is becoming more urgent.

Examples of use cases from OpenAI, Anthropic, and Google are given, including OpenAI's move into search and Anthropic's new AI tool.

The development and deployment of AI agents are expected to be a game changer.

The impact of AI on software is discussed, with the stock market already taking note.

The future growth of Nvidia is questioned, given the potential for slower-than-expected scaling of AI improvements.