So today actually marks a very historic moment for the AI community, as it's going to be probably regarded as the day

where AGI actually happened. Now, if you guys don't know why this is the case, this is because OpenAI today

released—announced, I guess you could say—their new 03 model, which is the second iteration of their 01 series, the

model that thinks. For a long time now, if you don't understand why this is potentially AGI, this is because the new

system managed to surpass human performance in the ARC benchmark. Now, the reason that the ARC benchmark is such an

important benchmark is because it is resistant to memorization. Sure, so ARC is intended as a kind of IQ test for

machine intelligence, and what makes it different from most benchmarks out there is that it's designed to be resistant to

memorization. So if you look at the way LLMs work, they're basically this, uh, big interpolative memory, and the way you

scale up the capabilities is by trying to cram as much, uh, knowledge and patterns as possible into them. And, uh, by

contrast, ARC does not require a lot of knowledge at all. It's designed to only require what's known as core knowledge,

which is, uh, basic knowledge about things like elementary physics, objectness, counting, that sort of thing—um, the sort

of knowledge that any four-year-old or five-year-old possesses, right? Um, but what's interesting is that each puzzle

in ARC is novel; it's something that you've probably not encountered before, even if you've memorized the entire internet. Now,

if you want to know what ARC actually looks like in terms of this test that humans are so easily able to pass but

you know, these AI systems currently aren't, you can take a look at the current examples right here in this

Video example here. AGI is all about having input examples and output examples. Well, they're good.

Okay, input examples and output examples. Now, the goal is you want to understand the rule of the transformation and guess

it on the output. So, Sam, what do you think is happening in here? Probably putting a dark blue square in the empty

space. See? Yes, that is exactly it. Now, that is really—um—it's easy for humans to intuitively guess what that is.

It's actually surprisingly hard for AI to understand what's going on. Um, what's interesting though is AI has not

been able to get this problem thus far, even though we verified that a panel of humans could actually do it. Now,

the unique part about AR—AGI is every task requires distinct skills. And what I mean by that is we won't ask—

there won't be another task that you need to fill in the corners with blue squares. But we do that on purpose,

and the reason why we do that is because we want to test the model's ability to learn new skills on the fly. We don't

just want it to repeat what it's already memorized. That's the whole point here. Now, Ark AGI version one took

five years to go from 0% to 5% with leading frontier models. However, today I'm very

excited to say that 0.3 has scored a new state-of-the-art score that we have

verified on low compute. For 0.3, it has scored 75.7 on the Ark AGI semi-private holdout set.

Now, this is extremely impressive because this is within the compute requirements that we have for our public leaderboard, and this is the new number

one entry on the Ark AGI public leaderboard. So, congratulations to that. Now, I know those of you outside the AI community might not think this is

a big deal, but this is a really big deal because this is something that we've been trying to solve for, I think, around 5

years now. This is a benchmark that many would have heralded to be the golden standard for AI and would, of course, mark

the first time that we've actually managed to get to a system that can actually outperform humans at a task

that traditionally AI systems would particularly fail at. Now, what was interesting was that they had two versions: we had 003 with low tuning

and we had 003 with high tuning. So, the 003 with low tuning is the low reasoning effort, and this is the model that

operates with minimum computational effort, which is optimized for speed and cost efficiency. And this one is suitable

for simpler tasks where deep reasoning is not required; so, you know, basic coding, straightforward tasks. And then, of course,

you have the high-tuned one, which is where the model takes more time and resources to analyze and solve problems,

and this is optimized for performance on complex tasks requiring deeper reasoning or multi-step problem-solving. And what we can see here is that when we actually

you know, tune the model and make it think for longer, we can see that we actually manage to surpass where humans

currently are. Now, what's crazy about this is that the people that have created the ARC AGI Benchmark said that

the performance on ARC AGI highlights a genuine breakthrough in novelty adaptation. This is not incremental progress; we are in new territory. So they

start to ask, 'Is this AGI?' 003 still fails on some very easy tasks, indicating fundamental differences with human

intelligence. And the guy that you saw at the beginning, François Chollet, actually spoke about how he doesn't believe that

this is exactly AGI, but it does represent a big milestone on the way to what's AGI. He says there's still a fair

Number of easy ARC AGI; one talks to 003 can't solve, and we have early indications that ARC AI 2 will remain

extremely challenging.  So he's stating that it shows that it's feasible to create unsaturated, interesting benchmarks

that are easy for humans yet impossible for AI without involving specialist knowledge, and he states now

which is, you know, some people could argue that he's moved the goal; that we will have AGI when creating such evals

becomes outright impossible. But this is a little bit in contrast to what he said earlier this year. Take a look at what he

said six months ago about the benchmark surpassing 80%, which it did today.  Turning the question around to you, so suppose that

it's the case that in a year, a multimodal model can solve ARC, let's say get 80%, whatever the average human would

get, then AGI? Quite possibly, yes. I think if you—if you start—so honestly, what I would like to see is a large language model type

model solving ARC at like 80%, but after having only been trained on core knowledge-related stuff. Now, one of the

limitations of the model is actually the compute cost. So you can see right here that he says, "Does this mean that the ARC

prize competition is beaten?" He says, "No." The ARC prize competition targets the fully private data set, which is a

different and somewhat harder evaluation, but the ARC prize is, of course, the one where your solutions must run with a

fixed amount of compute, which is about 10 cents per task. And the reason that that is really interesting is because I'm not

sure you guys may have seen this, but if we actually take a look down the bottom here, you can see that 003 high-tuned the

amount of compute that is being put into the model. There means that this model cost around $11,000 per task, which is

Pretty pricey if you're trying to use that AI for anything at all. I think these models that search over many

different solutions are going to be really, really expensive, and we can see that reflected here with this one being

over $1,000 per task, which is ridiculously expensive when you think about using it to perform any kind of

task. Of course, as we've seen with AI, costs will eventually come down. So the fact that they've managed to get the

benchmark is just the very best thing. We can use the similarities between how technology was pretty bulky in the early

days—you know, like the big TVs, the really bulky phones—but now with technology, you find ways to become more

and more efficient, and eventually you do things faster and, of course, cheaper. So it's quite likely that this will happen

in AI, too. And what I do find crazy about this is that, you know, two years ago, he said that the ARC AGI benchmark fully

solved is not going to be within the next 8 years, and he says 70% hopefully

less than 8 years, perhaps four or five. Of course, you can see AI managing to speed past most people's predictions. We

also got the fact that it did very well on the SWE benchmark, which is, of course, a very hard software engineering benchmark, and

I'm guessing that if you are a software engineer, this is probably not the best news for you. But I'm sure that now with

a bunch of people who are now coding, there's probably a lot more demand for software engineers that actually

understand the code that's actually being written here. But this is actually rather interesting because one thing that I realized when, you know, doing this

video was the fact that this is actually O(2) and not O(3). Just want to go on a

Tangent quickly here because, whilst I was reading this breakthrough about 03, one thing I needed to

remember is that this is actually the second iteration of the model. I think some people might be thinking that 03 is

opening a third iteration, but 02 is simply being skipped because there's a conflict with O2, which is a British

mobile service provider. So, the fact that this is only the second iteration of the model does go to show that

potentially 03 or even 04 is going to be quite a large jump and might reach benchmark saturation. Now, we also look at

the math benchmarks and the PhD-level science benchmarks; we can see that there's a decent improvement there as well. I

do think, though, that this is sort of reaching the benchmark saturation area because we can see this one on

competition math is around 96.7%, and this one is 87.7%. And, like I said before,

one of the things that most people are starting to say is that, like, okay, AI has slowed down because it's no longer

increasing at the same rates as it was before. What we have to understand is that, as these benchmarks get to 95% plus

or around that 89%, the incremental gains are going to be harder and harder to reach because, number one, you only have

10% left to get, and number two, it's quite likely that 3 to 5% of all the

questions are contaminated, meaning that there are errors in those questions anyway. Which means that 100% is simply

not possible on certain benchmarks, which is why they decided to create the Frontier Math benchmark. Now, at the time

when the benchmark was released, which I think was around two or three months ago, these were the current models that could

Only 2% on these questions. You can see at the lead was Gemini 1.5, Pro, Claude 3,

Sonic. And you can see 01 preview and 01 mini were there, but for those of you

that haven't realized just how good 02/03 is, this is a model that gets 25%.

So this is something that is really, really incredible when it comes to research, math. And you have to understand

that this kind of math is super, super difficult, and all of those questions are completely novel. So the previous

benchmarks that we recently had aren't truly showing the capabilities. I mean, think about it like this: okay, when we

take a look at these benchmarks, you can see that, okay, you would say maybe the model's gotten, you know, 10% better

overall, which is just not true. Because if we actually look at the really hard benchmarks where it's really solving

unseen math problems, we can see that there is a 20-times improvement over the current state-of-the-art, which is just

absolutely incredible. So I think this is probably the very important image that people need to realize because you can't

OpenAI Just Revealed They ACHIEVED AGI (OpenAI o3 Explained)

From the latest research and developments in machine learning and natural language processing, to practical applications and the ethical considerations surrounding AI, this channel is your go-to source for all things related to this exciting field. Subscribe now to stay up-to-date on the cutting-edge of AI and join the conversation about the future of intelligence.

contact@theaigrid.com (For Business Enquiries) 


OpenAI Just Revealed They ACHIEVED AGI (OpenAI o3 Explained)

OpenAI Just Revealed They ACHIEVED AGI (OpenAI o3 Explained)

Scripts