OpenAI just released o3; this is AGI. And before you flame me in the comments, let me explain why I'm saying this: OpenAI is saying this, and the ARC prize is basically showing it. When you see the results, you will be stunned. It is far better than anything

else out there right now. So, for the last day of this event, um, we thought it would be fun to go from one Frontier Model to our next Frontier Model. Today we're going to talk about that Frontier Model, um, which you would think logically maybe should be called o2, um, but out of

respect to our friends at Telia, and in the grand tradition of OpenAI being really truly bad at names, it's going to be called o3. All right, so there it is: their next Frontier Model. This is beyond o1, and yes, they had to skip o2 because

there is a telecom company that is named o2, and they didn't want to obviously have copyright issues. This is a brand new, next-generation Frontier Model beyond o1. Kind of insane to think about, because it feels like we just got o1. And when he says "reasoning," he's talking

about test-time compute, being able to give these models plenty of time to think through a problem and then come up with a solution. This is the next generation of model; this is the next generation of AI technology. Let's keep watching. Actually, we're not going to

launch—uh, not launch—we're going to announce two models today: o3 and o3 mini. o3 is a very, very smart model; o3 mini is an incredibly smart model, but still—but a really good at performance and cost. So, to get the bad news out of the

Well, first, we're not going to publicly launch these today. Um, the good news is we're going to make them available for public safety testing starting today. You can apply. So, like he said, we're not going to get the actual models; we're not going to be able to play around with

them yet. And as soon as we can, you know, I'm going to do a full test of it. But we get to see the benchmarks, and they did benchmark it against the ARC prize, and the results are stunning. Not only the ARC prize, but of course, they benchmarked

it on math, coding, reasoning, everything, and it is leaps and bounds better than anything else that we've seen. We'll talk more at the end about when these models, uh, when we expect to make these models generally available, but we're so excited, uh, to show you what they can do, to talk

about their performance. Got a little surprise; we'll show you some demos. Uh, and without further ado, I'll hand it over to Mark to talk about it. Cool. Thank you so much, Sam. So my name is Mark; I lead research at OpenAI, and I want to talk

a little bit about o3's capabilities. Now, o3 is a really strong model at very hard technical benchmarks, and I want to start with coding benchmarks. If you can bring those up. So, on software style benchmarks, we have the Sweet Bench Verified, which is a benchmark consisting of real-world

software tasks. All right. So, first of all, here's the benchmark. You can see here's o1 preview, o1, and then o3 at 71.7%. The Sweet Bench benchmark is the best coding benchmark out there. It is real-world coding tasks, and o3 can

accomplish it at a 71.7% rate, that is far better than anything else out there, even far better than 01. Then, on the right, you can see the competition code benchmark and here's the ELO score on the Y axis. 01...

...preview 01 and all the way up here, 03. Just absolutely outstanding performance! We're seeing that 03 performance at about 71.7% accuracy, which is over 20% better than our 01 models. Now, this really signifies that we're really climbing the

frontier of utility. As well, on competition code, we see that 01 achieves an ELO on this contest coding site called Codeforces of about 1891. At our most aggressive, high test-time compute settings, we're able to achieve almost like a 2727 ELO here.  So Mark was a

competitive programmer, actually still coaches competitive programming. Very good. What... what is your... I think my best at a comparable site was about 2500. That's tough. Well, okay, so let's just break down what just happened, and I'm going to reference AGI one more time.

First, what is the definition of AGI, according to at least Sam Altman and OpenAI? AGI is AI that outperforms humans at most economically viable work. Now, this person, Mark, the head of research at OpenAI, also a competition coder, was

beaten by 03 in the competition code benchmark. Now, if that is not AGI, at least on this dimension, I don't know what is. So think about this: in chess, the best person in the world, Magnus Carlsen, has a ranking in Elo of 2831. The best AI

A chess engine is rated 3700 and above, showing AI has exceeded some of our best minds in fields like chess and Go.  That's proof point number one that AGI has been achieved. I want to mention one other...

thing: OpenAI can't technically say AGI has been achieved, or they can, but here's the problem: as soon as they define one of their systems as AGI, Microsoft no longer gets access to it. That's in their charter. So let's keep watching. I...

will say, you know, our chief scientist... this is also better than our chief scientist Yakov's score. I think there's one guy at OpenAI who still has a 300-something... Yeah, a few more months, too. Hopefully we have a couple months to enjoy this.  That's...

I mean, this model is incredible.  The programming, yeah, and not just programming, but also mathematics. So we see that on competition math benchmarks, just like competitive programming, we achieve very strong scores. Okay, let's pause again. Competition math benchmark: 96.7%, almost a flawless score. This is a...

math machine. Then on the right, we have PhD-level science questions; nearly 10 points higher than O1 preview and O1. These are massive gains for this new Frontier Model, O3. So O3 gets about 96.7% accuracy versus an O1 performance of...

83.3% on the Amy. What's your best Amy score? I did get a perfect score once, so I'm safe. But yeah, okay, that's super impressive. He got a perfect score! Wow. All right, so nearly beating Mark, the head of research at OpenAI. Really, what...

This signifies that o3, um, often just misses one question whenever we test it on this very hard feeder exam for the USA Mathematical Olympiad. There's another very tough benchmark which is called the GPQ Diamond, and this measures the model's performance on PhD-level science

questions. Here we get another state-of-the-art number: 87.7%, which is about 10% better than our o1 performance, which was at 78%. Just so I'm going to talk about why this is so important to these benchmarks in particular, especially the PhD-level

science benchmark. Now, one of the requirements for starting the intelligence explosion, as per the situational awareness paper, is having AI that can do self-research and self-improvement. And so if you have a model that is at the frontier of math, the frontier of science, and is able to

actually discover new science, discover new math, and then apply those new discoveries to itself, that is the literal definition of what is required to hit the intelligence explosion. And that's what we're seeing right here: automated AI research. So you take an o3

model, clone it a million times, and just let it run and let it self-improve indefinitely. Crazy to think about! Just to put this in perspective, if you take an expert PhD, they typically get about 70% in kind of their field of strength here. So one thing that you might notice

yeah, from some of these benchmarks is that we're reaching saturation for a lot of them, or nearing saturation. So the last year has really highlighted the need for really harder benchmarks to accurately assess where our frontier models lie. Where do you think he's going?

With this, I already hinted at it at the beginning of the video, but he's right: these benchmarks are becoming saturated; they're basically beaten or close to it. And so that's why it's important to have a very hard benchmark, something that truly tests if it's AGI or not. Let's

keep watching, and I think a couple have emerged as fairly promising over the last few months. One in particular I want to call out is Epic AI's Frontier math benchmark. Now you can see the scores look a lot lower than they did for the previous benchmarks we showed, and

this is because this is considered today the toughest mathematical benchmark out there. This is a dataset that consists of novel, unpublished, and also very hard—extremely hard—problems. Even for top researchers, you know, it would take professional mathematicians hours or even days to solve one of these

problems. And today, all offerings out there have less than 2% accuracy on this benchmark. And we're seeing with o3, in aggressive test-time settings, we're able to get over 25%. That's a 10x improvement on this benchmark as

compared to everything else that's come before it. This is wild! In addition to the Epic AI Frontier math benchmark, we have one more surprise for you guys. So I want to talk about the ARC benchmark at this point, but I would love to invite one of

our friends, Greg, who is the president of the ARC foundation, on to talk about this benchmark. Wonderful, Sam and Mark, thank you very much for having us today. Of course! Awesome, Greg Brockman. So very cool to see him on here. I've talked to him a bunch; he's such a cool, nice guy. And yeah,

He's running the ARC Benchmark, so really cool to see him here. Let's see what they do with the ARC Benchmark. Hello everybody, my name is Greg Camard, and I'm the president of the ARC Prize Foundation. Now, ARC Prize is a non-profit with the goal of being a North star towards

AGI through and during benchmarks. So our first benchmark, ARC AGI, was developed in 2019 by Francois Chollet in his paper on the measure of intelligence. However, it has been unbeaten for five years now in the AI world. That's like—it feels like

centuries, is where it is. So the system that beats ARC AGI is going to be an important milestone towards general intelligence. But I'm excited to say today that we have a new state-of-the-art score to announce. Before I get into that, though, I want to

talk about what ARC AGI is. So I would love to show you an example here. ARC AGI is all about having input examples and output examples. Well, they're good, they're good. Okay, input examples and output examples. Now, the goal is you want to understand the rule of the

transformation and guess it on the output. So Sam, what do you think is happening here? Probably putting a dark blue square in the empty space. See? Yes, that is exactly it. Now that is really—um—it's easy for humans to intuitively guess what that is; it's

actually surprisingly hard for AI to understand what's going on. So I want to show one more hard example here. Now, Mark, I'm going to put you on the spot. What do you think is going on in this task? Okay, so you take each of

These yellow squares—you count the number of colored squares there, and you create a border of that list; that is exactly it, and that's much quicker than most people. So congratulations on that. Um, what's interesting, though, is AI has not been able to get this problem thus

far, and even though we verified that a panel of humans could actually do it. Now, the unique part about Arc—so I'm going to pause for a second just to reiterate what Greg already said—the reason why the Arc benchmark, the Arc prize, is so cool and so important is

because it is so easy for most humans to do, to accomplish, to solve, and simultaneously very, very difficult for AI to do. And the reason why this is maybe the best definition of AGI is because it takes learning something—basically taking what you've learned

from one example and applying it to another example, both of which have never been seen in the core training set. And that's the important part. So it's able to infer the solution to a new problem based on something it has seen in a previous prompt or previous piece of

data. Let's keep watching. AGI is—every task requires distinct skills, and what I mean by that is we won't ask—there won't be another task that you need to fill in the corners with blue squares—but we do that on purpose. And the reason why we

do that is because we want to test the model's ability to learn new skills on the fly. We don't just want it to, uh, repeat what it's already memorized; that's the whole point here. Now, Arc AGI version one took 5 years to go from 0%

to 5% with leading Frontier models. However, today I'm very excited to say that o3 has scored a new state-of-the-art score that we have verified on low compute. For o3, it has scored 75.7 on ARC AI's semi-private holdout

set. Now this is extremely impressive because this is within the compute requirements that we have for our public leaderboard, and this is the new number one entry on ARC AI public. So congratulations to that. Thank you so much. Yeah. Now, as a capabilities

demonstration, when we ask o3 to think longer and we actually ramp up to high compute, o3 was able to score 87.5% on the same hidden holdout set. This is especially important,

because human performance is comparable at an 85% threshold. So being above this is a major milestone, and we have never tested a system that has done this, or any model that has done this beforehand. Okay, this is point number two: why this is AGI. And by the way, if you

disagree, leave a comment below; tell me why. Because this is the clearest proof; this is not clickbait, this is true. So if 85% is the human average and now o3 beats that and essentially beats PhD-level science expert-level math, what is

left for the definition of AGI? Let's keep watching. So this is new territory in the AI world. Congratulations on that. Congratulations for making such a great benchmark. Yeah, um, when I look at these scores, I realize, um, I need to switch my worldview a little bit. I need

to fix my AI intuitions about what AI can actually do and what it's capable of, uh, especially in this O3 world. But the work also is not over yet, and these are still the early days of AI, so um, we need more enduring benchmarks like ARC AGI to

help measure and guide progress, and I am excited to accelerate that progress and I'm excited to partner with OpenAI next year to develop our next Frontier Benchmark. Amazing. You know, it's also a benchmark that we've been targeting, been on our mind for a very long time, so

we're excited to work with you in the future. Worth mentioning that we didn't... we targeted, and we think it's an awesome benchmark. We didn't go do specific... this is just, you know, the general O3, but yeah, yeah. I'm glad they cleared that up; they

did not train on the ARC prize; they did not go out and specifically research the ARC prize and how to make O3 work with it; they just made O3 and it just happened to do really well on the ARC prize. Really appreciate the partnership.

And this was a fun one to do, absolutely. And even though this has done so well, the ARC prize will continue in 2025, and anybody can find out more at ARCprize.org. Great, thank you. And by the way, there is a million-dollar prize for the ARC prize, and

it seems like that might go to OpenAI. I kind of hope it goes to a small team, but it is what it is. Okay, so next up we're going to talk about O3 mini. Um, O3 mini is a thing that we're really, really

Excited about, and Hongyu, who trained the model, will come out and join us. So, not only are we getting o3, but we also get the mini version, which is going to be cheaper, faster, and probably nearly as performant. Hi everyone, um, I'm Hongyu.

I'm an OpenAI researcher working on reasoning. So this September, we released o1 mini, uh, which is an efficient reasoning model in the o1 family that's really capable of, uh, math and coding; probably one of the best in the world given the low cost. So now, together

with o3, I'm very happy to, to, uh, tell you more about, uh, o3 mini, which is a brand new model in the o3 family that truly defines a new cost-efficient reasoning frontier. It's incredible. Um, yeah, though it's not available to our users today, we

are opening access to the model to, uh, our safety and security researchers to test the model out. Um, with the release of adaptive thinking time in the API a couple of days ago, o3 mini will support three different options: low, medium, and high reasoning effort, so the

users can freely adjust the, uh, thinking time based on their different use cases. And that's really cool. So if you already know your use case, if you already know this is more of an easier problem or a harder problem, you can adjust the setting accordingly and probably save a

lot of money by doing so. Let's keep watching. So, for example, for some, we may want the model to think longer for more complicated problems and, uh, uh, think shorter with, like, simpler ones. Um, with that, I'm happy to show the first set of evaluations of o3

minimum. So, on the left-hand side, we show the coding Elo. So it's like Codeforces Elo, which measures how good a programmer is, and the higher is better. So, as we can see on the plot, with more thinking time, all three Minis are able to have like

increasing Elo. All are performing well. One Mini, with like median thinking time, is able to perform even better than all 1. Yeah, so it's like, for an order of magnitude more speed and cost, we can deliver the same code performance on this instance. Right? So, although it's

like the ultra-high is still a couple hundred points away from Mark, it's not far. That's better than me, probably. Um, but just an incredible sort of cost-to-performance gain over what we've been able to offer with O1, and we

think people will really love this. Yeah, I hope so. So, on the right-hand plot, we showed the estimated cost versus Codeforces Elo tradeoff. Uh, so it's pretty clear that all three Minis define like a new, uh, cost-efficient reasoning frontier on coding. Uh, so it's achieving like better

performance, comparable better performance than O1 with a fractional cost. Amazing. Um, with that being said, um, I would like to do a live demo of O3 Mini. All right, so we get to see it. Let's, uh, let's see how it goes. So I'm testing out all

three Mini High first, and the task is that, um, asking the model to, to, uh, use Python to implement a code generator and executor. So if I launch this, uh, run this Python script, it will launch a server, um, and, um, locally with a, with a, with a UI that contains a text box and

Then we can, uh, make coding requests in a text box. It will send the request to call the o3 Mini API, and the o3 Mini API will solve the task and return a piece of code. It will then, uh, save the code locally on my desktop and then open

a terminal to execute the code automatically. So it's a very complicated, pretty complicated, house, right? Um, and it also put, like, a big triangle code. So if we copy the code and paste it to our server, and then we would like to run/launch this server, so we should get a

text box when you're launching it. Yeah, okay, great. Oh yeah, I see. So it seems to be launching something. Um, okay, great. We have a we have a UI where we can enter some coding prompts. Let's try out a simple one, like "print open the eye and a random"

number." Submit. So it's sending the request to the o3 mini API, so it should be pretty fast, right? So, all right, so he used o3 to write code to then hit the o3 model separately. So meta, so cool! Terminal, yeah, 41, that's

the number, right? So it saves the generated code to this, like, local script, um, on the desktop and prints out, open and...  Do you want... um, is there any other task you guys want to test it out? I wonder if you could get it to get its own GPT QA

numbers. That's excellent; that's a great ask, just as what I expected. We practiced this a lot yesterday. Um, okay, so now let me copy the code and send it in the code UI. So, in this task, we ask the model to evaluate o3 mini with the low

reasoning effort on this hard GPQ data set and the model needs to first download the raw file from this URL, and then you need to figure out which part is a question, which part is a, um, which part is the answer, and/or which

part is the options, right? And then formulate all the questions and to, and then ask the model to answer it, and then parse the result and then to grade it. That's actually blazingly fast. Yeah, and it's actually really fast because it's calling the L3 mini with low reasoning

effort. Um, yeah, let's see how it goes. I guess two tasks are really hard here. Yeah, the problem... Nothing like a live demo while we're waiting for this. Do you want to show what the request was again? Mhm. Oh, it actually returns the results; it's

uh, 61.6%, 62% right? This low reasoning effort model, it's actually pretty fast. Then full evaluation in a minute. And somehow, very cool to just ask a model to evaluate itself like that. Yeah, exactly. Right. And speaking of

models evaluating themselves, there's obviously a lot of risk there. We've been talking a lot over the past week about models basically sandbagging and scheming and faking alignment during training time, so we'll see, we'll see what this all means, but it is very cool.

And if we just summarize what we just did: we asked the model to write a script to evaluate itself on this hard GQ set from a UI, right? From this code generator and executor created by the model itself in the first place. Next year we're going to bring you on

And you're going to have to improve; ask the model to improve it. So yeah, let's definitely ask the model to improve it next time, maybe. Um, so, um, besides Codeforces and GBQ, the model is also a pretty good math model, so we

show on this plot, uh, with like, on this AM 2024 data set. Also, Mini Low achieves comparable performance with 01 Mini, and 03 Mini Medium achieves comparable, better performance than 01. We check the solid bars, which are pass ones, and we can further push the performance

with 03 Mini High, right? And on the right-hand side plot, when we measure the latency on this, like, anonymized O preview traffic, we show that 03 Mini Low drastically reduces the latency of 01 Mini, right? Almost like achieving comparable latency with, uh, GPT-4, where

under a second, so probably like instant response. Uh, and also 03 Mini Medium is like half the latency of 01. Um, and here's another set of evaluations I'm even more excited to show you guys is, um, uh, API features, right? We get

a lot of requests from our developer community to support like function calling, structured outputs, developer messages, and these are the benchmarks that are going to be important if you're coding with AI or if you're building agents with AI. Look at that! Tool calling is nearly perfect up here with 03 Mini

across the board. Very, very impressive, all Mini series models. And here, um, all three Mini will support all these features, same as 01, um, and notably it achieves comparable, better performance than 01 on most of the evaluations, providing a more cost-effective

solution for our developers. Cool. Um, and if we actually unveil the true GPT-4 Diamond performance that I ran a couple of days ago, and it actually also means L is actually 62% right. We basically asked the model to evaluate itself. Yeah. Right. Next time you should

totally just ask the model to automatically do the evaluation instead of us doing it. Um, yeah. So with that, um, that's it for AltaMe, and I hope our users can have a much better user experience next year. Fantastic work! Yeah, thank you. Great work. Thank you. Cool. So I know

you're excited to get this in your own hands, um, and we're working very hard to post-train this model to do some safety interventions on top of the model, and we're doing a lot of internal safety testing right now. But something new we're doing this time is

we're also opening up this model to external safety testing, starting today with o3 mini and also eventually with o3. So how do you get early access as a safety researcher or a security researcher? You can go to our website and you can see a form like this one that

you see on the screen. And applications for this form are rolling; they'll close on January 10th, and we really invite you to apply. Uh, we're excited to see what kind of things that you can explore with this and what kind of, um, jailbreaks and

other things you discover. Play. Do your thing. So those are all the announcements about o3 and o3 mini. It is astounding; it really is, and I'm so excited to test it out. I'm going to dive more into it; I'll put some thoughts together maybe in

Another video! If you enjoyed this video, please consider giving it a like and subscribing. I'll see you in the next one!

OpenAI Unveils o3! AGI ACHIEVED!

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

OpenAI's o3: A Next-Generation Frontier Model

OpenAI released o3, claiming it's AGI (Artificial General Intelligence), supported by the ARC prize results.

The model was named o3 instead of o2 to avoid copyright issues with a telecom company.

o3 features advanced reasoning capabilities, allowing for extensive problem-solving time.

Two models were announced: o3 and o3 mini, both available for public safety testing.

Public launch is not immediate, but public safety testing is starting.

o3 excels in coding benchmarks, achieving 71.7% on Sweet Bench, surpassing o1 significantly.

o3 demonstrates superior performance in competition code benchmarks, achieving an ELO score of approximately 2727.

o3 shows significant improvement over o1 in various benchmarks, indicating a leap in utility.

o3 achieves nearly flawless scores in competition math benchmarks (96.7%) and excels in PhD-level science questions.

o3 demonstrates state-of-the-art performance on the GPQ Diamond benchmark (87.7%), surpassing o1 by 10%.

Addressing Benchmark Saturation and Introducing the ARC Benchmark

Existing benchmarks are nearing saturation, necessitating harder benchmarks to assess frontier models.

The Epic AI Frontier math benchmark is introduced as a significantly harder test.

o3 achieves over 25% accuracy on the Epic AI benchmark, a 10x improvement over previous models.

The ARC benchmark is introduced, emphasizing its difficulty for AI while being relatively easy for humans.

o3 achieves a groundbreaking score of 75.7% on the ARC AGI benchmark, surpassing human performance at higher compute settings (87.5%).

Introducing o3 mini: A Cost-Effective Reasoning Model

o3 mini is introduced as a cheaper, faster alternative to o3, with comparable performance.

o3 mini supports adjustable reasoning effort (low, medium, high), optimizing cost and performance based on task complexity.

o3 mini demonstrates strong coding performance in Codeforces Elo, improving with increased thinking time.

o3 mini offers a significant cost-to-performance improvement over o1.

A live demo showcases o3 mini's ability to generate and execute code, including a self-evaluation task.

OpenAI is opening o3 and o3 mini for external safety testing, inviting researchers to apply for early access.

The presenters express excitement about the models' potential and future testing.