Good morning. We have an exciting one for you today. We started this 12-day event 12 days ago with

the launch of o1, our first reasoning model. It's been amazing to see what people are

doing with that, and very gratifying to hear how much people like it. We view this as sort of the beginning of

the next phase of AI, where you can use these models to do increasingly complex tasks that require a lot of reasoning.

And so for the last day of this event, we thought it would be fun to go from one

frontier model to our next frontier model. Today, we're going to talk about that next frontier model, which you would

think logically maybe should be called o2. But out of respect to our friends at Telefónica and in the grand tradition of OpenAI being really,

truly bad at names, it's going to be called o3. Actually, we're going to announce

two models today, o3 and o3 mini. o3 is a very, very smart model.

o3 mini is an incredibly smart model, but still, uh, but really good at performance and cost.

So, to get the bad news out of the way first, we're not going to publicly launch these today.

The good news is we're going to make them available for public safety testing starting today. You can apply, and we'll talk about that later.

We've taken safety testing seriously as our models get more and more capable. And at this new level of capability, we want

to try adding a new part of our safety testing procedure, which is to allow public access for researchers that want to help us test.

We'll talk more at the end about when we expect to make these models generally available.

But we're so excited to show you what they can do, to talk about their performance. Got a little surprise.

We'll show you some demos. Uh, and without further ado, I'll hand it over to Mark to talk about it.

Cool. Thank you so much, Sam. So my name is Mark. I lead research at OpenAI, and I want to talk

a little bit about O3 capabilities. Now O3 is a really strong model at very hard technical benchmarks.

And I'm going to start with coding benchmarks, if you could bring those up. So on software style benchmarks, we have

SuiteBench verified, which is a benchmark consisting of real-world software tasks. We're seeing that O3 performs at about 71.7

percent accuracy, which is over 20% better than our O1 models. Now this really signifies that we're really

climbing the frontier of utility as well. On competition code, we see that O1 achieves an Elo on this contest coding

site called CodeForces is about 1891. At our most aggressive high test-time compute settings, we're able to

achieve almost like a 2700 Elo here. So Mark was a competitive programmer.

Actually, still coaches competitive programming. Very, very good. What, what is your— I think my best at a

comparable site was about 2500. That's tough. Well, I, I will say, you know, our chief

scientist, um, this is also better than our chief scientist, Ilya's score. I think there's one guy at OpenAI

who's still like a 3000 something. Yeah. Got a few more months to— Yeah.

Hope to enjoy. Hopefully, we have a couple of months to enjoy that. Great. That's— I

I mean, this model is incredible at programming. Yeah. And not just programming, but also mathematics.

So we see that on competitive math benchmarks, just like competitive programming, we achieve very, very strong scores.

So O3 gets about 96.7% accuracy versus an O1 performance of 83.3% on the AME.

What's your best AIME score? I did get a perfect score once. All right.

So I'm safe. But yeah. Um, really what this signifies is that O3, um, often just misses

one question whenever we test it on this very hard feeder exam for the USA Mathematical Olympiad.

There's another very tough benchmark which is called GPQA Diamond, and this measures the model's performance on PhD-level science questions.

Here we get another state-of-the-art number, 87.7%, which is about 10% better than our

O1 performance, which was at 78%. Just to put this in perspective, if you take

an expert PhD, they typically get about 70% in their field of strength.

So one thing that you might notice from some of these benchmarks is that we're reaching

saturation for a lot of them, or nearing saturation. So the last year has really highlighted the need for really harder benchmarks to

accurately assess where our frontier models lie. And I think a couple have emerged as fairly promising over the last few months.

One in particular I want to call out is Epic AI's Frontier Math benchmark. Now, you can see the scores look a lot lower than

they did for the previous benchmarks we showed. And this is because this is considered today the toughest mathematical benchmark out there.

This is a data set that consists of novel, unpublished, and also very hard problems. These are extremely hard.

Yeah. Very, very hard problems. Even for experts, you know, it would take professional mathematicians hours or

even days to solve one of these problems. And today, all offerings out there, um, have

less than 2% accuracy, um, on this benchmark. And we're seeing with o3, in aggressive test

time settings, we're able to get over 25%. Yeah. That's awesome. In addition to Epic AI's Frontier Math

benchmark, we have one more surprise for you guys. So I want to talk about the ARC benchmark at this point.

But I would love to invite one of our friends, Greg, who is the president of the ARC

Foundation, on to talk about this benchmark. Wonderful. Sam and Mark, thank you very much

for having us today. Of course. Hello, everybody. My name is Greg Kamrad, and I am the

president of the ARC Prize Foundation. Now, ARC Prize is a nonprofit with the mission of being a north star towards

AGI through enduring benchmarks. So our first benchmark, ARC AGI, was developed in 2019 by Francois Chollet in

his paper on the measure of intelligence. However, it has been unbeaten for 5 years.

Now, in the AI world, that's like, it feels like centuries. So the system that beats ARC AGI is going to be an

important milestone towards general intelligence. But I'm excited to say today that we have a new state-of-the-art score to announce.

Before I get into that, though, I want to talk about what ARC AGI is. So I would love to show you an example here.

Arc AGI is all about having input examples and output examples. Well, they're good.

Okay. Input examples and output examples. Now the goal is you want to understand the rule of the transformation and guess it on the output.

So, Sam, what do you think is happening here? Probably putting a dark blue square in the empty space.

See? Yes. That is exactly it. Now that is really, um, it's easy for

humans to intuitively guess what that is. It's actually surprisingly hard for AI to understand what's going on.

So I want to show one more hard example here. Now, Mark, I'm going to put you on the spot.

What do you think is going on in this task? Okay. So you take each of these yellow squares, you

count the number of colored squares there, and you create a border of that count. That is exactly it, and that's much quicker

than most people, so congratulations on that. Um, what's interesting though is AI has not been able to get this problem thus

far, even though we verified that a panel of humans could actually do it. Now the unique part about Arc AGI is

every task requires distinct skills. And what I mean by that is we won't ask—there won't be another task that you need

to fill in the corners with blue squares. And but we do that on purpose. And the reason why we do that is

because we want to test the model's ability to learn new skills on the fly. Mhmm.

We don't just want it to repeat what it's already memorized. That's the whole point here.

Now, Arc AGI version 1 took 5 years to go from 0% to 5% with leading frontier models.

However, today, I'm very excited to say that O3 has scored a new state-of-the-art score that we have verified.

On low compute for O3, it has scored 75.7 on Arc AGI's semi-private holdout set.

Now, this is extremely impressive because this is within the compute requirements that we have for our public leaderboard, and

this is the new number one entry on RKGPub. So congratulations to that. Thank you so much.

Yeah. Now, as a capabilities demonstration, when we ask O3 to think longer and we actually ramp

up to high compute, O3 was able to score 87.5 percent on the same hidden holdout set.

This is especially important because human

performance is comparable at an 85% threshold. So being above this is a major milestone.

And we have never tested a system that has done this, or any model that has done this beforehand.

So this is new territory in the Arc AGI world. Congratulations on that. Congratulations for making such a great benchmark.

Yeah. Um, when I look at these scores, I realize I need to switch my worldview a little bit.

I need to adjust my AI intuitions about what AI can actually do and what it's capable of,

especially in this O3 world. But the work also is not over yet, and these are still the early days

of AI. So we need more enduring benchmarks like Arc AGI to help measure and guide progress.

And I am excited to accelerate that progress, and I'm excited to partner with OpenAI next year to develop our next frontier benchmark.

Amazing. You know, it's also a benchmark that we've been targeting and have had in mind for a very long time.

So excited to work with you! Worth mentioning that we did target it, and we think it's an awesome platform.

We didn't do specific work with you. This is just, you know, the general case of 3.

But, yeah, really appreciate the partnership. Yeah. This was a fun one to do. Absolutely.

And even though this has done so well, the Arc Prize will continue in 2025, and anyone

can find out more at arcprize.org. Great. Thank you so much for coming on.

Yep. Absolutely. Okay. So next up, we're going to talk about O3 Mini. Um, O3 Mini is something that

we're really, really excited about. And Hong Yu, who trained the model, will come out and join us.

Hey. Hey. Hey. Um, hi, everyone. Um, I'm Hong Yu. I'm an OpenAI researcher, uh, working on reasoning.

So this September, we released O1 Mini, which is an efficient reasoning model in the O1 family that's really capable

[한영자막]인간 능력 뛰어넘는 AI? OpenAI O3 & O3 Mini가 바꿀 미래—12 Days of OpenAI Day 12

🚀 안녕하세요, '평범한 사업가' 채널에 오신 것을 환영합니다! 
제 채널에서는 최신 AI 기술, 혁신적인 비즈니스 전략, 그리고 유튜브 채널 성장을 위한 실질적인 인사이트를 공유합니다. 🤖💡

🌟 이 채널은 기술과 비즈니스에 열정을 가진 사람들을 위한 공간입니다. AI의 최신 동향부터 초 개인화를 위한 실용적인 팁까지, 여러분의 비즈니스 여정에 필요한 모든 것을 제공합니다. 📈🔥

🔍 구독하시고, AI와 ChatGPT를 활용하여 엄청난 효율화와 생산성을 경험하시고 성공적인 여정을 함께 시작해보세요. 매주 업데이트되는 콘텐츠로 여러분을 기다리고 있습니다! 🎥🎉"


[한영자막]인간 능력 뛰어넘는 AI? OpenAI O3 & O3 Mini가 바꿀 미래—12 Days of OpenAI Day 12

Scripts