[Music] Good morning! We have an exciting one for you today. We started this 12-day event

12 days ago with the launch of O1, our first reasoning model. It's been amazing to see what people are doing with that

and very gratifying to hear how much people like it. We view this as sort of the beginning of the next phase of AI,

where you can use these models to do increasingly complex tasks that require a lot of reasoning. And so, for the last

day of this event, um, we thought it would be fun to go from one Frontier Model to our next Frontier Model. Today we're

going to talk about that next Frontier Model, um, which you would think logically maybe should be called O2, um, but out of

respect to our friends at Teleca, and in the grand tradition of OpenAI being really truly bad at names, it's going to

be called O3. Actually, we're going to announce—not launch—we're going to announce two models today: O3 and O3 Mini.

O3 is a very, very smart model. O3 Mini is an incredibly smart model, but still—but a really good performance and

cost. So, to get the bad news out of the way first: we're not going to publicly launch these today. Um, the good news is

we're going to make them available for public safety testing starting today. You can apply, and we'll talk about that

later. We've taken safety testing seriously as our models get more and more capable, and at this new level of

capability, we want to try adding a new part of our safety testing procedure, which is to allow public access for

researchers that want to help us test. We'll talk more at the end about when these models—when we expect to make

these models generally available. But we're so excited to show you what they can do, to talk about their

performance got a little surprise; we'll show you some demos, uh, and without further ado, I'll hand it over to Mark to

talk about it. Cool. Thank you so much, Sam. So my name is Mark; I lead research at OpenAI, and I want to talk a little bit

about GPT's capabilities. Now, GPT is a really strong model at very hard technical benchmarks, and I want to start with

coding benchmarks. If you can bring those up, so on software style benchmarks, we have SBENCH verified, which is a

benchmark consisting of real-world software tasks. We're seeing that GPT-3 performs at about 71.7% accuracy, which is over 20% better

than our GPT-1 models. Now, this really signifies that we're really climbing the frontier of utility. As well on

competition code, we see that GPT-1 achieves an ELO on this contest coding site called Codeforces of about 1891. At our

most aggressive, high test-time compute settings, we're able to achieve almost like a 2727 ELO here. Ju—so Mark was a

competitive programmer, actually still coaches competitive programming. Very, very good. What—what is your—I think my best at a comparable site was about 2500.

That's tough. Well, I—I will say, you know, our chief scientist, um, this is also better than our chief scientist, Ilya Sutskever's

score. I think there's one guy at OpenAI who's still like a 3000 something. Yeah, a few more months to—yeah, enjoy.

Hopefully, we have a couple of months to enjoy that. Great. That's—I mean, this is—it's—this model is incredible at

programming. Yeah, and not just programming, but also mathematics. So we see that on competition math benchmarks, just like competitive

programming, we achieve very, very strong scores. So GPT-3 gets about 96.7% accuracy versus a GPT-1 performance of 83.3% on the

MATH dataset. What's your best MATH score? I did get a perfect score once, so I'm safe. But yeah, um, really what this signifies is

That 03, um, often just misses one question whenever we tested it on this very hard feeder exam for the USA Mathematical Olympiad.

There's another very tough benchmark, which is called the GQA Diamond, and this measures the model's performance on PhD-level science questions.

Here we get another state-of-the-art number: 87.7%, which is about 10% better than our 01 performance, which was at 78%.

Just to put this in perspective, if you take an expert PhD, they typically get about 70% in kind of their field of strength.

So one thing that you might notice from some of these benchmarks is that we're reaching saturation for a lot of them, or nearing saturation.

So the last year has really highlighted the need for really harder benchmarks to accurately assess where our frontier models lie.

And I think a couple have emerged as fairly promising over the last months. One in particular I want to call out is Epic AI's Frontier Math benchmark.

Now you can see the scores look a lot lower than they did for the previous benchmarks we showed, and this is because this is considered today the toughest mathematical benchmark out there.

This is a dataset that consists of novel, unpublished, and also very hard—extremely hard—problems.  Even seasoned mathematicians would take hours or even days to solve one of these problems.

And today, all offerings out there have less than 2% accuracy on this benchmark. And we're seeing with 03, in aggressive test time settings, we're able to get over 25%.

Yeah, that's awesome! In addition to Epic AI's Frontier Math benchmark, we have one more surprise for you guys.

So I want to talk about the ARC benchmark at this point, but I would love to invite...

One of our friends, Greg, who is the president of the Arc foundation, is on to talk about this benchmark. Wonderful, Sam!

And Mark, thank you very much for having us today. Of course, hello everybody. My name is Greg Camad, and I'm the president

of the Arc Prize Foundation. Now, Arc Prize is a nonprofit with the mission of being a North star towards AGI through

developing benchmarks. So, so our first benchmark, Arc AGI, was developed in 2019 by Francois Chollet in his paper on the

measure of intelligence. However, it has been unbeaten for five years now in the AI world. That's like—it feels like

centuries.  So the system that beats Arc AGI is going to be an important milestone towards general

intelligence. But I'm excited to say today that we have a new state-of-the-art score to announce.

Before I get into that, though, I want to talk about what Arc AGI is. So I would love to show you an example here. Arc AGI

is all about having input examples and output examples. Well, they're good, they're good. Okay, input examples and

output examples. Now, the goal is you want to understand the rule of the transformation and guess it on the

output. So, Sam, what do you think is happening in here? Probably putting a dark blue square in the empty space. See?

Yes, that is exactly it. Now that is really—um—it's easy for humans to—uh—guess what that is. It's actually

surprisingly hard for AI to understand what's going on. So I want to show one more hard example here. Now, Mark,

I'm going to put you on the spot. What do you think is going on in this uh task?

Okay, so you take each of these yellow squares, you count the number of colored squares there, and you create a

Border of that with that; that is exactly it, and that's much quicker than most people. So congratulations on that. Um, what's

interesting, though, is AI has not been able to get this problem thus far, and even though we verified that a

panel of humans could actually do it. Now, the unique part about ARC AGI is every task requires distinct skills, and what I

mean by that is we won't ask—there won't be another task that you need to fill in

the corners with blue squares. But we do that on purpose, and the reason why we do that is because we want to test the

model's ability to learn new skills on the fly. We don't just want it to, uh, repeat what it's already memorized; that

that's the whole point here. Now, ARC AGI version 1 took 5 years to go from 0% to

5% with leading frontier models. However, today I'm very excited to say that 03 has scored a new state-of-the-art score

that we have verified on low compute. For, uh, 03, it has scored 75.7 on ARC AI's semi-private holdout

set. Now, this is extremely impressive because this is within the, uh, compute requirement that we have for our public

leaderboard, and this is the new number one entry on RKG Pub. So congratulations on that! So much, yeah. Now, uh, as a

capabilities demonstration, when we ask 03 to think longer and we actually ramp up to high compute, 03 was able to score

87.5% on the same hidden holdout set. This is especially important because

um, human performance is comparable at an 85% threshold. So being above this is a major milestone, and we

have never tested a system that has done this, or any model that has done this beforehand. So this is new territory in

The RCGI world, congratulations with that! Congratulations for making such a great benchmark. Yeah, um, when I look at these

scores, I realize, um, I need to switch my worldview a little bit. I need to fix my AI intuitions about what AI can actually

do and what it's capable of, uh, especially in this ARC-AGI world. But the work also is not over yet, and these are still

the early days of AI. So, um, we need more enduring benchmarks like ARC AGI to help measure and guide progress, and I am

excited to accelerate that progress, and I'm excited to partner with OpenAI next year to develop our next frontier

benchmark. Amazing. You know, it's also a benchmark that we've been targeting and been on our mind for a very long time, so

excited to work with you in the future. Worth mentioning that we didn't... we targeted, and we think it's an awesome

event. We didn't go for specific... the general... but yeah, really appreciate the partnership. This was a

fun one to do. Absolutely. And even though this has done so well, ARC prize will continue in 2025, and anybody can find

out more at arcprize.org. Great. Thank you so much. Absolutely. Okay, so next up we're going

to talk about O3 mini. Um, O3 mini is a thing that we're really, really excited about, and Hongyu, who trained the model,

will come out and join us. Hey, hey, you. Hey. Um, hi everyone. Um, I'm Hongyu. I'm an OpenAI

researcher, uh, working on reasoning. So this September, we released O1 mini, uh, which is an efficient reasoning model

in the O1 family that's really capable of math and coding, probably one of the best in the world given the low cost.

So now, together with O3, I'm very happy to, uh, tell you more about O3 mini.

which is a brand-new model in the 03 family that truly defines a new cost-efficient reasoning frontier. It's

incredible. Um, yeah, though it's not available to our users today, we are opening access to the model to our

safety and security researchers to test the model out. Um, with the release of adaptive thinking time in the API a

couple of days ago, all three mini will support three different options: low, median, and high reasoning effort. So the

users can freely adjust the thinking time based on their different use cases. So, for example, for some, we may want the

model to think longer for more complicated problems and think shorter with simpler ones. Um, with that, I'm happy to show the

first set of evaluations of all three mini. Um, so on the left-hand side, we show the coding eval, so it's like Codeforces

ELO, which measures how good a programmer is. Uh, the higher, the better. So, as we can see on the plot, with more thinking

time, all three mini is able to have increasing ELO, outperforming all one mini, and with median thinking time,

is able to perform even better than all one. Yeah, so it's like for an order of magnitude more speed and cost, we can

deliver the same code performance, or even better performance, right? So, although the ultra-minimum high is

still a couple hundred points away from the mark, it's not far. That's better than me, probably. Um, but just an

incredible sort of cost-to-performance gain over what we've been able to offer with 01, and we think people will really love

모든 분야 인간 넘어선 AGI??! O3의 등장 | 수학올림피아드 96.7%, 코드포스 2727점 달성! , OpenAI o3 and o3-mini—12 Days of OpenAI

코난쌤 conanssam

O3 및 O3 Mini 모델 발표 및 안전 테스트

12일간의 이벤트 마지막 날, 새로운 프론티어 모델 O3와 O3 Mini 발표

이전 모델 O1의 성공적인 출시 이후, 더욱 복잡한 추론 작업을 수행하는 모델 개발

O3는 매우 뛰어난 성능을 가진 모델이며, O3 Mini는 성능과 비용 면에서 효율적인 모델

공개 출시는 하지 않고, 공개 안전 테스트를 위한 신청 접수 시작

향상된 모델 성능에 따른 강화된 안전 테스트 절차 도입 및 공개 연구자 참여

코딩 벤치마크(SBENCH, Codeforces)에서 GPT-3보다 훨씬 높은 점수 달성

수학 벤치마크(MATH dataset, USA Mathematical Olympiad)에서도 매우 높은 정확도 기록

GQA Diamond 벤치마크에서 최첨단 성능 달성 (87.7%, 전문가 수준 능가)

Epic AI의 Frontier Math 벤치마크에서 25% 이상의 정확도 달성 (기존 모델 대비 압도적)

ARC AGI 벤치마크 결과 및 AGI 발전 방향

Arc Prize Foundation의 Greg Camad 소개 및 ARC AGI 벤치마크 설명

O3가 ARC AGI에서 75.7점(저사양), 87.5점(고사양)을 기록하며 최고점수 경신 (인간 수준 성능 상회)

ARC AGI와 같은 지속적인 벤치마크의 중요성 강조 및 향후 협력 계획 언급

O3 Mini 모델 개발자 Hongyu 소개 및 모델 특징 설명 (비용 효율적인 추론 성능)

Codeforces ELO 및 추론 시간에 따른 성능 변화 분석 (O1 Mini 대비 향상된 성능)

실시간 데모를 통해 O3 Mini의 코드 생성 및 실행 기능 시연

수학 벤치마크(AM 2024)에서도 O1 Mini 및 O3 대비 우수한 성능

O3 및 O3 Mini 모델 안전 테스트 및 향후 계획

외부 안전 테스트 참여 신청 안내 (마감일: 1월 10일)

새로운 안전 프로그램 '숙고적 정렬(deliberative alignment)' 기술 소개 및 성능 향상 결과 발표

O3 Mini는 1월 말, O3는 그 이후 출시 예정