ARC 벤치마크에서 인간의 성능을 뛰어넘는 OpenAI의 GPT-3: 이것이 AGI인가?

### OpenAI의 GPT-3, AGI 벤치마크에서 획기적인 성과 달성
* 🤖 OpenAI의 GPT-3 모델 출시는 ARC 벤치마크에서 인간의 성능을 능가하는 잠재적인 AGI 이정표가 될 것입니다.
* 🧠 ARC 벤치마크는 암기식 학습이 아닌 핵심 지식을 테스트하기 때문에 중요한 의미를 갖습니다.
* 💡 이 벤치마크는 일반적인 AI 작업과 달리 즉각적인 학습이 필요한 새로운 퍼즐을 제시합니다.  인간은 쉽게 달성할 수 있지만 AI는 이전에는 불가능했던 작업입니다.
* 📈 GPT-3는 ARC AGI 세미 프라이빗 홀드아웃 세트에서 75.7점을 획득하여 공개 리더보드에서 새로운 최고 점수를 기록했습니다. 이는 지난 5년간의 진전 끝에 이룬 의미 있는 성과입니다.
* ⚙ 속도와 비용 효율성에 최적화된 로우 튜닝과 복잡한 작업에 최적화된 하이 튜닝의 두 가지 버전의 GPT-3를 테스트했습니다.  하이 튜닝 버전은 인간의 성능을 뛰어넘었습니다.
* 🤔 전문가들은 이를 중요한 이정표로 인정하지만, 이것이 진정한 AGI를 구성하는지에 대해서는 논쟁이 계속되고 있습니다.  앞으로 더 많은 벤치마크가 나올 것으로 예상됩니다.
* 💲 고도로 튜닝된 GPT-3는 작업당 약 11,000달러의 비용이 들 정도로 계산 비용이 많이 들기 때문에 현재로서는 한계가 있습니다. 하지만 향후 비용 절감 효과가 기대됩니다.
* 🚀 이 모델은 소프트웨어 엔지니어링과 고급 수학을 포함한 다른 벤치마크에서도 상당한 개선을 보여 AI 역량이 빠르게 발전하고 있음을 보여주었습니다.

OpenAI의 새로운 모델인 003은 ARC 벤치마크에서 인간의 성능을 능가하며 AGI의 잠재적 돌파구를 마련했습니다.

So today actually marks a very historic moment for the AI community, as it's going to be probably regarded as the day

where AGI actually happened. Now, if you guys don't know why this is the case, this is because OpenAI today

released—announced, I guess you could say—their new 03 model, which is the second iteration of their 01 series, the

model that thinks. For a long time now, if you don't understand why this is potentially AGI, this is because the new

system managed to surpass human performance in the ARC benchmark. Now, the reason that the ARC benchmark is such an

important benchmark is because it is resistant to memorization. Sure, so ARC is intended as a kind of IQ test for

machine intelligence, and what makes it different from most benchmarks out there is that it's designed to be resistant to

memorization. So if you look at the way LLMs work, they're basically this, uh, big interpolative memory, and the way you

scale up the capabilities is by trying to cram as much, uh, knowledge and patterns as possible into them. And, uh, by

contrast, ARC does not require a lot of knowledge at all. It's designed to only require what's known as core knowledge,

which is, uh, basic knowledge about things like elementary physics, objectness, counting, that sort of thing—um, the sort

of knowledge that any four-year-old or five-year-old possesses, right? Um, but what's interesting is that each puzzle

in ARC is novel; it's something that you've probably not encountered before, even if you've memorized the entire internet. Now,

if you want to know what ARC actually looks like in terms of this test that humans are so easily able to pass but

you know, these AI systems currently aren't, you can take a look at the current examples right here in this

Video example here. AGI is all about having input examples and output examples. Well, they're good.

Okay, input examples and output examples. Now, the goal is you want to understand the rule of the transformation and guess

it on the output. So, Sam, what do you think is happening in here? Probably putting a dark blue square in the empty

space. See? Yes, that is exactly it. Now, that is really—um—it's easy for humans to intuitively guess what that is.

It's actually surprisingly hard for AI to understand what's going on. Um, what's interesting though is AI has not

been able to get this problem thus far, even though we verified that a panel of humans could actually do it. Now,

the unique part about AR—AGI is every task requires distinct skills. And what I mean by that is we won't ask—

there won't be another task that you need to fill in the corners with blue squares. But we do that on purpose,

and the reason why we do that is because we want to test the model's ability to learn new skills on the fly. We don't

just want it to repeat what it's already memorized. That's the whole point here. Now, Ark AGI version one took

five years to go from 0% to 5% with leading frontier models. However, today I'm very

excited to say that 0.3 has scored a new state-of-the-art score that we have

verified on low compute. For 0.3, it has scored 75.7 on the Ark AGI semi-private holdout set.

Now, this is extremely impressive because this is within the compute requirements that we have for our public leaderboard, and this is the new number

one entry on the Ark AGI public leaderboard. So, congratulations to that. Now, I know those of you outside the AI community might not think this is

a big deal, but this is a really big deal because this is something that we've been trying to solve for, I think, around 5

years now. This is a benchmark that many would have heralded to be the golden standard for AI and would, of course, mark

the first time that we've actually managed to get to a system that can actually outperform humans at a task

that traditionally AI systems would particularly fail at. Now, what was interesting was that they had two versions: we had 003 with low tuning

and we had 003 with high tuning. So, the 003 with low tuning is the low reasoning effort, and this is the model that

operates with minimum computational effort, which is optimized for speed and cost efficiency. And this one is suitable

for simpler tasks where deep reasoning is not required; so, you know, basic coding, straightforward tasks. And then, of course,

you have the high-tuned one, which is where the model takes more time and resources to analyze and solve problems,

and this is optimized for performance on complex tasks requiring deeper reasoning or multi-step problem-solving. And what we can see here is that when we actually

you know, tune the model and make it think for longer, we can see that we actually manage to surpass where humans

currently are. Now, what's crazy about this is that the people that have created the ARC AGI Benchmark said that

the performance on ARC AGI highlights a genuine breakthrough in novelty adaptation. This is not incremental progress; we are in new territory. So they

start to ask, 'Is this AGI?' 003 still fails on some very easy tasks, indicating fundamental differences with human

intelligence. And the guy that you saw at the beginning, François Chollet, actually spoke about how he doesn't believe that

this is exactly AGI, but it does represent a big milestone on the way to what's AGI. He says there's still a fair

Number of easy ARC AGI; one talks to 003 can't solve, and we have early indications that ARC AI 2 will remain

extremely challenging.  So he's stating that it shows that it's feasible to create unsaturated, interesting benchmarks

that are easy for humans yet impossible for AI without involving specialist knowledge, and he states now

which is, you know, some people could argue that he's moved the goal; that we will have AGI when creating such evals

becomes outright impossible. But this is a little bit in contrast to what he said earlier this year. Take a look at what he

said six months ago about the benchmark surpassing 80%, which it did today.  Turning the question around to you, so suppose that

it's the case that in a year, a multimodal model can solve ARC, let's say get 80%, whatever the average human would

get, then AGI? Quite possibly, yes. I think if you—if you start—so honestly, what I would like to see is a large language model type

model solving ARC at like 80%, but after having only been trained on core knowledge-related stuff. Now, one of the

limitations of the model is actually the compute cost. So you can see right here that he says, "Does this mean that the ARC

prize competition is beaten?" He says, "No." The ARC prize competition targets the fully private data set, which is a

different and somewhat harder evaluation, but the ARC prize is, of course, the one where your solutions must run with a

fixed amount of compute, which is about 10 cents per task. And the reason that that is really interesting is because I'm not

sure you guys may have seen this, but if we actually take a look down the bottom here, you can see that 003 high-tuned the

amount of compute that is being put into the model. There means that this model cost around $11,000 per task, which is

Pretty pricey if you're trying to use that AI for anything at all. I think these models that search over many

different solutions are going to be really, really expensive, and we can see that reflected here with this one being

over $1,000 per task, which is ridiculously expensive when you think about using it to perform any kind of

task. Of course, as we've seen with AI, costs will eventually come down. So the fact that they've managed to get the

benchmark is just the very best thing. We can use the similarities between how technology was pretty bulky in the early

days—you know, like the big TVs, the really bulky phones—but now with technology, you find ways to become more

and more efficient, and eventually you do things faster and, of course, cheaper. So it's quite likely that this will happen

in AI, too. And what I do find crazy about this is that, you know, two years ago, he said that the ARC AGI benchmark fully

solved is not going to be within the next 8 years, and he says 70% hopefully

less than 8 years, perhaps four or five. Of course, you can see AI managing to speed past most people's predictions. We

also got the fact that it did very well on the SWE benchmark, which is, of course, a very hard software engineering benchmark, and

I'm guessing that if you are a software engineer, this is probably not the best news for you. But I'm sure that now with

a bunch of people who are now coding, there's probably a lot more demand for software engineers that actually

understand the code that's actually being written here. But this is actually rather interesting because one thing that I realized when, you know, doing this

video was the fact that this is actually O(2) and not O(3). Just want to go on a

Tangent quickly here because, whilst I was reading this breakthrough about 03, one thing I needed to

remember is that this is actually the second iteration of the model. I think some people might be thinking that 03 is

opening a third iteration, but 02 is simply being skipped because there's a conflict with O2, which is a British

mobile service provider. So, the fact that this is only the second iteration of the model does go to show that

potentially 03 or even 04 is going to be quite a large jump and might reach benchmark saturation. Now, we also look at

the math benchmarks and the PhD-level science benchmarks; we can see that there's a decent improvement there as well. I

do think, though, that this is sort of reaching the benchmark saturation area because we can see this one on

competition math is around 96.7%, and this one is 87.7%. And, like I said before,

one of the things that most people are starting to say is that, like, okay, AI has slowed down because it's no longer

increasing at the same rates as it was before. What we have to understand is that, as these benchmarks get to 95% plus

or around that 89%, the incremental gains are going to be harder and harder to reach because, number one, you only have

10% left to get, and number two, it's quite likely that 3 to 5% of all the

questions are contaminated, meaning that there are errors in those questions anyway. Which means that 100% is simply

not possible on certain benchmarks, which is why they decided to create the Frontier Math benchmark. Now, at the time

when the benchmark was released, which I think was around two or three months ago, these were the current models that could

Only 2% on these questions. You can see at the lead was Gemini 1.5, Pro, Claude 3,

Sonic. And you can see 01 preview and 01 mini were there, but for those of you

that haven't realized just how good 02/03 is, this is a model that gets 25%.

So this is something that is really, really incredible when it comes to research, math. And you have to understand

that this kind of math is super, super difficult, and all of those questions are completely novel. So the previous

benchmarks that we recently had aren't truly showing the capabilities. I mean, think about it like this: okay, when we

take a look at these benchmarks, you can see that, okay, you would say maybe the model's gotten, you know, 10% better

overall, which is just not true. Because if we actually look at the really hard benchmarks where it's really solving

unseen math problems, we can see that there is a 20-times improvement over the current state-of-the-art, which is just

absolutely incredible. So I think this is probably the very important image that people need to realize because you can't

OpenAI Just Revealed They ACHIEVED AGI (OpenAI o3 Explained)

From the latest research and developments in machine learning and natural language processing, to practical applications and the ethical considerations surrounding AI, this channel is your go-to source for all things related to this exciting field. Subscribe now to stay up-to-date on the cutting-edge of AI and join the conversation about the future of intelligence.

contact@theaigrid.com (For Business Enquiries) 


TheAIGRID

OpenAI의 GPT-3 모델, ARC 벤치마크에서 획기적인 성과 달성

OpenAI는 ARC 벤치마크에서 인간의 성능을 능가하는 새로운 GPT-3 모델을 출시했습니다.

ARC 벤치마크는 핵심 지식에 초점을 맞춘 암기식 시험이라는 점에서 의미가 있습니다.

암기에 의존하는 LLM과 달리 ARC는 참신한 퍼즐을 통해 추론 능력을 테스트합니다.

ARC의 각 퍼즐은 고유하며 암기만으로는 쉽게 풀 수 없습니다.

ARC 벤치마크는 입력에서 출력 예제로의 변환 규칙을 이해하는 것을 포함합니다.

GPT-3의 성능 분석 및 AGI에 대한 시사점

GPT-3는 ARC AGI 벤치마크에서 75.7점을 획득하여 이전 모델보다 크게 개선되었습니다.

로우 튜닝(속도에 최적화)과 하이 튜닝(복잡한 작업에 최적화)의 두 가지 버전의 GPT-3를 테스트했습니다.

고도로 튜닝된 버전은 벤치마크에서 인간의 성능을 뛰어넘었습니다.

전문가들은 이를 새로운 적응의 진정한 돌파구라고 인정하지만, GPT-3가 여전히 몇 가지 쉬운 작업에서 어려움을 겪고 있다고 지적합니다.

이것이 AGI에 해당하는지에 대한 논의가 진행 중이며, 일부에서는 이를 주요한 이정표로 간주하지만 진정한 AGI는 아니라고 생각합니다.

1년 이내에 ARC에서 80%를 달성하는 것이 AGI를 의미하는지에 대한 논의가 있었습니다.

높은 컴퓨팅 비용은 GPT-3의 한계로, 작업당 약 11,000달러의 높은 튜닝 비용이 소요됩니다.

과거의 기술 발전과 마찬가지로 시간이 지남에 따라 비용이 감소할 것으로 예상됩니다.

GPT-3의 성능은 벤치마크 달성 일정에 대한 이전의 예측을 뛰어넘었습니다.

GPT-3는 SWE 벤치마크에서도 강력한 성능을 보여 잠재적으로 소프트웨어 엔지니어링 분야에 영향을 미칠 수 있습니다.

GPT-3이 모델의 두 번째 반복임을 설명합니다(0.2는 이름 충돌로 건너뜀).

GPT-3는 수학 및 과학 벤치마크가 크게 향상되어 벤치마크 포화 상태에 근접했습니다.

벤치마크 포화 및 잠재적인 질문 오류로 인한 AI 발전 속도 둔화에 대한 논의가 있었습니다.

프론티어 수학 벤치마크에서 GPT-3의 성능을 다른 모델과 비교한 결과, 상당한 개선이 있었습니다.

다른 모델에 비해 보이지 않는 수학 문제 풀이 능력이 크게 향상(20배)되었다는 점을 강조합니다.

OpenAI의 노암 브라운은 GPT-3가 개선의 궤도를 이어갈 것이라고 말합니다.

진화하는 정의와 다단계 프레임워크를 강조하는 샘 알트먼의 AGI에 대한 관점.

알트먼은 2025년 말까지 놀라운 인지 작업을 수행할 수 있는 시스템을 구축할 수 있을 것으로 예상합니다.

GPT-3 테스트를 도와줄 안전 연구원을 모집하고 일반 출시에 대비하세요.

벤치마크가 깨진 것에 대한 생각과 앞으로의 발전에 대한 기대감을 마무리합니다.