프로젝트 스트로베리: OpenAI의 QAR 알고리즘 살펴보기

### OpenAI의 프로젝트 스트로베리 해독하기: AGI 연구의 잠재적인 돌파구
* 🍓 오픈AI의 프로젝트 스트로베리는 **QAR로도 알려져 있으며, 공식 발표 자료나 공개 정보 없이 베일에 싸여 있습니다.**
* 🤖 이 프로젝트에는 기존 트랜스포머 아키텍처에는 없는 기능인 외부의 영향 없이 **AI 모델이 논리적, 수학적 기술을 학습할 수 있는 새로운 알고리즘**이 포함된다는 소문이 돌고 있습니다.
* 💡 이러한 혁신은 인공지능이 **'환각'이나 논리적 오류 없이 다단계 추론**을 수행할 수 있게 함으로써 **인공 일반 지능(AGI)을 달성하기 위한 중요한 단계**가 될 수 있습니다.
* 🔍 QAR 알고리즘은 **Q-학습과 AAR 검색**을 결합하여 모델이 단계별 사고와 프로세스 세분화를 통해 보다 인간과 유사한 '시스템 2' 방식으로 사고할 수 있도록 하는 것으로 알려져 있습니다.
* 🧠 AI가 보다 반복적이고 체계적으로 사고할 수 있도록 지원함으로써 **QAR은 대규모 언어 모델이 복잡한 작업과 의사 결정 프로세스에 접근하는 방식에 혁신**을 가져올 수 있습니다.
* 💻 QAR의 계산 강도는 문제가 될 수 있지만, **셀프 플레이와 같은 기술**을 사용하고 **더 작고 에너지 효율적인 모델에 QAR 변형**을 통합하면 이를 완화할 수 있습니다.
* 📈 OpenAI 경영진은 회사가 AI 개발 계층 구조에서 **'추론자'로 알려진 두 번째 단계**에 도달하기 직전이라고 밝힌 바 있습니다.
* 🚀 AGI 연구의 이러한 잠재적 돌파구는 AI 커뮤니티 내에서 흥분과 추측을 불러일으켰으며, 많은 사람들이 **OpenAI의 추가 개발**을 간절히 기다리고 있습니다.

프로젝트 스트로베리는 추론과 자가 학습 능력을 향상시키는 것을 목표로 하는 AI의 잠재적 혁신을 상징합니다.

Twitter has been on absolute fire for the last few days with strawberry memes. The hype is real, the memory is strong, and I know a lot of you come to this channel for the more serious stuff, and of course, that's going to be my bread and butter. But every once in a while, I like to join in on the memory and the hype, so I'm having a little fun here. Hopefully, you're having fun too. But in this video, I'm going to actually break down everything we really know about strawberry, what it could be, all of the research papers that it could be built on top of, and so we're going to be taking things a little bit more seriously right after this intro. So just a quick catch-up on the memes and the hype. Sam Altman just yesterday posted I love somewhere in the garden, featuring a picture of strawberries. And the point is, OpenAI has this project internally that has been rumored, leaked, whatever you want to call it. It's called QAR, then it was renamed to Project Strawberry, and

that is where all the memes come from. So, from there, we have the user I rule the world Mo basically creating this intense hype train around project strawberry. He posts every other minute. He's gone from just a few thousand followers to now almost 12,000 in a matter of days. And for what it's worth, the memes are awesome, and everybody seems to be getting in on the fun. Jimmy Apple's, of course, replying to Sam Alman's post with less recommending more skitso strawberry posting. Sam Alman saying patience, Jimmy, again patience, Jimmy. And of course, he replies, Sam, please. So, this is in reference back to the last time he said patience, Jimmy, really telling us to hold our horses. They're cooking. We have the CEO of perplexity tweeting out a strawberry yesterday night. We have MC Hammer, the entire world would take notice, strawberry emoji. We have Run, the famous OpenAI leaker, posting a picture, SF is so back with a picture of strawberries at the center of a get-together in San Francisco. And of course,

Jimmy apples again. Google's corporate marketing can't compete with the online a non-strawberry religion. Yes, so we have I rule the world. We have Sam Alman at the top. We have Jimmy apples right there and a dead Google. And we have Gary Tan, the president of Y Combinator, even getting in on the fun. But now what we're here to talk about. Chubby on Twitter has posted a thorough breakdown of everything we know about strawberry. And if you don't follow Chubby, I really recommend it. He is a fantastic follow. He posts really technical stuff and really well-researched stuff. So please follow him. So that's what we're going to be doing today. We're going to be going through this very lengthy post and we're going to be breaking down everything we know about strawberry. So QAR has not been published or made publicly available. There are no papers on it. And OpenAI is holding back on information about it. So Sam Alman in this video says, We are not ready to talk about that. Let's just watch that video quickly. Can

you speak to what qar is. We are not ready to talk about that. See, but an answer like that means there's something to talk about. It's very mysterious, Sam. I mean, we work on all kinds of research. We have said for a while that we think better reasoning in these systems is an important direction that we'd like to pursue. We haven't cracked the code yet. Since the first hints, the community has been trying to figure out what qar might be. In this article, I compile all of the information I could find to paint a possible picture of qar. And of course, a lot of this is rumors, a lot of this is hypotheticals, but this is what we know so far. So let's find out. All right, so the beginning of this entire two-star strawberry drama about half a year ago, the information and Reuters learned from OpenAI employees at a scientific breakthrough had been achieved at the research facility. Although the first rumors emerged in December 2023, and the nice thing is he

also gives references to everything that he posts, so really well done. So basically, for the first time, a model had succeeded in learning by itself with the help of a new algorithm and acquiring logical mathematical skills without external influence, something that Transformer architecture cannot normally do due to their characteristics, as their results are outputs of probabilities. Logical thinking and self-learning are a necessity that many thinkers believe is a prerequisite for artificial general intelligence (AGI), and AGI therefore requires absolute correctness in its output in order to be transferred to all human processes, something that OpenAI itself repeatedly emphasizes in blog posts. And so here's a quote: In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still produce logical errors, also known as hallucinations. Mitigating hallucinations is a critical step in building aligned AGI. And he also references my video. Thank you, Chubby, for that. And by the way, I'll drop all of the links to my QAR videos and my strawberry videos in the

description below if you want to check those out. So, let's read what Reuters wrote, and this is back in 2022. Some at OpenAI believe QSTAR could be a breakthrough in the startup search for what is known as artificial general intelligence. Given vast computing resources, the new model was able to solve certain mathematical problems, though only performing math on the level of grade school students. Acing such tests made researchers very optimistic about QAR's future success, but conquering the ability to do math where there is only one right answer—that's really important—implies AI would have greater reasoning capabilities resembling human intelligence. This could be applied to novel scientific research, for instance. AI researchers believe—and so this is really an important quote from Sam Altman, and I remember watching this clip for the first time where he said this, and it's kind of mind-blowing just hearing this, so listen to this four times now—in the history of OpenAI, the most recent time was just in the last couple of weeks. I've gotten to be in the room. We sort of

push the veil of ignorance back and the frontier of Discovery forward, and getting to do that is the professional honor of a lifetime. Now, what he's talking about here, of course, we can't be sure, but a lot of people think it is QAR and this logical and mathematical ability of large language models. Now, at the same time, we know a lot of other things happened around this time and after this time. Ilia Sus tried to get Sam Altman fired. The board fired him. There was a mutiny. Then Sam Altman came back. We know that multiple super alignment researchers have left, stating that they're not getting the resources they need. There's a lot of memes around what did Ilas see. So, there's just a lot going on where maybe they discovered something and they're a little bit afraid of it. And then the information around that time also reported OpenAI made an AI breakthrough before Altman firing, stoking excitement and concern. One day before he was fired by OpenAI's board last week, he alluded to a recent

technical advance the company had made that allowed it to push the veil of ignorance back and the frontier of discovery forward, which we just covered. Jacob and Simon Sedor, two top researchers, used SS's work to build a model called QAR that was able to solve math problems that it hadn't seen before—an important technical milestone. So here's a small note from Chubby, which I think is super relevant to what we're talking about. The great fear and concern about QAR also stemmed from imagining that if QAR could already teach itself math without prior training—only elementary school level at first, but certainly more than that with enough compute—it could be at risk in the foreseeable future due to exponential development on all data encryption. What would stop an AI that teaches itself math from finding a solution to encryption if you just give the model enough time and compute? So basically, what he's saying is if all of a sudden AI is able to teach itself math, and if we throw enough compute at it, it's going

to get really good at math, and encryption is just math. That's all it is. And so, if a model gets good enough, then all of a sudden, it might actually be able to crack encryption, putting everything at risk: banks, nuclear codes, all of your passwords, of course, and really so much more. And it's interesting. I was just rewatching the show Silicon Valley, by the way, fantastic show if you haven't seen it. And in the show, and this is going to be a spoiler, towards the end of the entire show, they basically discover artificial intelligence mixed with some unique compression algorithm that is able to crack every encryption algorithm. And so, that's kind of how the show ends. So, it's interesting to see that even years ago, the creators of the show were thinking about this problem. But it's not just about reasoning and math. A lot of people, including myself, think that qar is more about long-term thinking. Right now, when you give a prompt to a large language model, you get back the first thing that they quote unquote think of, essentially. They

predict what the next token is, and they tell it to you instantly. There's no longer processing behind it. They just give it to you. And so, what qar and strawberry really could be is the ability for large language models to spend time coming up with a better solution, to think about it more akin to how a human thinks about things. When we're asked a question or we're assigned a research task, we don't just answer with the first thing we think of. We write things down, we research, we have deep thought about it, and then finally, when we're ready, we present our answers. And making large language models and AI in general perform like that could be a huge unlock. So, let's read a little bit more about what chubby says. One can say that qar is a method algorithm for approximating language models to human thinking and its conclusions. It is a method for mapping step-by-step thinking, iterative thinking, and thinking in process subdivisions, and adapting them to large language models. According to Conan, there are two thought processes, two systems in

which people think system one thinking and system two thinking. System one thinking is intuitive thinking, thinking that takes place automatically and intuitively at the moment. The large language models can only think in System One Way by outputting a probability result based on their training data. This corresponds with intuitive thinking. However, System 2 Thinking is complex thinking. It is the kind of thinking that involves thinking steps and process divisions. If we want to solve difficult mathematical problems, we cannot arrive at the results intuitively but have to approach the solution step by step. This is what we need to teach language models: to approach results slowly. So how do you actually teach language models to think in system two thinking? To start, he quotes himself, so let's read a little bit about what he says. Instead of laborious training, LLMs using RHF (reinforcement learning through human feedback), a model with QAR (query-answering reinforcement), is able to learn on its own. So this is very similar to how the AlphaGo system works. We didn't give it any feedback. It is the best at the world at

the game goes by far simply because it is able to play the game with itself over and over and over again, and it just gets better and better and better by itself. We gave it no instruction, so here in this example, GPT with Q-star taught itself math without any external intervention. So he also references AAR, which is a Monte Carlo tree search algorithm, and let's read a little bit about it. By dividing thinking into subprocesses, the model is given a sense of security. Solution finding using AAR is a search algorithm that finds the best solution. AAR is a graph traversal and pathfinding algorithm, which is used in many fields of computer science due to its completeness, optimality, and optimal efficiency. So given a weighted graph, a source node, and a goal node, the algorithm finds the shortest path from source to goal. The combination of Q-learning and AAR SE Arch teaches the model to think and find solutions independently and to correct itself. So think about this: it thinks about something, it moves forward, it predicts what's next, it kind of checks its work.

and is able to go back and try different paths to finding the most optimal solution. So, this means that the hallucination will stop, and correct solutions will be output as a result because the solution is not simply taken from the training data and is based on probability. However, QAR is likely to be very computationally intensive, and this is where GPT mini comes into play. My hypothesis is that OpenAI will use GPT mini to reduce energy requirements and save compute, and perhaps will incorporate a QAR variant into the small model. This is just speculation. So, because QAR and AAR have to go back and forth and do this longer-term compute, they are going to be incredibly compute-intensive, and that is why it's going to be so costly to run these models. So, if you have a smaller, much cheaper model using this technique, that might be the right way to approach it. So, System 2 Thinking is explained in more detail in a research paper by OpenAI. Let's verify step by step, published by Ilas Husk and

Yan Leakey, both of whom are exop AI at this point. So, the idea is already used today in prompts by telling a model to think step by step. And if you've watched any of my LLM test videos, you know that I put that in most of my prompts because we tend to get a much better response when you say think step by step or divide the task into subsections, which is, of course, only a superficial attempt to apply system to thinking. So, we can somewhat apply some of these techniques simply by structuring our prompts using some of these nudges to think in the right direction. He also talks about the tree of thoughts, which is another version of this that you can accomplish with prompts alone. And of course, there's the tree of thought, there's the chain of thought, there's a lot of different methods that we've talked about on this channel that you can apply just strictly through a little bit of programming, a little bit of prompt engineering. So, just a reminder, the tree of thought framework consists of four main components. There's the

thought process decomposition, basically breaking the problem into smaller manageable steps. There's thought generation, so giving a bunch of options to choose from, status evaluation, evaluating those options, and then the actual search algorithm, systematic exploration of the thought tree using algorithms such as breadth-first search or depth-first search, and referencing that paper in experiments with tasks such as the game of 24, creative writing, and mini crossword puzzles. Tree of thought showed significant improvements over conventional methods, and again, this is simply prompt engineering. So imagine if these techniques are built in natively to the model, that's when it becomes really powerful. So QAR is probably a combination of Q-learning and AAR search. Q-star combines elements of Q-learning and AAR search, which leads to an improvement in goal-oriented thinking and solution finding. This algorithm shows impressive capabilities in solving complex mathematical problems without prior training data and symbolizes an evolution towards General Artificial Intelligence. It is a fusion of Q-learning and AAR search. There is also an element of self-play, which we've

already talked about that is how AlphaGo works. Self-play is the idea that an agent can improve its gameplay by playing against slightly different versions of itself because it'll progressively encounter more challenging solutions in the space of LLMs. It is almost certain that the largest portion of self-play will look like AI feedback rather than competitive processes. There we have another term, look-ahead planning. It's the idea of using a model of the world to reason into the future and produce better actions or outputs. So we have a nice summarization at the bottom. QAR algorithm combines Q-learning and AAR search for improved goal-oriented thinking. We have process monitoring providing step-by-step feedback to improve model performance. The STAR approach generates rational thought with each token prediction and then overcoming hallucinations basically by process monitoring and STAR. It makes the decision-making more accurate and reliable. And what we will do with these is develop a more robust and reliable AI system. And a couple last things, OpenAI executives told employees that the company believes it is currently on the

first level, according to the spokesperson, but on the cusp of reaching the second, which it calls reasoners. So, that's what we have for today. That is a breakdown of what qar strawberry, a star search, what all of these things could be, and it's super fascinating to me. Hopefully, you enjoyed this. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

OpenAI Q* Strawberry - Everything We Know So Far

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

딸기 밈의 과대 광고와 프로젝트 분석

트위터는 샘 알트먼의 게시물과 소문으로만 떠돌던 OpenAI 프로젝트인 QAR(나중에 프로젝트 스트로베리로 이름이 변경됨)에 힘입어 딸기 밈으로 들썩이고 있습니다.

"나는 세상을 지배한다"라는 사용자는 이 프로젝트에 대해 큰 화제를 불러일으키며 빠르게 팔로워를 확보하고 있습니다.

트위터의 Chubby는 프로젝트 스트로베리에 대해 알려진 내용을 자세히 분석하여 게시했지만, OpenAI는 어떠한 정보도 공개하지 않았습니다.

OpenAI에서 개발한 새로운 알고리즘인 QAR은 인공 일반 지능(AGI)에 필수적인 자가 학습과 논리적 추론에서 획기적인 성과를 거둔 것으로 알려졌습니다.

이 모델이 초등학교 수준의 수학 문제를 푸는 데 성공하면서 AI의 추론 능력을 향상시킬 수 있다는 낙관적인 전망이 제기되고 있습니다.

OpenAI는 논리적 오류 또는 환각을 완화하는 것이 정합적인 AGI를 구축하는 데 있어 중요한 단계라고 강조합니다.

QAR과 AI의 자가 학습 능력이 미치는 영향

최고의 연구자들이 구축한 모델인 QAR은 사전 교육 없이도 스스로 수학을 학습할 수 있는 잠재력을 가지고 있어 향후 암호 해독 능력에 대한 우려가 제기되고 있습니다.

샘 알트먼이 OpenAI에서 해고된 시기는 이 회사의 AI 혁신이 보고된 시기와 맞물려 흥분과 두려움을 불러일으켰습니다.

AI의 수학적 능력의 기하급수적인 발전은 은행 및 보안과 같은 중요한 시스템에 위험을 초래할 수 있으며, 이는 프로그램 실리콘 밸리에서 다룬 주제를 반영하고 있습니다.

언어 모델에 인간처럼 생각하도록 가르치기

QAR(질의-응답 강화)의 개념은 단계별, 반복적 사고를 사용하여 대규모 언어 모델(LLM)이 인간처럼 사고하도록 하는 것을 목표로 합니다.

현재 LLM은 직관적이고 자동화된 시스템 1 사고로 작동하지만, 복잡하고 구조화된 문제 해결을 포함하는 시스템 2 사고를 가르치는 것이 목표입니다.

Q러닝과 AAR(몬테카를로 트리 검색 알고리즘)의 조합은 LLM이 스스로 생각하고, 해결책을 찾고, 셀프 플레이와 반복 학습 과정을 통해 스스로 수정할 수 있도록 돕기 위해 제안되었습니다.

AI 솔루션 최적화: QAR 및 GPT Mini의 역할

최적의 솔루션을 위해 다양한 경로를 탐색하는 QAR의 기능은 환각을 줄이고 올바른 결과를 제공하는 것을 목표로 하지만 계산 집약적입니다.

OpenAI는 에너지 요구 사항과 컴퓨팅 비용을 낮추면서 잠재적으로 QAR 변형을 통합하기 위해 GPT mini를 사용할 것으로 추측됩니다.

생각의 나무 프레임워크와 프롬프트 엔지니어링은 AI 모델에서 목표 지향적 사고와 솔루션 찾기를 개선하기 위해 시스템 2 사고를 적용하는 데 사용되고 있습니다.

셀프 플레이를 사용하면 AI가 약간 다른 버전의 자신과 경쟁하여 게임 플레이를 향상시킬 수 있습니다.

미래 계획은 세계 모델을 사용하여 미래를 추론함으로써 더 나은 의사 결정을 내릴 수 있도록 합니다.

QAR 알고리즘, 프로세스 모니터링, STAR 접근 방식은 함께 작동하여 모델 성능을 개선하고 환각을 극복하여 더욱 강력하고 안정적인 AI 시스템을 개발하는 것을 목표로 합니다.