Project Strawberry: Exploring OpenAI's QAR Algorithm

Project Strawberry symbolizes a potential breakthrough in AI, aiming to enhance reasoning and self-learning capabilities.

### Decoding OpenAI's Project Strawberry: A Potential Breakthrough in AGI Research
* 🍓  OpenAI's Project Strawberry, also known as QAR, has been shrouded in mystery, with **no official publications or public information available**.
* 🤖  The project is rumored to involve a new algorithm that enables AI models to learn **logical and mathematical skills without external influence**, a capability traditional Transformer architectures lack.
* 💡  This breakthrough could be a significant step towards achieving Artificial General Intelligence (AGI), as it enables AI to perform **multi-step reasoning without "hallucinations" or logical errors**.
* 🔍  The QAR algorithm is believed to combine Q-learning and AAR search, allowing models to think in a more human-like "**System 2**" manner, with **step-by-step thinking** and **process subdivisions**.
* 🧠  By enabling AI to think more **iteratively and systematically**, QAR could revolutionize how large language models approach **complex tasks and decision-making processes**.
* 💻  The computational intensity of QAR poses a challenge, but it may be mitigated by using techniques like **self-play** and incorporating QAR variants into **smaller, more energy-efficient models**.
* 📈  OpenAI executives have reportedly stated that the company is on the cusp of reaching the **second level, known as "reasoners,"** in their AI development hierarchy.
* 🚀  This potential breakthrough in AGI research has sparked **excitement and speculation** within the AI community, with many eagerly awaiting **further developments from OpenAI**.

Twitter has been on absolute fire for the last few days with strawberry memes. The hype is real, the memory is strong, and I know a lot of you come to this channel for the more serious stuff, and of course, that's going to be my bread and butter. But every once in a while, I like to join in on the memory and the hype, so I'm having a little fun here. Hopefully, you're having fun too. But in this video, I'm going to actually break down everything we really know about strawberry, what it could be, all of the research papers that it could be built on top of, and so we're going to be taking things a little bit more seriously right after this intro. So just a quick catch-up on the memes and the hype. Sam Altman just yesterday posted I love somewhere in the garden, featuring a picture of strawberries. And the point is, OpenAI has this project internally that has been rumored, leaked, whatever you want to call it. It's called QAR, then it was renamed to Project Strawberry, and

that is where all the memes come from. So, from there, we have the user I rule the world Mo basically creating this intense hype train around project strawberry. He posts every other minute. He's gone from just a few thousand followers to now almost 12,000 in a matter of days. And for what it's worth, the memes are awesome, and everybody seems to be getting in on the fun. Jimmy Apple's, of course, replying to Sam Alman's post with less recommending more skitso strawberry posting. Sam Alman saying patience, Jimmy, again patience, Jimmy. And of course, he replies, Sam, please. So, this is in reference back to the last time he said patience, Jimmy, really telling us to hold our horses. They're cooking. We have the CEO of perplexity tweeting out a strawberry yesterday night. We have MC Hammer, the entire world would take notice, strawberry emoji. We have Run, the famous OpenAI leaker, posting a picture, SF is so back with a picture of strawberries at the center of a get-together in San Francisco. And of course,

Jimmy apples again. Google's corporate marketing can't compete with the online a non-strawberry religion. Yes, so we have I rule the world. We have Sam Alman at the top. We have Jimmy apples right there and a dead Google. And we have Gary Tan, the president of Y Combinator, even getting in on the fun. But now what we're here to talk about. Chubby on Twitter has posted a thorough breakdown of everything we know about strawberry. And if you don't follow Chubby, I really recommend it. He is a fantastic follow. He posts really technical stuff and really well-researched stuff. So please follow him. So that's what we're going to be doing today. We're going to be going through this very lengthy post and we're going to be breaking down everything we know about strawberry. So QAR has not been published or made publicly available. There are no papers on it. And OpenAI is holding back on information about it. So Sam Alman in this video says, We are not ready to talk about that. Let's just watch that video quickly. Can

you speak to what qar is. We are not ready to talk about that. See, but an answer like that means there's something to talk about. It's very mysterious, Sam. I mean, we work on all kinds of research. We have said for a while that we think better reasoning in these systems is an important direction that we'd like to pursue. We haven't cracked the code yet. Since the first hints, the community has been trying to figure out what qar might be. In this article, I compile all of the information I could find to paint a possible picture of qar. And of course, a lot of this is rumors, a lot of this is hypotheticals, but this is what we know so far. So let's find out. All right, so the beginning of this entire two-star strawberry drama about half a year ago, the information and Reuters learned from OpenAI employees at a scientific breakthrough had been achieved at the research facility. Although the first rumors emerged in December 2023, and the nice thing is he

also gives references to everything that he posts, so really well done. So basically, for the first time, a model had succeeded in learning by itself with the help of a new algorithm and acquiring logical mathematical skills without external influence, something that Transformer architecture cannot normally do due to their characteristics, as their results are outputs of probabilities. Logical thinking and self-learning are a necessity that many thinkers believe is a prerequisite for artificial general intelligence (AGI), and AGI therefore requires absolute correctness in its output in order to be transferred to all human processes, something that OpenAI itself repeatedly emphasizes in blog posts. And so here's a quote: In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still produce logical errors, also known as hallucinations. Mitigating hallucinations is a critical step in building aligned AGI. And he also references my video. Thank you, Chubby, for that. And by the way, I'll drop all of the links to my QAR videos and my strawberry videos in the

description below if you want to check those out. So, let's read what Reuters wrote, and this is back in 2022. Some at OpenAI believe QSTAR could be a breakthrough in the startup search for what is known as artificial general intelligence. Given vast computing resources, the new model was able to solve certain mathematical problems, though only performing math on the level of grade school students. Acing such tests made researchers very optimistic about QAR's future success, but conquering the ability to do math where there is only one right answer—that's really important—implies AI would have greater reasoning capabilities resembling human intelligence. This could be applied to novel scientific research, for instance. AI researchers believe—and so this is really an important quote from Sam Altman, and I remember watching this clip for the first time where he said this, and it's kind of mind-blowing just hearing this, so listen to this four times now—in the history of OpenAI, the most recent time was just in the last couple of weeks. I've gotten to be in the room. We sort of

push the veil of ignorance back and the frontier of Discovery forward, and getting to do that is the professional honor of a lifetime. Now, what he's talking about here, of course, we can't be sure, but a lot of people think it is QAR and this logical and mathematical ability of large language models. Now, at the same time, we know a lot of other things happened around this time and after this time. Ilia Sus tried to get Sam Altman fired. The board fired him. There was a mutiny. Then Sam Altman came back. We know that multiple super alignment researchers have left, stating that they're not getting the resources they need. There's a lot of memes around what did Ilas see. So, there's just a lot going on where maybe they discovered something and they're a little bit afraid of it. And then the information around that time also reported OpenAI made an AI breakthrough before Altman firing, stoking excitement and concern. One day before he was fired by OpenAI's board last week, he alluded to a recent

technical advance the company had made that allowed it to push the veil of ignorance back and the frontier of discovery forward, which we just covered. Jacob and Simon Sedor, two top researchers, used SS's work to build a model called QAR that was able to solve math problems that it hadn't seen before—an important technical milestone. So here's a small note from Chubby, which I think is super relevant to what we're talking about. The great fear and concern about QAR also stemmed from imagining that if QAR could already teach itself math without prior training—only elementary school level at first, but certainly more than that with enough compute—it could be at risk in the foreseeable future due to exponential development on all data encryption. What would stop an AI that teaches itself math from finding a solution to encryption if you just give the model enough time and compute? So basically, what he's saying is if all of a sudden AI is able to teach itself math, and if we throw enough compute at it, it's going

to get really good at math, and encryption is just math. That's all it is. And so, if a model gets good enough, then all of a sudden, it might actually be able to crack encryption, putting everything at risk: banks, nuclear codes, all of your passwords, of course, and really so much more. And it's interesting. I was just rewatching the show Silicon Valley, by the way, fantastic show if you haven't seen it. And in the show, and this is going to be a spoiler, towards the end of the entire show, they basically discover artificial intelligence mixed with some unique compression algorithm that is able to crack every encryption algorithm. And so, that's kind of how the show ends. So, it's interesting to see that even years ago, the creators of the show were thinking about this problem. But it's not just about reasoning and math. A lot of people, including myself, think that qar is more about long-term thinking. Right now, when you give a prompt to a large language model, you get back the first thing that they quote unquote think of, essentially. They

predict what the next token is, and they tell it to you instantly. There's no longer processing behind it. They just give it to you. And so, what qar and strawberry really could be is the ability for large language models to spend time coming up with a better solution, to think about it more akin to how a human thinks about things. When we're asked a question or we're assigned a research task, we don't just answer with the first thing we think of. We write things down, we research, we have deep thought about it, and then finally, when we're ready, we present our answers. And making large language models and AI in general perform like that could be a huge unlock. So, let's read a little bit more about what chubby says. One can say that qar is a method algorithm for approximating language models to human thinking and its conclusions. It is a method for mapping step-by-step thinking, iterative thinking, and thinking in process subdivisions, and adapting them to large language models. According to Conan, there are two thought processes, two systems in

which people think system one thinking and system two thinking. System one thinking is intuitive thinking, thinking that takes place automatically and intuitively at the moment. The large language models can only think in System One Way by outputting a probability result based on their training data. This corresponds with intuitive thinking. However, System 2 Thinking is complex thinking. It is the kind of thinking that involves thinking steps and process divisions. If we want to solve difficult mathematical problems, we cannot arrive at the results intuitively but have to approach the solution step by step. This is what we need to teach language models: to approach results slowly. So how do you actually teach language models to think in system two thinking? To start, he quotes himself, so let's read a little bit about what he says. Instead of laborious training, LLMs using RHF (reinforcement learning through human feedback), a model with QAR (query-answering reinforcement), is able to learn on its own. So this is very similar to how the AlphaGo system works. We didn't give it any feedback. It is the best at the world at

the game goes by far simply because it is able to play the game with itself over and over and over again, and it just gets better and better and better by itself. We gave it no instruction, so here in this example, GPT with Q-star taught itself math without any external intervention. So he also references AAR, which is a Monte Carlo tree search algorithm, and let's read a little bit about it. By dividing thinking into subprocesses, the model is given a sense of security. Solution finding using AAR is a search algorithm that finds the best solution. AAR is a graph traversal and pathfinding algorithm, which is used in many fields of computer science due to its completeness, optimality, and optimal efficiency. So given a weighted graph, a source node, and a goal node, the algorithm finds the shortest path from source to goal. The combination of Q-learning and AAR SE Arch teaches the model to think and find solutions independently and to correct itself. So think about this: it thinks about something, it moves forward, it predicts what's next, it kind of checks its work.

and is able to go back and try different paths to finding the most optimal solution. So, this means that the hallucination will stop, and correct solutions will be output as a result because the solution is not simply taken from the training data and is based on probability. However, QAR is likely to be very computationally intensive, and this is where GPT mini comes into play. My hypothesis is that OpenAI will use GPT mini to reduce energy requirements and save compute, and perhaps will incorporate a QAR variant into the small model. This is just speculation. So, because QAR and AAR have to go back and forth and do this longer-term compute, they are going to be incredibly compute-intensive, and that is why it's going to be so costly to run these models. So, if you have a smaller, much cheaper model using this technique, that might be the right way to approach it. So, System 2 Thinking is explained in more detail in a research paper by OpenAI. Let's verify step by step, published by Ilas Husk and

Yan Leakey, both of whom are exop AI at this point. So, the idea is already used today in prompts by telling a model to think step by step. And if you've watched any of my LLM test videos, you know that I put that in most of my prompts because we tend to get a much better response when you say think step by step or divide the task into subsections, which is, of course, only a superficial attempt to apply system to thinking. So, we can somewhat apply some of these techniques simply by structuring our prompts using some of these nudges to think in the right direction. He also talks about the tree of thoughts, which is another version of this that you can accomplish with prompts alone. And of course, there's the tree of thought, there's the chain of thought, there's a lot of different methods that we've talked about on this channel that you can apply just strictly through a little bit of programming, a little bit of prompt engineering. So, just a reminder, the tree of thought framework consists of four main components. There's the

thought process decomposition, basically breaking the problem into smaller manageable steps. There's thought generation, so giving a bunch of options to choose from, status evaluation, evaluating those options, and then the actual search algorithm, systematic exploration of the thought tree using algorithms such as breadth-first search or depth-first search, and referencing that paper in experiments with tasks such as the game of 24, creative writing, and mini crossword puzzles. Tree of thought showed significant improvements over conventional methods, and again, this is simply prompt engineering. So imagine if these techniques are built in natively to the model, that's when it becomes really powerful. So QAR is probably a combination of Q-learning and AAR search. Q-star combines elements of Q-learning and AAR search, which leads to an improvement in goal-oriented thinking and solution finding. This algorithm shows impressive capabilities in solving complex mathematical problems without prior training data and symbolizes an evolution towards General Artificial Intelligence. It is a fusion of Q-learning and AAR search. There is also an element of self-play, which we've

already talked about that is how AlphaGo works. Self-play is the idea that an agent can improve its gameplay by playing against slightly different versions of itself because it'll progressively encounter more challenging solutions in the space of LLMs. It is almost certain that the largest portion of self-play will look like AI feedback rather than competitive processes. There we have another term, look-ahead planning. It's the idea of using a model of the world to reason into the future and produce better actions or outputs. So we have a nice summarization at the bottom. QAR algorithm combines Q-learning and AAR search for improved goal-oriented thinking. We have process monitoring providing step-by-step feedback to improve model performance. The STAR approach generates rational thought with each token prediction and then overcoming hallucinations basically by process monitoring and STAR. It makes the decision-making more accurate and reliable. And what we will do with these is develop a more robust and reliable AI system. And a couple last things, OpenAI executives told employees that the company believes it is currently on the

first level, according to the spokesperson, but on the cusp of reaching the second, which it calls reasoners. So, that's what we have for today. That is a breakdown of what qar strawberry, a star search, what all of these things could be, and it's super fascinating to me. Hopefully, you enjoyed this. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

OpenAI Q* Strawberry - Everything We Know So Far

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

The Strawberry Meme Hype and Project Breakdown

Twitter has been buzzing with strawberry memes, driven by Sam Altman's posts and the rumored OpenAI project, QAR, later renamed to Project Strawberry.

The user "I rule the world Mo" has created a significant hype around the project, rapidly gaining followers.

Chubby on Twitter has posted a detailed breakdown of what is known about Project Strawberry, but OpenAI has not made any information publicly available.

The Mysterious QAR and Its Potential Impact

QAR, a new algorithm developed at OpenAI, has reportedly achieved a breakthrough in self-learning and logical reasoning, which are crucial for artificial general intelligence (AGI).

The model's success in solving grade school-level math problems has raised optimism about its potential to enhance AI's reasoning capabilities.

OpenAI emphasizes the importance of mitigating logical errors, or hallucinations, as a critical step in building aligned AGI.

The Impact of QAR and AI's Self-Teaching Abilities

QAR, a model built by top researchers, has the potential to teach itself math without prior training, raising concerns about its ability to crack encryption in the future.

Sam Altman's firing from OpenAI coincided with the company's reported AI breakthrough, stoking excitement and fear.

The exponential development of AI's mathematical abilities poses a risk to critical systems like banking and security, echoing themes explored in the show Silicon Valley.

Teaching Language Models to Think Like Humans

The concept of QAR (query-answering reinforcement) aims to make large language models (LLMs) think more like humans, using step-by-step and iterative thinking.

LLMs currently operate in System One Thinking, which is intuitive and automatic, but the goal is to teach them System Two Thinking, which involves complex and structured problem-solving.

The combination of Q-learning and AAR (Monte Carlo tree search algorithm) is proposed to help LLMs think independently, find solutions, and correct themselves through a process of self-play and iterative learning.

Optimizing AI Solutions: The Role of QAR and GPT Mini

QAR's ability to explore different paths for optimal solutions aims to reduce hallucinations and provide correct outputs, but it is computationally intensive.

GPT mini is speculated to be used by OpenAI to lower energy requirements and compute costs while potentially incorporating a QAR variant.

The Tree of Thought framework and prompt engineering are being used to apply System 2 Thinking, with the goal of improving goal-oriented thinking and solution finding in AI models.

Self-play allows an AI to enhance its gameplay by competing against slightly different versions of itself.

Look-ahead planning uses a model of the world to reason into the future, leading to better decision-making.

The QAR algorithm, process monitoring, and the STAR approach work together to improve model performance and overcome hallucinations, aiming to develop a more robust and reliable AI system.