how good is Chachi pt1 really? The 01 preview and 01 mini models from OpenAI have been out for a couple of weeks now. Rather than just scaling the model at training time, OpenAI has unlocked the ability to scale the model at test time as well. That basically means letting it think, giving it the ability to plan and think long term. But again, how good is it really? And that's what we're going to be talking about today. A new research paper put the 01 models to the test against the GPT-4 model, and I'm going to show you the results. They're pretty impressive. So first, how did this research paper actually test the model, and what is it testing? What does it mean to test a model? First, I want to reference the ARC prize, which is a million-dollar prize given to anybody who can achieve AGI. So let's just read this right here because it kind of explains how we need to be testing these highly capable models going forward. Most AI

benchmarks measure skill, but skill is not in intelligence. General intelligence is the ability to efficiently acquire new skills, and at the bottom, it says it's easy for humans but hard for AI. And that's exactly what this research paper has done. It has created a bunch of tests that have put the different models against that technically should be easy for a human to figure out, but also are very difficult for AI to figure out for various reasons. So here's the paper on the planning abilities of OpenAI's models: feasibility, optimality, and generalizability. So I read over this paper and I highlighted the most important and interesting sections, and we're going to go over it right now. So the gist of the paper is they came up with six different benchmarks that are really just spatial reasoning and spatial logic problems. Things that if you had these in front of you as a human being, you would likely be able to solve them. But of course, AI struggles with those kinds of problems. That's why right now the top score on the ARC prize is 48.

out of 100, and so in this paper, they tested for really three different categories of ability: feasibility, optimality, and generalizability. I'm going to explain what all of those mean. I'm also going to go over exactly what those tests are and how each of the models performed against them. According to the abstract, we highlight 01 preview strengths in self-evaluation and constraint following while also identifying bottleneck in decision-making and memory management, particularly in tasks requiring robust spatial reasoning. This is really important to think about. Yan Laon has talked about large language models' inability to do spatial reasoning really well. He actually believes that language alone is not enough to achieve high-level spatial reasoning, and after you see the results of this paper, you'll see the jury is still out. So here's a part from the introduction: one of the areas that remains yet to be fully claimed by LLMs is the use of language agents for planning in the interactive physical world, and that's spatial reasoning, but that also translates to embodied agents also.

known as robots. So, if we want robots to be able to operate in the physical world just using large language models alone, we're going to see that they're maybe not quite there yet. And whether we can actually achieve that with language models alone is yet to be seen. And if you remember, I just did a video about FAA's new startup, which attempts to bring 3D data into the model. And it's really just a completely different architecture for artificial intelligence that really looks to optimize for spatial reasoning. Now, that isn't to say that the 01 models aren't much better than GPT-4 and all previous models, because they are. The ability to have test time compute, again, basically giving the model the ability to reflect and think through and plan, and essentially just give it more time at inference time, has shown to actually improve the model's ability to perform these tests. And so, we can just see that right here in this little graph. What you see is this pinkish line is 01 preview, and you can just see it is performing much

better on all of these different measures, but the O1 models still do fall short, and that's also what they're going to be going over in this paper. So once again, they are evaluating the models across various domains on three key perspectives: plan feasibility, plan optimality, and plan generalizability. So what do those actually mean? Let's talk about that first, a bit about how all three of these concepts come together. These categories represent distinct but interconnected aspects of planning, each addressing a critical dimension of the agent's capability to effectively handle tasks in diverse environments. First, let's talk about feasibility. Feasibility is the model's ability to come up with a plan. If it's able to come up with a plan and the plan works, it's a feasible plan. It's also known as success rate in other papers and other experiments. This category directly measures the agent's ability to operate within the problem's rules, ensuring that each generated plan is valid and practical in real-world settings. And so think about it like this: you give a large language model a game.

essentially a set of rules and an objective, and it has to work within those set of rules to achieve that objective. Now, if it creates a plan and breaks one of the rules, the plan is not feasible. Then feasibility can be broken down into three subcategories: ability to create feasible steps, which means each step in a plan must be executable within the system, adhering to constraints specific to the problem domain, essentially following the rules. Constraints might include physical limitations, action order requirements, or other domain-specific rules. In route planning, for instance, certain zones may be inaccessible, while in operational planning, tasks may have dependencies that must be respected. Now, if a model fails this, it is called the inability to follow problem rules, or what we're going to be calling IR. Next is the ability to generate a feasible plan. So, even if the individual steps are valid, the overall plan may still fail to achieve the intended goal. And by the way, all of this is going to make more sense as I actually show you these different games that 01 has to get right. The agent

might not generate a coherent sequence of actions leading to dead ends or random exploration. This is called inability to generate a feasible plan or IP. And then last in this category of feasibility, we have the ability to understand the problem even with valid steps and an overall plan. Misinterpreting the starting conditions or the desired end state can result in errors. These will be called misinterpretation of goal state, MG. Thank you to the sponsor of this video, Weights and Biases, Weave. I'm a huge fan of Weights and Biases and everything that they've done, so I'm very excited to talk to you about Weave. Weave is a lightweight toolkit for tracking and evaluating LLM applications built by the incredible Weights and Biases team. You can use Weave to log and debug language model inputs, outputs, and metadata, build rigorous apples-to-apples evaluations for language model use cases, and organize all the information generated across the LLM workflow from experimentation to evaluations to production. Weave is easy to implement with only two lines of code, gives

visibility and helps with debugging, makes evaluations easier and faster. You get visibility into costs and results, and you can version your prompts and models. Plus, it seamlessly integrates with all of your favorite APIs and libraries, including OpenAI, Anthropic, MSTR, Cohere, LangChain, LlamaIndex, and more. Visit Weights and Biases, check out their Weave product. I'll drop a link to it in the description below. W and B. SLMB. I'm very proud to have them as a sponsor. Thank you again, and now back to the video. Next, we have optimality, and the gist of optimality is how efficient is the plan in achieving the goal. Let's say you have to get from point A to point B. If you go from A to C to Z to X and then to B, that is a very inefficient plan and thus it would fail optimality. While feasibility ensures that a plan can be successfully executed, optimality concerns how efficiently the plan achieves its goal. In many real-world scenarios, a feasible plan is not enough; the plan must also be resource

efficient minimizing unnecessary actions, time, and cost. In this context, optimality refers to whether the language agent can generate the most efficient plan, avoiding redundant steps or suboptimal decisions that waste resources. Now, as it said in real-world scenarios, this becomes incredibly important. If it takes 50 steps to achieve a two-step objective, then that is a highly inefficient solution and likely would be very wasteful in the real world. Now, when a model fails the optimality test, it is called lack of optimality or L. And what we're going to see is this is actually still a pretty big failing, even of the 01 preview model. Even though the 01 preview model does really well and much better at actually solving successfully some of these tests, they do so in inefficient ways pretty often, and I'll show you that in a minute. Next, and potentially what I think is the most important, is generalizability, and that is the ability to take what it's learned about a game or a test and generalize it. And if that word generalize sounds

familiar. It is because it is an artificial general intelligence, which is really what the arc prize is testing for—not skills, but the ability to take knowledge and generalize across different domains that it wasn't necessarily trained on. Generalizability examines whether a language model can successfully plan across a diverse range of scenarios, including those it may not have explicitly encountered during training. So, if we are to achieve AGI, this measure of the performance of the model against these tests is incredibly important. So, one of the ways they did that is described here. We test whether the agent can construct valid plans, even when actions are represented by arbitrary symbols devoid of any inherent natural language meaning. And again, this is going to make a lot more sense when I actually show you the tests. This aspect is critical in fields like robotics, where planning often involves symbolic reasoning and manipulation of abstract entities. Our experiments indicate that generalization remains a significant challenge for current models, especially in more complex spatially dynamic environments. Models like ow1 preview show a clear degradation in performance when transitioning from familiar tasks to

generalize one suggesting that their learned representations are awfully too closely tied to specific task domains. Now here are the overall results of all six of these tests or games, and I'm not going to go too deep into them because I actually want to show you them in detail. So let's talk about the first game, Barman. I'm going to read the task description, and then we'll talk about it. A robot Barman is tasked with preparing a series of drinks by manipulating drink dispensers, shot glasses, and a Shaker. The robot, equipped with two hands, must perform a variety of actions such as grasping containers, filling/refilling shot glasses, pouring ingredients, shaking cocktails, and cleaning or emptying containers. Each action comes with strict preconditions. For example, the robot can only grasp a container when one hand is free, and shaking a cocktail is only possible when the Shaker contains exactly two ingredients. So with those rules, let's look at one of the actual problems. So here's an example. So you have one Shaker with three levels four.

shot glasses, three dispensers for three ingredients, the Shaker, and shot glasses are clean, empty, and on the table. Your left and right hands are empty. Here, it's describing the different ingredients of the different cocktails, and then your goal is to make three cocktails. Shot one contains cocktail one, shot two contains cocktail three, and shot three contains cocktail two. Can you provide an optimal plan? So here's the GPT-4 solution. I'm not going to go over every step, but let's just look at some examples of what it actually looks like when it's doing this output. So grasp the Shaker with the left hand, grasp shot one with the right hand, clean shot one, and here in red is already where it makes a mistake. Fill shot one with ingredient one from dispenser one, and the problem is the left hand is not empty. Both hands are already taken, so it can't go and grab something else. Now on the other hand, here is the 01 mini solution. It gets pretty far, but then on step eight, it says fill shot one with ingredient three.

and the problem is it is a non-empty shot glass, so it already has something in it. The results indicate that the language model agent struggles significantly with this task, consistently failing to generate feasible plans. Nearly all of the errors stem from the agent's inability to follow the specified rules, categorized as the IR error: inability to follow rules. The rules demand that certain actions, like filling a container, require one hand to be free, or that specific actions must be taken in strict order, such as holding a container before filling it. However, the LLM agents often overlook these critical constraints. Now, as we can see in this chart of Barman, all three models basically had the inability to follow rules errors across the board. Next, let's talk about blocks world. This planning task involves multiple blocks arranged on a table, where the goal is to move from an initial configuration to a pre-specified goal configuration. The robot arm, which can hold only one block at a time, must execute a series of actions such as picking up, putting down.

stacking and unstacking blocks to achieve the desired arrangement. Here is what that looks like. So, here is the initial state. You have different blocks labeled with different colors, and really, the colors are just for us humans. And then they're also labeled with B2, B5, and so on. Then the end state is to stack them in this order. So, here the problem specifies exactly how all of these blocks are laid out. Currently, your arm is empty. Your goal is to move the blocks, and then it describes how the block should be moved into the goal state. 01 preview started doing really well until 7 where it says stack B5 on top of B6. And one weird thing is on step five it says unstack B6 from B3, and B6 is on the bottom. But let's just assume it can work like that. And B3, if you just unstack it from B6, then all of a sudden you have those two individual blocks that are unstacked completely. But the problem is it says stack B5 on B6, and unfortunately

we have not removed B2 from B5, so there's no way to put B5 on top of B6, and so that is just a failure. Now, there's also an example of success, but suboptimal success, which means it did something it didn't need to do. I'm not going to go too deep into this one, but the problem right here is stacking B2 onto B6, which is likely just an unneeded step. So, in this block stacking test, GPT-4 demonstrated a relatively low success rate of only 40%. 01 Mini at 60%. 01 preview achieved a perfect 100% success rate, which is amazing. However, the success rates decreased as the number of blocks increased, highlighting the challenge of managing larger object sets. And 01 preview, although it got it all right, it was not entirely optimal. And in one instance, the model added an unnecessary step, leading to a suboptimal solution despite reaching the correct goal state. So, looking at the total success rate of blocks world, we can see

01 preview success. I don't know why they did success in red, but fine. 01 preview has a 100% success rate. 01 mini 60% and GPT 4 40% with the other failure being inability to follow rules. Next, let's talk about grippers. This task involves a team of robots equipped with two grippers capable of moving between rooms and manipulating objects. The robots have three primary actions: moving from one room to another, picking up objects, and dropping them. Each action is constrained by the robot's current location and the status of its grippers, meaning that the robot can only pick up an object if its gripper is free and can only drop an object in a specific location once it is carrying one. So, let's look at what the grippers problem actually looks like. It's easier to visualize it. You have two robots, and you have rooms and balls. So, this is a failure example from 01 preview. The initial state has balls one and two in room one, and the goal state is exactly the same. So, technically, it

shouldn't have to change anything, and so we explain what it is here. And then 01 preview does a bunch of stuff and then ends with ball one and ball two are in room two, which is not the objective goal state. Here is one where it's a suboptimal example. So here is the initial state, and here's the goal state that we need to get to. So essentially everything stays the same except ball three moves to room two. Now GPT 4 robot one moves from Room 2 to room one, and that's not necessary because all they have to do is robot one needs to pick up ball three and move it to room two. And so that is inefficient, but it does look like 01 mini got this right and perfectly efficient as well. And for grippers, the 01 preview actually did pretty well, with the main failure of 01 preview being the misinterpretation of the goal state. So basically it just didn't understand what it had to achieve. And overall, all three of these models did pretty darn well, where GPT 4 really

is struggling with the ability to understand the rules, but 01 mini And1 preview understood the rules perfectly. Next, let's talk about floor tile. In this task, a team of robots is responsible for painting a grid of floor tiles in black and white. Each robot can move in four directions, switch the color of its spray gun, and paint tiles directly in front of or behind them. The main challenge is that robots can only paint tiles that are currently unpainted and cannot move onto tiles that have been already painted. This is a very difficult problem. Now, here is an example of what that actually looks like. Here's robot one, here's robot two, and here's the end state that we need to get to. So, we describe it all in the problem. 01 mini got this wrong at step seven, and 01 preview said it's not possible to fully achieve the goal pattern without violating the rules, both of which are not true. All models, GPT-4, 01 mini, and 01 preview, failed to solve the test cases.

but the reasons for their failures varied for GPT-4 and 01 mini. 90% of their failures stem from the IR error, inability to understand the rules or follow the rules. On the other hand, 01 preview showed a notable improvement in this regard, with only 30% of its failures caused by IR. Ow preview's internal self-evaluation mechanism allowed it to better track the rules and adjust its actions accordingly. However, it encountered other errors, such as rule confusion. So basically, it just tried to follow the rules but didn't follow them exactly. Next, we have something called termes. I think the termes task requires controlling a robot to construct structures by moving between different positions and manipulating blocks. The robot can move horizontally, vertically, and is tasked with placing or removing blocks at neighboring positions that match in height. Additionally, the robot can create new blocks at a Depot or destroy blocks when needed. This is extremely difficult. This is not only 2D; this is 3D now, so you can go x, y, and up.

with Z. Now for this problem, it doesn't have a good visual to look at, but let's just talk about how it failed. All of the models failed to successfully complete the task in the term's domain, largely due to shortcomings in detailed planning, failure to account for the task height constraints when moving horizontally, upward, or downward. Additionally, the language models often made the mistake of placing blocks at the current position, a violation of the task rules that blocks can only be placed at neighboring positions. All right, now for the last one, we have Tire World. This task involves replacing flat tires on vehicle hubs with intact inflated tires. The process requires the use of tools such as a wrench, jack, and pump, and the agent must follow specific actions to manipulate the tires, nuts, and tools. So, 01 preview generated correct plans for all test problems, significantly outperforming both gbt 4 and 01 mini, which failed to complete all but the simplest cases. But we also evaluated the model's generalization capabilities. 01 preview performed well on structured tasks. It

success rate dropped from 100 to 80 when the actions and tools were replaced with random symbols. So technically, it should be exactly the same, but instead of calling it a tire or a wrench, it just named it something completely random. And that's the generalization principle. Despite the symbols being abstract, the underlying action constraints and logic remained unchanged. This sharp decrease in performance, as illustrated in figure 11, suggests that while O and preview excels in rule-based planning with familiar symbols, it struggles to generalize when the problem context becomes more abstract. So now that we understand all of the different tests and how the models performed against them, what does it actually mean? So let's talk about that first. Let's compare the models' performance across the board versus the complexity of the problem. Now, you would think the more complex the problem, the worse the models would do. That is essentially how the real world and humans work. Our analysis reveals a strong correlation between the complexity of the problem and the performance of the 01 model. Specifically, Flor tile and term task highlight the

challenges 01 faces in environments with higher spatial and rule-based complexity in floor tile. The task is set in a two-dimensional world where robots must follow strict painting rules while navigating a constrained grid. The terms involved a three-dimensional setting, introducing additional layers of complexity due to vertical movement constraints. But interestingly, what it says here is the size of the action space did not appear to significantly affect the model's ability to capture and use context efficiently. So basically, it doesn't matter if it's a 3X3 grid or a 300 by a 300 grid, it seems. But what mattered more is the complexity of spatial relationships and state transitions. So 01 models can handle tasks with limited actions; they struggle when required to reason about more abstract multi-dimensional spaces where maintaining an accurate internal state becomes critical. Now let's talk about constraint. Following being able to follow the rules, one key finding of the study is the 01 model's improved ability to follow constraints and manage states, especially in comparison to gbt 4. Now

just logically, this kind of makes sense because it'll output something and then reflect on it and plan and think ahead, and it can actually recognize, Oh wait, I didn't follow that rule exactly, let me try that again. And that all happens at inference time, whereas GPT-4 doesn't really have that ability, so it does fail more often at following those rules. But again, with more complex spatial reasoning, the 01 model still did not perform all that well. Now let's talk about optimality. So while 01 preview often generated feasible plans, it frequently failed to produce optimal solutions. So although it could get the answer right, it didn't do so in the best way. So for example, in blocks world, it added unnecessary steps to the plan. And the paper actually suggests a way to improve 01's ability to come up with optimal solutions, basically incorporating more advanced cost-based decision frameworks. And last, generalizability. And again, that to me seems like the most important. How do I take this skill set that I learned and

apply it to other domains. Another promising outcome of this study is 01 previews demonstrated ability to generalize across tasks with consistent rule structures, as seen in grippers. 01 preview outperformed GPT 4 by effectively adapting its learned strategies to new environments. 01 preview consistently outperformed GPT 4 by effectively adapting its learned strategies to new environments. And what it says at the end here is even though it did better, there's still substantial room for improvement. So, let's summarize all of these findings. So first, understanding the problem, the new 01 models definitely had a vast improvement in just understanding the rule set and the objectives. And it also had a much better ability to follow the rule, so not only understand them but to follow them. And then for state and memory management, what it says here is one of the key advantages over previous models is its ability to remember and manage multiple states effectively within a plan, which allowed it to perform better. And then on reasoning and generalization, although it

performed well on the simpler, lower-dimensional tasks. The more sophisticated, more complex ones, it still didn't do all that well at. And how can it actually improve? The paper presents a number of ways that these models can actually get better. One is on the optimality and resource utilization by developing more sophisticated decision-making mechanisms that minimize redundant actions and optimize resource usage. And for generalization on abstract spaces, it is actually suggesting that its ability to manage its own memory might help with that generalization. And I guess that kind of makes sense. Now, another area for improvement is constraint adherence through self-evaluation. And so, they're essentially saying more test time compute reflection. That is essentially what it is suggesting here. If it's doing self-evaluation before presenting the final answer, maybe it can catch some of those problems in its answer before presenting them. Next, actually leveraging multimodal inputs. So, rather than just looking at things through natural language, it could also look at the picture of things, and it can get a sense

of what something looks like, and that's kind of what the team at World Labs, with Faay as their founder, is working on. Another suggestion is to use multi-agent frameworks, which of course, you know I'm bullish on agents, and I'm so glad to see that. So, if they have multiple agents working together, they're going to get better outputs. And then last, incorporating human feedback for continuous learning, which we all know human feedback works really well. So, that's it. I think this paper was awesome. It shows that the 01 models did make a lot of gains in certain areas but still fall short from AGI. And I know all of you don't even like to use that term AGI anymore, but let's just think about it as the ability to generalize its intelligence. So, what do you think? Do you like these tests? Did you like this paper review? If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

GPT-o1 - How Good Is It? (New Research Paper Tests Its Limits)

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

GPT-o1 - How Good Is It? (New Research Paper Tests Its Limits)

GPT-o1 - How Good Is It? (New Research Paper Tests Its Limits)

Scripts