Okay. So back in December OpenAI
released o1 and this was proclaimed as the first reasoning model. And we saw them make a
lot of claims about this. We saw them release the
o1 mini and the o1 preview.

And supposedly in the next couple of
weeks, they're going to release the actual full o1 model, which is going
to be interesting when that comes out. In this video, what I want to look
at is exactly how fast is open source

catching up because literally in the
past week or two, we've seen a number of different companies release open weights
versions of these kinds of models. And it's interesting to see actually how
close to they come to the OpenAI models.

And what I want to investigate is how
much of an edge does OpenAI actually have at the moment, especially if
all these Chinese open weights models are breathing right down their neck. So just to recap a little bit, OpenAI
basically released this model and they

released it with a bunch of new different,
evals and benchmarks at the time. Probably the big ones that they
focused on were these three main benchmarks of this competition math. Which I think is aimed at high school
students trying to get into the math

Olympiad, code forces, a code one. And also a Google proof PhD level
science questions and Google proof here basically means that even if you
gave people the access to Google, the chances of them getting the right answer
are still going to be very low here.

Now, of course at the time OpenAI didn't actually say how this model was done. One of the key things obviously was that they were using some kind of test-time compute because the inference generation was quite different based on the sort of difficulty of this.

And if we kind of scan through some of the Twitter posts from the people who worked on this, we could certainly see that chain of thought and training chain of thought into the model was a really key factor. So just to revise a bit, up until recently a standard LLM was created with

a very large amount of pre-training. So we're talking about sort of web-scale pre-training that it's just doing autoregressive prediction of the next token on, in there. And then it would basically have some kind of post-training.

And traditionally up until now, most of the post-training that people have talked about publicly has been some kind of supervised fine-tuning or instruction tuning followed by reinforcement learning, whether that's RLHF, so reinforcement learning from human feedback, or RLAIF, where it's from AI feedback in there.

And then at inference time, you didn't use a lot of compute. You would just do a forward pass through the model. Now this new way of creating reasoning models, really what you're doing here now is you still have the sort of pre-training and post-training, but

your post-training will be quite different in that you might have some kind of self-play or RL version in predicting out and scoring what would be the next chain of thought or reasoning trace to come out of this? And we saw this very clearly in a paper that actually came out

from OpenAI a year and a half ago. And this is what led to the whole concept
of this strawberry model, et cetera. This idea of let's verify
step-by-step as we go through and you can use RL to do that.

But the big difference with this often
is going to be at inference time. So at inference time, you're
going to produce these reasoning traces or trees out. And you could imagine that you've
got something that scores them where you're going to keep the best ones and
you're not going to use the other ones.

And gradually building up over these
reasoning traces you then come to the right conclusion in the end here. As I mentioned before OpenAI had
some key papers on this before. So back in 2021, they talked about
training verifiers for this kind of thing.

In 2023, they had looked at the whole sort
of thing around the verify step-by-step. And even people who had worked
on the original chain of thought paper, like Jason Wei have
moved from Google to OpenAI now.

So this whole idea of using chain of
thoughts for, this sort of reasoning trace thing is definitely something
that we know a lot of the researchers at OpenAI have been very interested in
doing. But the interesting thing when the O1 models came out, or rather the O1 mini
and the O1 preview came out, was that

OpenAI made out that this is something
that was really difficult to do, required a lot of different styles of training. Obviously, it requires a lot more
compute for the inference, et cetera. And the general consensus that they
were pushing was that this is something

that they would be able to do. And only other sort of proprietary model
companies would be able to catch up. In fact, if we ask the o1 preview
itself, how long before open source is able to reproduce OpenAI's o1 models, 

we actually see that it's kind of saying
that, this open source community typically lags behind the state of proprietary
models by one to two years, in here. And it might take anywhere
between 18 months to three years. So that brings us to the open models
and over the space of the last 

two weeks, we've literally seen
three different AI labs come out with their own reasoning models. So in this video, I'm going to look
at the DeepSeek R1 lite preview. Which they've got up
for use on their site. 

And they're saying that They're
going to release the weights. We'll look at the Qwen QwQ,
model, which again, the weights are already out for this. And we'll look at the Marco o1 model
which comes out of Alibaba international. 

Now all three of these are all
coming out two to two and a half months after the OpenAI release. This is not 18 months later. This is not three years later. This is not even sort of nine months
to 12 months later, like GPT-4 was. 

We're seeing this acceleration in the
more open lab models that are out there. Let's jump in and take a
look at some of these models. See what they can do. Let's see actually how close they
come to o1 preview, to o1 mini. 

And then my guess is that the ball's going to be back in OpenAI's court, in the next week or two for them to release the full o1 as the open weights labs are basically catching up really quickly. All right.

Let's jump in. Okay. So the first one that we'll look at is the DeepSeek R1 light preview. So I think this is the first one that was released, about a week and a half ago now. And we can see that they're running on the same benchmarks as OpenAI for this.

So their benchmarks are showing that they're doing substantially better than the o1 preview in here. I don't think OpenAI actually released benchmarks for the o1 mini for all of these. So I think basically people have taken just the o1 preview benchmarks.

Now, they're getting 52 for this. And my guess is that this is like a one-shot thing. And so yes, the o1 preview was getting 44 for this. The o1 preview with more samples where they were doing sort of

consensus voting was getting about 56. And I think the o1 they're getting at the peak around 83. So it's going to be interesting to see when the o1 comes out here, but clearly this is a good result.

Again, with the math benchmark here. Interestingly for the PhD level questions, they're actually quite a bit behind where o1 preview is. So I wonder if this is just to do with o1 preview using a bigger model, et cetera.

We should say that the O1 score is, and I'll just flash these up here. The O1 as opposed to the O1 preview is supposedly being given a lot more test time compute for answering this. Now we don't know how long those answers are going to be.

Is it going to take 15 minutes to be able to get to the end? All of those things are still open questions at the moment. Another nice thing that the DeepSeek people have done is, put together something where they actually show what happens if they increase these reasoning traces.

So you can see on the X axis here, we've got the average number of thought tokens per problem. And basically the longer those thought tokens are, the better the accuracy is here, which is not surprising. And this is one of the reasons why the O1 model may actually just do so

much better than the O1 preview is that it just generates much longer chain of thought traces here. Now remember OpenAI is not giving us those train of thought traces. We can't actually see what they are. With these open models, they are actually making that available

for you to be able to look at, learn from, see what's going on. See how the model thought through the problem, et cetera. So at the time I'm recording this, DeepSeek hasn't actually open sourced the model weights or the APIs, et cetera.

One of the cool things though, is DeepSeek actually does serve their own models. So you will be able to just put a few dollars into their account and be able to use this as an API with the full size version, that kind of thing.

Now, if we jump into their chat interface,
where we can actually play with the model you can try this out and you can see,
how these things are working, et cetera. So what I wanted to show you is one of the
first things that I started off with these

things is trying to trick it, obviously. So the common thing is
counting Rs in strawberry. But if you notice very carefully here,
I've deliberately spelled strawberry wrong. So I've got four Rs in there. So DeepSeek allows us to basically
turn on this deep think, which is

the sort of reasoning version of it. Or not have it on. And you can see these first examples
I've started out with it off. So when I ask it here, how many Rs are
in strawberry, it just comes back three.

And you could imagine that nowadays
there's so much talk on the internet about how many Rs in strawberry that even, any
model that's been pre-trained on recent internet data, it's probably going to get it right

just because people have talked about it. So this is deliberately trying to
confuse it and sure enough it's it comes back and says that there were
three Rs when clearly there are not three Rs, there are four in there. All right.

Next one is another common sort of
logic question that people use a lot. Sally has four brothers. Each of her brothers has three sisters. How many sisters does Sally have
now presuming that Sally's female. We could guess that she's going to
have two, but here we get three.

so she has three sisters
including herself, right? Now I want it to show you that if we turn
this deep think on and we ask it this question, now we start to see that, okay,
it actually starts to go through this.

Now, they don't have any sort of markdown
or anything that's marking out the different chains of thought in here. So we've just got a stream of sort
of chain of thought of going in. Now, my guess is that each of these
are divided up into sort of sub

chunks of where it's thinking about
the problem in different ways. So first off, quite commonly, you want
it to rephrase the problem to itself. So you can see that
Sally has four brothers. That part seems straightforward. So there are four male
siblings in the family.

Sally, now it says
that, has three sisters. So I'm thinking about one
brother, he has three sisters, but Sally is one of those sisters. So does that mean that there
are only three sisters in total?

And you can see that it goes
through and it does some, in a nice thinking, is it's going through
this to come out with this answer. And you can see now that it
comes back with a conclusion saying Sally has two sisters.

And so that would fit with the whole thing
of Sally plus her two sisters are each of her brothers' three sisters in there. Alright, next back to our, how
many Rs in strawberry with that the strawberry spelt wrong here. So you can see here that, I need to
figure out how many Rs are in it.

Let's see. I should look at the problem. And so the first thing it does is it actually spells it out with hyphens in between. Now this is pretty impressive because what it's doing here is actually using the tokenizer to split these up rather than have them as one thing.

This is where a lot of the models would go wrong is that they've already got a token for this for wrapping it and they would just stick to that token. But we can see that okay, this is managed to split it up and then it

can go through it letter by letter. And then you could see that it comes back and says, right positions 3, 4, 9, 10 are Rs, that seems like 4 Rs, but wait, let me make sure that

I didn't miss or count any extra. So it actually does a check and it does this sort of thinking of where it's going to go through and circle each of the Rs as it goes through. And then it comes back finally with the answer, there are 4 Rs in strrawberry.

Again, remember that I've deliberately spelled this wrong to see okay, will it be able to pick that up? So that was very impressive with this model for me. The next thing is you can try that with different kinds of logical

statements and stuff like that. One of the ones that impressed me, I'll just scroll down. You can see that with this one it did quite a lot of thinking, but one of the ones that impressed me is the question that I often use for checking sort of thinking is basically

to ask it about something in the
past, if something had been different, how would that have affected things? So here, I'm saying, explain the
consequences of what would have happened in World War Two, if
the nuclear weapon wasn't used? 

So you can see here that it will, we'll
start off and say, okay, I'm trying to figure out rephrase it to itself
again, and try and figure out what would have happened in World War Two,
if nuclear weapons hadn't been used. 

First, I need to understand
the role of nuclear weapons played in the actual outcome. And then it's going through that. And then it goes through the
different thinking sort of lines of, okay, what are some of the
options that could have happened? 

So another option might have
been an invasion of Japan. One of the things that I think is
fascinating here is it even goes on to think through the, okay, what
would have the Soviet Union done here? And had the Soviet Union attacked
from the top that you may have had a 

situation where, Japan was split in two,
similar to Germany was split in two. Now this kind of thing, there is no
really wrong answer or right answer. But for me, this is really interesting
to look at how it reasons to itself about 

the different things going through here. So it ends up with these conclusions of you know, a continuation
of conventional warfare, possible invasion of Japan. It talks about the political
and diplomatic implications. The role of the Soviet Union, the
economic impact, a whole bunch of 

different things where it's coming at
this problem from different angles. And I think this is one of the key
things of these really good chain of thought traces is where you can get it
to focus on the same problem in many different ways going through there.

After this, I tried a few different
things around composing a piece of music with prime numbers. Also very interesting. I did it quite a long
thinking process for that? And then asking it for things like,
I have a certain number of coins if I

group them in certain ways, work out. What's the lowest number
of coins that I can have? Now, interestingly, this
one took 23 seconds for the inference coming out of this. And if we look at the OpenAI o1 preview,
it was able to come up with the exact

same answer, but in five seconds. So I do wonder how relevant is that going
to be when we look at this in the o1? Is it basically a lot more
efficient in its way of generating a lot of these traces.

Now you could imagine that DeepSeek,
if I understand correctly, they only have 10,000 GPUs and I say
that only in inverted commas there. But that's certainly way less than,
most of the sort of top labs have.

So perhaps for inference,
they're not using as big a stack as OpenAI is et cetera. All right, let's jump on and
let's look at the next model. All right, so the second
model comes from Qwen.

These guys are really on a roll. They've had the coding model, which I wanted to do a video about, I just didn't get around to it. So they've had a Qwen 2.5 30B coding model, which has done really well.

And I feel that a lot of their models are the sort of go-to default now for open source models for a lot of things. So they've introduced this sort of reasoning and reflect deeply, idea here. Now they talk a little bit about it, that it's got these recursive

reasoning loops going on here. They talk about some of the limitations, et cetera. If we come down and look at their benchmarks. So their benchmarks are kind of interesting in that, on the GPQA, they're

actually beating DeepSeek quite easily. Now, they're not getting as high as the o1 preview, but they're certainly beating the DeepSeek model. But for the AIME one, or the sort of for the Math Olympiad one, they're actually

a bit below the DeepSeek R1 model in here, but in this case they're beating the OpenAI o1 preview and both them and DeepSeek are being beat by the o1 Mini. So it really does show you that it's probably going to be really

hard to benchmark these things unless you're giving it the exact same amount of compute. How many iterations do you allow it to think about the thing before it comes to a final answer, versus you could imagine that the o1 model, when it

comes out, is just going to be really
much longer for compute and therefore probably more expensive, et cetera. Now on the live code bench, they're
a little bit behind, the DeepSeek R1. But let's jump in and test it
on the same questions that I

gave the previous model before. Okay, So I'll start off with the Sally
has four brothers problem. And we've certainly come a long way from where
the early LLMs would basically work out four times three, or give that
an answer, or, that kind of thing.

here we can see that it's
got quite extensive sort of, thinking as it goes through. Now, there's steps in here, There
are little sort of steps in here. Now, I'm not sure, again, we're missing
Markdown, so we don't see where,

each chain of thought trace actually
starts and begins, going through this. but you can see that it comes down and
it actually doesn't get this, it comes back with that she has three sisters. So on this one, it's not doing as good.

If we try the strawberry one,
where we're deliberately spelling strawberry with four Rs, you can
see, going through this, it comes out and gets that the answer is four. Now, interestingly, each time I've
run this, and I've run these multiple

times, each time I've run this, it
gets slightly different thinking. And sometimes what it will say to itself
is, no, no, no, no, Strawberry's supposed to have three R's, I need to check again.

and it will go back through and
check that, again, only to realize that there are actually four R's
in the way that I've spelled it. So they, it will often give an answer
where it says, you may have mistyped

this, which in case it would be
three, but the final answer is four. Another thing that we see sometimes,
which I think is really interesting in here, is that, and you can see this is
a different run of the same question,

is you can see that it gets to the four
Rs, and it gets the positions and stuff like that. But in this case, it gets into
a loop where it starts going through this loop and it's saying, alternately,
perhaps the question is a trick, right?

And make me think that there are
more Rs than there actually are. And then perhaps the answer is
four, and I'm overthinking it. And so it goes through this loop. Now, this loop goes on and on and on.

You can see that where it's
basically saying, okay, the answer is 4, I should accept that. The answer is 4, I don't
need to stop overthinking. The answer is 4, I should just write that.

The answer is 4, I should conclude here. The answer is 4, I should
finalize my response. This is perhaps a bit scary, right? That, that it's gone into this
loop and you can see that the loop just goes on and on and on.

Till eventually, I think we, as you can
see down here, it's still generating out, as I'm going through this. So this has been going, I think,
for a few minutes at this point, and it's still looping through, and it
seems like it can't come to a final

conclusion, that it's going back through
the loop again and again and again. I will leave it and maybe come back
to that and see what's going on there. Okay, so the last question that I wanted
to try was the World War II question

if nuclear weapons hadn't been used. For me, I guess this is a much
more sort of subjective thing, but it's interesting to sort of look at
how it groups the traces together. So you can see, again, we've
got first I need to understand

the context leading up to. So you've got this idea of where
these things tend to rephrase the thing to themselves, and then be able
to bring this back, through this. Now when the first LLM models came
out, I actually had a go at trying

to make something like this myself,
with a Phi model, so a very small kind of model, to see what it could do. And you certainly can get it to
generate longer chains of context out, for this kind of thing. And, the challenge becomes
how do you preseed it for

doing those kinds of things. So I gave a whole talk at a meetup at
Google around creating a poor man's LLM. That's a couple of months ago now. I'll put that up on the Patreon
for the full talk and stuff like

that, where people can go through
and watch it if they're interested. Again, with this one, we can see
that it's almost getting into loops where it's coming up with an
answer, but it's not coming up with

a sort of conclusion, very well. Now I don't know if it's exactly
repeating itself going through this, but it's certainly doing stuff. Okay, we've got to a final answer. We don't have the depth that
we got in the DeepSeek one.

And my guess, to be fair with the Qwen
team is that, often they will sort of start out with something, release
something quite early and then iterate on getting it better and better. So I really expect that they will
make future versions of this model

even better as they're going along. But it's good to see where they're at now. Okay, so the third model is
the Marco-01 model in here. And one of the cool things that they've
done is actually release a paper.

I think at this point they would
certainly say that this is not a fully sort of fleshed out model
as much as the O1 models, perhaps even as the DeepSeek models, etc. But this also comes from a
different team in Alibaba.

And you can see that what they've
tried to do is they're actually looking at how to reproduce the O1s using,
MCTS or Monte Carlo Tree Search and basically the idea here is that
you have, the model sort of generate

out, different trees of chain of
thought, and then you have a way of sort of scoring those, And training
the model to generate better trees going forward, for this kind of thing. So the cool thing here is that they've
used a very small model, Qwen 2 7B

Instruct as their sort of base model for
doing this, and then just built on that. Now they've put the model up on Hugging
Face so you can have a play with it. I don't see an official space for this,
so I'm reluctant to try out the other ones

in case they're not set up properly, etc. And like I said, I think this one
is a bit more of an academic test to actually see what's going on. One of the things that I do like
that they've done here though is

that they've released this dataset. So they've actually released a whole bunch
of, these sorts of chain of thought demos, going through, and showing out, okay,
what the different thoughts are and how they actually do this and stuff like this.

Open Reasoning vs OpenAI

HI my name is Sam Witteveen, I have worked with Deep Learning for 9 years and with Transformers and LLM for 5+ years.  I was appointed a Google Developer Expert for Machine Learning in 2017 and I currently work on LLMs and and since earlier in 2023 on Autonomous Agents. 

Open Reasoning vs OpenAI

Scripts