Hey, welcome back! So, Microsoft have just released the brand new 5414 billion parameter model, which is insanely good.

For the size of model that it is, Microsoft is claiming that it beats GPT-4 and Code Llama 34B in the area of math,

which we all know is complete nonsense. However, that doesn't make it not a good model, because actually the one thing

that this model does really well is chain-of-thought reasoning. Now, it's a little bit frustrating, but they have set

themselves up really well for the future, and I think there may be one or two models short of having something that

really performs in the areas of inference time and compute. Now, in this video I'm going to show you how to get started

with the 54B model using Ollama on your local machine, and I'll show you some of the pros and cons of the model. Um, and as

you see, it's it's a really great model. I just think it's maybe one version too early, but let's get started. I'll show

you how to get it on your machine so you can start playing with it yourself. Now, obviously to get started you're

going to have to have Ollama already installed on your machine. I cover that in another video. Um, and then really what

you're going to do is go into ollama.com, and then you're going to click on "Models." And then if you just search maybe on

newest at the moment, you'll see that at the time of recording, 54B isn't officially in the library. But if I do a

search for "54B," for example, you see there is a model there called "llama-2-7b-chat-hf".

And when Microsoft released it, it's just going to be called 54B. Um, and you will be able to uh run that by just

doing `ollama run llama-2-7b-chat-hf` because that's how it's worked for every other LLaMA model in their series. However, for today we are

I'm going to run llama 2 7B and I'm going to use the q80 version of that model

because I've got a lot of memory and I want to get the best results possible, but you can just run the latest

on your machine, and you should be able to run it if you've got something like 16 gigabytes of memory or whatever. So, as you can

see, I've got 7B now downloaded, and if I want to chat on the terminal, I can just

type in something like...and you see it comes back, and it's pretty good in that sense. Now, of course,

I don't want to be interacting with it in the terminal in this sense, so I'm going to use open web UI. If you want to

get started with open web UI, you can just go to openweb ui.com, click on docs, and then there's a pretty good getting

started guide. In fact, if you go to the quick start and then do the docker pull followed by the docker run, then

you're going to run that locally with Docker on your machine, and that's what we're going to do today. So I'm just

going to do a docker run, and that is going to have open web UI running on my machine, and then within my

browser, I can go to localhost 3000 and I can chat with the model directly. So you see I've got a new chat, and then I'm

going to select llama 2 7B there, and then I'm just...so that gives me the 7B

model (that will be 7B when it's officially out on all llama), and then I can just chat with the model as before.

And you see it's coming back with answers, and then I can say, "What is your name?" So it's pretty easy to get started

with the 7B model; it's fairly quick, as you can see there, and it is coming back.

With good, nice responses. Now, they reckon that this model is super good at math. So if I give it a typical sort of math

style problem: a grocery store sells apples for $1 each and oranges for $1.50 each. If you have $10, and 20 is twice as

many apples as oranges, how many of each can you buy? So if I ask it that quick question, then it's going to work through

a little bit of chain of thought, and I think this is actually the key to why this model is doing really well in the

math domain; that it's not just coming back with an answer straight away, and it's actually working through its chain

of thought. So eventually it's going to come back there and say, uh, since o equals 3, access the budget, the only

feasible solution is o=2 and a=4. So it's worked through the problem; it's even

done the math there: 2 + 8 = 10, and it's claimed you can't buy a fraction of an orange, so it's done a really good job

there, especially for a 14 billion parameter model. But of course, that doesn't make it as good as LLaMa 35 son or

GPT-4, but but I think this is why it's doing so well in the math domain, because

it's working through these chain of thought type problems. And again, I can give it a few more queries. So if I give

it a kind of, 'You're given the equation blah, blah, blah; find all the integer solutions,' and then it's going to come

back, uh, it's going to do the calculations on this, and again you can see it's working through all the terms

and it's working through, uh, solving this equation again using chain of thought, and then finally it came back

with the answer, and that is absolutely the right answer. And for a 14 billion parameter model, this is absolutely

Insane! It is doing a great job and I can give it some pretty complicated solutions as well. So, in this case, I'm

going to ask it what I call the 650 Gates problem, which is kind of like the 100 Gates problem, but I've just been

lazy and I've made it 650 gates. Um, and in this case, it's a garden with 650

gates, all closed. You make 650 passes through the garden, you know, and then on each pass you're basically toggling the

gate as you go through, and you're going to work all the way through to 650.

Here, and then what I've done is I've asked it to come back with the first 10

open gates and the last 10 open gates. Now, in this particular case, it's done a fabulous job; it's actually got all the

right answers. So it's got the first 10 gates there. The only problem I would say with the last 10 is that it's missed the

final gate there, which is gate 625. It's got it over here, since there's only,

you know, it will include 625, but then when it summarized it at the bottom, it's missed the last 10. But I'm not really going to bother about that too much. The fact is that it did solve the problem, and again, it's doing a great job.

This is a really good math model, but of course, both GPT-4 and Claude solve this as

well. The problem I would say is if I give it an actual math question like 254 multiplied by 752 minus 3, blah, blah, blah,

you're going to see that it is actually going to mess it up and it's not going to give the right answer. So, as you can

see, it says the final result is 1,190,805, which is in fact the wrong answer. And as you can see, the answer according

to Claude is 119105, which is in fact the correct answer. However, the good news is for this

model, if I were to ask the exact same question of GPT-4, and in this case I'm

going to say "do not use tools," and I'm going to say, uh, what's that? As you can

see, GPT-4 messes up that question as well. But if I obviously say "use tools," I've been very shouty with my caps

locks there, uh, you see the right answer is 119105, which of course Claude got. So if

I'm honest, Claude is still the better model. However, for a 14 billion parameter model, it's done a really, really good job.

So as you can see there, it is actually really good at math, and it probably is pretty close to what Claude 3.5 can do and

what GPT-4 is doing in a mass say, but I would say Claude 3.5 is still

beating it. But what is really good is the chain of thought that it generates. So when you combine that later with an

inference time computer, which I think is the real solution for doing math problems, if you take away kind of using

tools and function calling for a second, then I think they're really setting themselves up for this kind of chain of

thought, uh, type mode. And you can see this—you can actually see this in the model. So if I, uh, if we do some typical

kind of puzzles that I like to do here. So in this case, I'm going to do my Sudoku grid problem. So I'm going to ask

it this question: "Is this a valid Sudoku grid?" Now in this case, it's going to generate the right chains of thought, but

it may or may not get it actually right. So if we look at this grid for a second here, you can see in this final column

You see the number seven, and the number seven appears there. So we're going to ask this question; it should pick up the

fact that this is not a valid grid. So the first thing is that it actually, back to my point, the chain of thought that it's

doing here is really good. This is what I'd expect these models to be doing. I would expect it to check it row by row,

column by column, grid by grid, to check if the answer is correct. So so far, it's reckoned all the rows are correct,

but it should now pick up that in the final column there that is invalid, and it's also picked up column 4 that is

invalid because it's got two twos. So, uh, we can check that there. We go, two threes there

and invalid 27. So it's picked up the columns there, and now it's doing the grid. So this is a really good job that

it's doing there. This is how you should be approaching this. So again, if I were to go into something like ChatGPT, we'll

do 40 in this case. So actually, if you look at this for a second, there, GPT 40

has actually done a pretty bad job at this as well, because, uh, it's kind of skipped the row-by-row validation and the

column validation there, and it's gone straight through the grids. So it, it really hasn't spent the time, uh, looking

at this, which is, of course, the, uh, the—which, of course, the older models would do. So, and again, this is really setting up

for inference time compute. You can see if you combine what is happening with this tiny 14-billion parameter model, but

then you put in for time and compute in front of this, then you're going to get, uh, kind of great results in this sense,

and so in some regards it is doing a better job than GPT 40, but it's obviously not going to do a better job

than anything using inference time compute such as 01. Now I'm going to give it a slightly harder question, and

this time I'm going to give it this grid here, um, asking the same question: is this a valid Sudoku grid?

Probably the key thing that I want you to realize from this grid is that the rows and the columns are generally okay;

the issue is going to be within the grid. So if you look at this center grid here, you see there are two fours—four and a

four. So let's see how the 54 model does. And again, it's going through that exact same sequence as it does before;

it's checking row by row by row. Okay, so it's come back now. It is saying that, um,

since the bottom middle subgrid contains a duplicate nine, the grid is not a valid Sudoku solution. But it's

missed the center grid; it's missed the fact that the four and the four there is invalid, but it's picked up at the bottom—

right here it's saying there's a duplicate nine. But you know, actually, if we were to compare the grid there, the

bottom right, there's not a duplicate nine in this case; it's a missing nine. So it's got the right answer and

it's doing the right chain of thought, but it's not quite um got to the right outcome. If I go into the DeepSeek R1

Phi 4 on Ollama - is it REALLY better than Claude 3.5?

hi there, welcome to my channel, hope you enjoy it and are as having as much fun as i am.

my name is chris hay and i love code, software architecture and technology in general.

as you may have figured, i work for IBM (who i love working for) but all opinions are most definitely my own and do not represent the views of my employer


Chris Hay

Introduction of the new 54B parameter model and its capabilities.

Microsoft released a new 54B parameter model claimed to surpass GPT-4 and Code Llama 34B in math.

The claim of superior math capabilities is considered nonsense.

The model excels in chain-of-thought reasoning.

The model shows promise but might be one or two versions away from optimal performance in inference time and compute.

The video demonstrates how to use the 54B model with Ollama.

Setting up and running the 54B model using Ollama and Open Web UI.

The 54B model might not be officially in the Ollama library initially.

The model can be run using the command `ollama run llama-2-7b-chat-hf`.

The presenter uses the q80 version of llama-2-7b-chat-hf for optimal results.

Using Open Web UI for easier interaction.

Evaluation of the model's mathematical capabilities.

The model demonstrates strong performance in solving math word problems using chain-of-thought reasoning.

While impressive for a 14B parameter model, it's not as good as Llama 3.5 or GPT-4.

Successfully solves the complex '650 Gates' problem.

Struggles with straightforward mathematical calculations, unlike GPT-4 and Claude.

Comparison with other models on reasoning tasks (Sudoku).

The model demonstrates good chain-of-thought in solving Sudoku puzzles.

Compared to GPT-4, the 54B model shows a more structured approach to solving Sudoku.

The model's accuracy in identifying errors in Sudoku grids is evaluated.

Comparison with DeepSeek R1 and 01 models highlights differences in reasoning strategies and accuracy.

Evaluation of the model's performance in Tic-Tac-Toe.

The model's ability to determine the optimal move in Tic-Tac-Toe is assessed.

Comparison with GPT-4, Claude, DeepSeek R1, GPT-4 0 mini, and 01 mini highlights differences in strategic thinking and accuracy.

Comparison with Llama 3.3 reveals differences in speed and chain-of-thought reasoning for game-playing.

Assessment of the model's tool-calling capabilities and limitations.

The model's inability to perform function calling using MCP CLI is highlighted.

Comparison with Llama 3.2 and 3.3 showcases the impact of tool-calling on task completion.

The lack of tool-calling support limits the model's applicability in certain use cases.

Evaluation of the model's role-playing capabilities.

The model's performance in role-playing scenarios is assessed.

Comparison with Llama 3.2 highlights differences in personality and storytelling abilities.

Further comparison with Llama 3.3 demonstrates the impact of model size on role-playing performance.

Assessment of the model's coding capabilities.

The model shows decent coding abilities for simple tasks.

Comparison with Claude in generating React code and AI-generated images is made.

The model struggles with complex coding tasks and fails to follow instructions accurately.

Even smaller models like GPT-4 Mini outperform the 54B model in code fixing tasks.

Conclusion and summary of the model's strengths and weaknesses.

The 54B model excels in chain-of-thought reasoning, summarization, and classification.

The model's lack of inference time compute and function calling limits its potential for agent workflows.

Coding capabilities are deemed okay but not comparable to Claude or GPT-4.

Mathematical abilities are good but limited without function calling.

Larger models like GPT-4 and Claude generally perform better in real-world tasks.