What if you learned of a cool feature that
isn't in the current version of Ollama. And then you learned that someone else implemented
it. If it's not in a downloadable release yet, how do you take advantage of it?
That's what you will learn to do today.

Ollama is a project that is always in motion.
The core maintainers do a lot of the work, but many of the biggest features come from
contributors outside of the main team. Both maintainers and external contributors work on
features that become PRs, or "Pull Requests",

on GitHub. When a Pull Request is tested and
deemed good enough by the maintainers, it gets merged into the main Ollama codebase. But until
then, it exists as its own little branch of code, separate from the main project. Sometimes these
PRs have that feature you would love to be able

to use right now. But until you learn how to
work with any PR, and then build and compile, you have to wait for it to be merged and released.
So let's take a look at what is involved in getting a PR deployed on your own machine, and
how you can get involved in the approval process.

If you have never done something like this before,
it can be complicated. There are errors that may pop up and some are critical and others are
not that important. When you don't know what you don't know, it can be especially hard to
figure out how and what needs to be fixed.

I'll try to break it down step by step, but I
can't cover every use case. That being said, the process is generally straightforward
once you understand the basics. Before working with a PR, you should first try
to compile the main branch of Ollama yourself.

If that doesn't work, then any PR will fail as well. So first, ensure that you have `git` installed on your machine. In addition to that, I would recommend installing the `gh` command-line tool, which is maintained by Github, as it makes working with GitHub-based repos much easier. You

CAN do everything with git, but `gh` just makes some hard stuff and processes with really long command lines easier to remember and to use. You can find out more about `gh` at cli.github.com. Then find a suitable location on your machine to store projects from Github.

I have a projects folder in my home directory, then `code` inside of that. Any of the projects I work on for my videos and for tools from a github repo I use all go in that code directory. So from there you can run `gh repo clone ollama/ollama`. Without the `gh` command,

we would have needed `git clone https://github.com/ollama/ollama.git`. Already `gh` is simpler and that’s not a very complex command. This will download the entire Ollama repository to your machine. Now change into that directory and we can start working with it.

The next step is to ensure you have Golang installed. Golang, or just Go, is a programming language created at Google. It has become very popular because you can learn everything built into the language in less than a day. The other way of looking

at that benefit is that nothing is built in and you have to recreate the basics or find a library elsewhere that has done that for you. A lot of system-level tools are written in Go, such as Ollama and Kubernetes. It's interesting to note that Kubernetes wasn't written in Go because

it was the best tool for the job, but rather the
developers of Kubernetes wanted to learn Go and at the
beginning, Kubernetes was just a hobby project to
learn it on. It's gotten a bit bigger since then. You can find out how to install Go for
your platform at go.dev. You will also

need gcc version 11.4.0 or later installed.
On Linux it should already be there. On Mac, it comes with the Xcode Command Line Tools
package. On Windows I am not sure what the best way to install it is, perhaps someone can
answer that in the comments below. The Ollama

developer docs recommend using MSYS2, but I
haven't spent much time with the Microsoft developer workflow since I worked at
Microsoft in Redmond two decades ago. The developer documentation in the Ollama
GitHub repo goes through all the steps involved,

but basically you want to run `make -j 5` then `go
build .` I don't actually know why they recommend using `-j 5` since that just tells make to use 5
jobs in parallel. It might speed things up a bit,

but I would think that depends on your
hardware. Now make sure the Ollama server isn't running. How to stop it depends on your
platform and how you installed it. And then, assuming the regular Ollama isn't running, you
can run `./ollama serve` to start your Ollama

server and then in another terminal window run
`./ollama run llama3.2` or whatever model you want to run. So if all that works we know your
development environment works. If Ollama was already running, you may have two environments
right now and it's difficult to know which is

which. That's why it's important to stop the
existing server before starting a new one. Now let's go to GitHub and discover a good
PR to try. There are a lot of PRs up on the site. Sometimes the biggest challenge for
the maintainer team is ensuring it works as

described in all situations. You can assist by
trying it in your environment and then adding your feedback to the PR in the format of a
comment. Now this is important. If you are just excited about a PR but don't want to
try it, adding a '+1' or 'I need this' or

something like that isn't all that helpful
and can actually slow down the process. One PR that I am excited about right now is #6279
which introduces K/V Context Quantization. This was created by Sam Macleod out of Australia. He
built this originally in August and has maintained

it ever since, hoping to get it merged into the
main branch. In case that name sounds familiar, he also created Gollama, a tool I
have recommended in another video. K/V Context Quantization is a technique used to
reduce the memory footprint of large language

models by quantizing the context embeddings. You
may remember that the GPU memory required to run any model is a combination of the needs of the
model itself plus the context size. And every model from Ollama starts with a context size of
2048 tokens. You can increase this by setting

num_ctx to a larger value. But doing so means you
need more memory. K/V context quantization lets you quantize just the context, allowing more
to fit in less memory. This becomes more and more important as models become available with
larger contexts. Depending on the quantization

configured, it can reduce the memory required
for a given context size by half or even more. So let's go ahead and get the PR installed and try
it out. First stop the server and client currently running. Visit the page for the PR and click the
Code button at the top right. Here you can see the

command to run: `gh pr checkout 6279`. Now you
can run those commands again: `make -j 5` then `go build .`. If you scroll through the comments
for this PR, there is one from Sam that says how

to use it. Run `OLLAMA_KV_CACHE_TYPE=q8_0 ./ollama
serve`. This will set the k/v quantization to q8 and start the server. The quantization for the
context is completely unrelated to the context

for the model, so you could still use a q4 model
with a q8 context. Now run `./ollama run llama3.2` and you should see better use of memory with
larger context sizes. Towards the top of the

PR is a screenshot of gollama showing how much
memory can be saved for different models. You should play around with that and see what
kind of savings you get with your own setup. I had chatted with Sam last week about this. We
all know that q4 is the 'sweet spot' for models.

Meaning it’s the quantization level that gives
the most benefit with minimal downside. Most folks wouldn't be able to tell the difference of the
output from a q4 model vs q8 or fp16 if they see the results on their own. I know there is probably
someone who is going to add a comment saying it’s

an obvious difference, but they are probably just
making that up. Well with context quantization,  q8 is that sweet spot according to Sam.
And this results in a pretty big savings. If you find any issues with the code from this
PR on your machine, you should add a comment

to the PR describing the issue. Be as specific as
possible about what happened, what steps you took, and what your environment looks like. Again,
don't add comments that are just "me too" or  " +1 " or "when will this be merged" because those
don't help anyone and if anything devalue the PR.

Try this process out with other PRs that look
interesting. First go back to main with `git checkout main`. Notice that I used the `git`
command for this and not `gh`. Then checkout the PR. Now that you see that the process is
pretty easy, there is nothing stopping you from

trying it on lots of other new features. If you
need multiple features from multiple branches, this is more difficult to do and
way out of scope for this video. Sam has done a fantastic job with this PR, and
it's a great example of how contributors can

make a real impact on the Ollama project. It
can be frustrating to wait for the maintainers to merge your changes, but he has remained
positive and supportive in all his updates. And it's looking like his persistence may
be paying off. Jeff, one of the founders of

Ollama started commenting on it in the last few
days. I hope by the time I release this video on Thursday it is already merged. Even if it is,
the video still shows how to work with any PR. But if it isn't, let's try it out and provide
some comments to help move it along. Again,

Skip the +1s and me too's and focus on specific details about your experience with this PR. Some other interesting ones to try include #5059 which adds Vulkan support. Vulkan is an alternative to ROCm and CUDA which would add support for older cards, though newer cards would

be faster with native drivers. #5872 which adds Ascend NPU support. I think Ascend is a line of chips from Huawei. #5593 adds support for Intel integrated GPUs and a lot of folks with laptops

and low-end desktops have Intel iGPUs so that would allow a lot of new folks to use Ollama. #1640 adds logprobs, which I think has to do with the probabilities for chosen tokens. #1606 adds

support for grammars...and well, there are a few of those, but many of them haven't been updated for the new build process. I also like #7281 which adds some cool formatting to ollama ls, and #6938 for command completion.

Oh my god, command completion would be so great to have. There are so many different PRs that look interesting to try. But every one of them needs to be thoroughly tested on every platform before getting merged and released. So I hope you will try them out in your own environment and then provide feedback in the

comments. This effort on your part will help the really great PRs move closer and closer to getting merged into the product sooner. And it will help identify issues in some of the PRs that the original author might not have experienced because they didn't have the hardware.

Have you tried out any interesting PRs that haven't been merged yet? If so, share them in the comments below. Thanks so much for watching, goodbye.

Find Your Perfect Ollama Build

I was a founding maintainer of Ollama, the first evangelist at Datadog, and a former organizer of DevOps Days Seattle, DevOps Days Boston, and Serverless Days Boston. In my day job, I tour the country speaking at conferences and writing about the company I work at. But I am also passionate about gadgets. Some folks don't think that word is appropriate, but it totally fits here. If you meet me in person you will see me light up when I can share with you my latest gadget, tool, or utility. And this is where I get to share that with all of you. You can find more about me at my website (http://technovangelist.com) or on Twitter (http://twitter.com/technovangelist)

Find Your Perfect Ollama Build

Find Your Perfect Ollama Build

Scripts