What if you learned of a cool feature that
isn't in the current version of Ollama. And then you learned that someone else implemented
it. If it's not in a downloadable release yet, how do you take advantage of it?
That's what you will learn to do today.

Ollama is a project that is always in motion.
The core maintainers do a lot of the work, but many of the biggest features come from
contributors outside of the main team. Both maintainers and external contributors work on
features that become PRs, or "Pull Requests",

on GitHub. When a Pull Request is tested and
deemed good enough by the maintainers, it gets merged into the main Ollama codebase. But until
then, it exists as its own little branch of code, separate from the main project. Sometimes these
PRs have that feature you would love to be able

to use right now. But until you learn how to
work with any PR, and then build and compile, you have to wait for it to be merged and released.
So let's take a look at what is involved in getting a PR deployed on your own machine, and
how you can get involved in the approval process.

If you have never done something like this before,
it can be complicated. There are errors that may pop up and some are critical and others are
not that important. When you don't know what you don't know, it can be especially hard to
figure out how and what needs to be fixed.

I'll try to break it down step by step, but I
can't cover every use case. That being said, the process is generally straightforward
once you understand the basics. Before working with a PR, you should first try
to compile the main branch of Ollama yourself.

If that doesn't work, then any PR will fail as well. So first, ensure that you have `git` installed on your machine. In addition to that, I would recommend installing the `gh` command-line tool, which is maintained by Github, as it makes working with GitHub-based repos much easier. You

CAN do everything with git, but `gh` just makes some hard stuff and processes with really long command lines easier to remember and to use. You can find out more about `gh` at cli.github.com. Then find a suitable location on your machine to store projects from Github.

I have a projects folder in my home directory, then `code` inside of that. Any of the projects I work on for my videos and for tools from a github repo I use all go in that code directory. So from there you can run `gh repo clone ollama/ollama`. Without the `gh` command,

we would have needed `git clone https://github.com/ollama/ollama.git`. Already `gh` is simpler and that’s not a very complex command. This will download the entire Ollama repository to your machine. Now change into that directory and we can start working with it.

The next step is to ensure you have Golang installed. Golang, or just Go, is a programming language created at Google. It has become very popular because you can learn everything built into the language in less than a day. The other way of looking

at that benefit is that nothing is built in and you have to recreate the basics or find a library elsewhere that has done that for you. A lot of system-level tools are written in Go, such as Ollama and Kubernetes. It's interesting to note that Kubernetes wasn't written in Go because

it was the best tool for the job, but rather the
developers of Kubernetes wanted to learn Go and at the
beginning, Kubernetes was just a hobby project to
learn it on. It's gotten a bit bigger since then. You can find out how to install Go for
your platform at go.dev. You will also

need gcc version 11.4.0 or later installed.
On Linux it should already be there. On Mac, it comes with the Xcode Command Line Tools
package. On Windows I am not sure what the best way to install it is, perhaps someone can
answer that in the comments below. The Ollama

developer docs recommend using MSYS2, but I
haven't spent much time with the Microsoft developer workflow since I worked at
Microsoft in Redmond two decades ago. The developer documentation in the Ollama
GitHub repo goes through all the steps involved,

but basically you want to run `make -j 5` then `go
build .` I don't actually know why they recommend using `-j 5` since that just tells make to use 5
jobs in parallel. It might speed things up a bit,

but I would think that depends on your
hardware. Now make sure the Ollama server isn't running. How to stop it depends on your
platform and how you installed it. And then, assuming the regular Ollama isn't running, you
can run `./ollama serve` to start your Ollama

server and then in another terminal window run
`./ollama run llama3.2` or whatever model you want to run. So if all that works we know your
development environment works. If Ollama was already running, you may have two environments
right now and it's difficult to know which is

which. That's why it's important to stop the
existing server before starting a new one. Now let's go to GitHub and discover a good
PR to try. There are a lot of PRs up on the site. Sometimes the biggest challenge for
the maintainer team is ensuring it works as

described in all situations. You can assist by
trying it in your environment and then adding your feedback to the PR in the format of a
comment. Now this is important. If you are just excited about a PR but don't want to
try it, adding a '+1' or 'I need this' or

something like that isn't all that helpful
and can actually slow down the process. One PR that I am excited about right now is #6279
which introduces K/V Context Quantization. This was created by Sam Macleod out of Australia. He
built this originally in August and has maintained

it ever since, hoping to get it merged into the
main branch. In case that name sounds familiar, he also created Gollama, a tool I
have recommended in another video. K/V Context Quantization is a technique used to
reduce the memory footprint of large language

models by quantizing the context embeddings. You
may remember that the GPU memory required to run any model is a combination of the needs of the
model itself plus the context size. And every model from Ollama starts with a context size of
2048 tokens. You can increase this by setting

num_ctx to a larger value. But doing so means you
need more memory. K/V context quantization lets you quantize just the context, allowing more
to fit in less memory. This becomes more and more important as models become available with
larger contexts. Depending on the quantization

configured, it can reduce the memory required
for a given context size by half or even more. So let's go ahead and get the PR installed and try
it out. First stop the server and client currently running. Visit the page for the PR and click the
Code button at the top right. Here you can see the

command to run: `gh pr checkout 6279`. Now you
can run those commands again: `make -j 5` then `go build .`. If you scroll through the comments
for this PR, there is one from Sam that says how

to use it. Run `OLLAMA_KV_CACHE_TYPE=q8_0 ./ollama
serve`. This will set the k/v quantization to q8 and start the server. The quantization for the
context is completely unrelated to the context

for the model, so you could still use a q4 model
with a q8 context. Now run `./ollama run llama3.2` and you should see better use of memory with
larger context sizes. Towards the top of the

PR is a screenshot of gollama showing how much
memory can be saved for different models. You should play around with that and see what
kind of savings you get with your own setup. I had chatted with Sam last week about this. We
all know that q4 is the 'sweet spot' for models.

Meaning it’s the quantization level that gives
the most benefit with minimal downside. Most folks wouldn't be able to tell the difference of the
output from a q4 model vs q8 or fp16 if they see the results on their own. I know there is probably
someone who is going to add a comment saying it’s

an obvious difference, but they are probably just
making that up. Well with context quantization,  q8 is that sweet spot according to Sam.
And this results in a pretty big savings. If you find any issues with the code from this
PR on your machine, you should add a comment

to the PR describing the issue. Be as specific as
possible about what happened, what steps you took, and what your environment looks like. Again,
don't add comments that are just "me too" or  " +1 " or "when will this be merged" because those
don't help anyone and if anything devalue the PR.

Try this process out with other PRs that look
interesting. First go back to main with `git checkout main`. Notice that I used the `git`
command for this and not `gh`. Then checkout the PR. Now that you see that the process is
pretty easy, there is nothing stopping you from

trying it on lots of other new features. If you
need multiple features from multiple branches, this is more difficult to do and
way out of scope for this video. Sam has done a fantastic job with this PR, and
it's a great example of how contributors can

make a real impact on the Ollama project. It
can be frustrating to wait for the maintainers to merge your changes, but he has remained
positive and supportive in all his updates. And it's looking like his persistence may
be paying off. Jeff, one of the founders of

Ollama started commenting on it in the last few
days. I hope by the time I release this video on Thursday it is already merged. Even if it is,
the video still shows how to work with any PR. But if it isn't, let's try it out and provide
some comments to help move it along. Again,

Skip the +1s and me too's and focus on specific details about your experience with this PR. Some other interesting ones to try include #5059 which adds Vulkan support. Vulkan is an alternative to ROCm and CUDA which would add support for older cards, though newer cards would

be faster with native drivers. #5872 which adds Ascend NPU support. I think Ascend is a line of chips from Huawei. #5593 adds support for Intel integrated GPUs and a lot of folks with laptops

and low-end desktops have Intel iGPUs so that would allow a lot of new folks to use Ollama. #1640 adds logprobs, which I think has to do with the probabilities for chosen tokens. #1606 adds

support for grammars...and well, there are a few of those, but many of them haven't been updated for the new build process. I also like #7281 which adds some cool formatting to ollama ls, and #6938 for command completion.

Oh my god, command completion would be so great to have. There are so many different PRs that look interesting to try. But every one of them needs to be thoroughly tested on every platform before getting merged and released. So I hope you will try them out in your own environment and then provide feedback in the

comments. This effort on your part will help the really great PRs move closer and closer to getting merged into the product sooner. And it will help identify issues in some of the PRs that the original author might not have experienced because they didn't have the hardware.

Have you tried out any interesting PRs that haven't been merged yet? If so, share them in the comments below. Thanks so much for watching, goodbye.

Find Your Perfect Ollama Build

I was a founding maintainer of Ollama, the first evangelist at Datadog, and a former organizer of DevOps Days Seattle, DevOps Days Boston, and Serverless Days Boston. In my day job, I tour the country speaking at conferences and writing about the company I work at. But I am also passionate about gadgets. Some folks don't think that word is appropriate, but it totally fits here. If you meet me in person you will see me light up when I can share with you my latest gadget, tool, or utility. And this is where I get to share that with all of you. You can find more about me at my website (http://technovangelist.com) or on Twitter (http://twitter.com/technovangelist)

Matt Williams

소개: 풀 리퀘스트를 통한 미출시 올라마 기능 활용하기

공식 릴리스 전에 올라마 풀 리퀘스트의 기능을 사용하는 프로세스를 설명합니다.

기여자의 풀 리퀘스트(PR)를 포함하는 Ollama의 개발 프로세스를 소개합니다.

메인 코드베이스에 병합하기 전에 PR의 기능에 액세스하는 데 따르는 어려움을 강조합니다.

PR을 로컬에 배포하고 승인 프로세스에 기여하는 데 관련된 단계를 간략하게 설명합니다.

프로세스 중에 발생할 수 있는 잠재적인 복잡성과 오류를 인정합니다.

메인 브랜치를 먼저 컴파일하고 `git`과 `gh`가 설치되어 있는지 확인하는 것이 좋습니다.

더 쉬운 GitHub 상호 작용을 위한 `gh` 명령줄 도구의 사용에 대해 설명합니다.

Gh Repo Clone` 또는 `Git Clone`을 사용하여 올라마 리포지토리를 복제하는 방법에 대한 안내서입니다.

Golang 및 gcc 버전 11.4.0 이상이 필요함을 지정합니다.

다양한 운영 체제에 대한 Go 및 gcc 설치에 대해 자세히 설명합니다.

make -j 5`와 `go build`를 사용하여 컴파일 과정을 설명합니다.

새 서버를 시작하기 전에 기존 올라마 서버를 중지하도록 지시합니다.

특정 풀 리퀘스트 탐색 및 테스트하기

PR을 테스트하고 건설적인 피드백을 제공하는 것의 중요성을 강조합니다.

'+1' 또는 '이게 필요해요'와 같이 도움이 되지 않는 댓글은 사용하지 마세요.

PR #6279에 초점을 맞춰 샘 맥클라우드의 K/V 컨텍스트 정량화를 소개합니다.

K/V 컨텍스트 정량화와 메모리 사용량 감소에 따른 이점을 설명합니다.

'gh pr checkout'을 사용하여 PR을 체크아웃하고 실행하는 과정을 자세히 설명합니다.

PR을 실행하는 방법과 명령줄 옵션을 설명합니다.

컨텍스트와 모델 정량화의 독립적인 특성을 강조합니다.

모델과 컨텍스트에 대한 정량화 수준의 '스위트 스팟'에 대해 설명합니다.

건설적인 피드백 제공 및 다른 PR 탐색하기

문제를 보고할 때 상세하고 구체적인 피드백의 중요성을 강조합니다.

'git 체크아웃 메인'을 사용하여 메인 브랜치로 다시 전환하는 방법을 설명합니다.

다른 흥미로운 PR과 함께 프로세스를 시도해 볼 것을 권장합니다.

샘 맥클라우드의 기여와 커뮤니티 참여의 잠재적 영향력을 강조합니다.

벌칸, 어센드 NPU, 인텔 iGPU 지원, 로그프롭, 문법, 명령어 완성, 서식 지정 개선 등 기타 흥미로운 PR에 대해 언급했습니다.

결론 커뮤니티 기여 및 행동 촉구

올라마 개선을 위한 커뮤니티 테스트의 중요성을 강조합니다.

플랫폼별 문제를 파악하는 데 있어 사용자 피드백의 가치를 강조합니다.

시청자가 흥미로운 PR을 통해 자신의 경험을 공유하도록 장려합니다.

Find Your Perfect Ollama Build

Find Your Perfect Ollama Build

스크립트