We have current-gen AMD graphics cards running on the Pi 5. And they run great! I can game in 4K with ray tracing. I can transcode video or stream with OBS. I mean, there are bottlenecks, and the drivers aren't perfect on Arm, but Vulkan works, and that's apparently all I

needed for GPU-accelerated local AI on the Pi. After I asked AMD and they said they wouldn't support ROCm on Arm, I thought it would be an uphill battle getting tools like Llama.cpp to run on the Pi 5. I mean, Ollama doesn't

install on the Pi out of the box and runs models like Llama 3.2:3b on the Pi's CPU, but it's a tiny model and it's slow. Like, only a few tokens per second slow.

But plug in a graphics card, and now we're in business, getting larger models to run at dozens of tokens per second. But why make a video on this? I mean, just plug a graphics card into any old computer, right? Well, the Pi is different. It's not your typical PC. It's tiny, the size of a credit card, and

it only sips 2-3 watts of power at idle. This whole setup with the Pi and this RX 6700 XT at idle, only sips 11 watts. That's less than the CPUs alone on a lot of PCs. Run one of these

in your home lab 24/7, and at least if you're not in the US, that's a significant power savings. The speed of this setup is almost as fast as my Mac Studio. It's almost as efficient, too. But what's best is,

It's a lot cheaper. This whole setup cost me about 700 bucks, and way over half of that was this GPU. You could buy used or use an AMD card that you already have and get a setup for like 300 to 500 bucks.

But I hesitate even working on this project. I'm still a bit of an AI skeptic. We're burning a lot of resources to make these models and get incremental improvements. And it's not even AI, really. That's just a marketing term.

I'd rather call it what it is: Large Language Models. But they can be useful. Like with Home Assistant. I've been tinkering with running my own local voice assistant. More on that soon. But the idea is, instead of inviting Amazon or Google into your private life, you could keep everything local,

running inside your house. But before I show you a demo, let me show you how I have this all set up. This is the Raspberry Pi 5. I have an 8GB model with the active cooler on top. And it has a PCI Express

external connection, an FFC connection, that goes out to this Pineboards board. There's other boards like it, but this is the one that I'm using. And this converts the Pi's FFC connection into an M.2 slot.

And that M.2 slot has an OcuLink to M.2 adapter plugged in. And that came with this GPU riser. It's an OcuLink riser, and I'll link to it in the description. But I had to buy this cable,

This Oculink cable that goes over to the riser. That costs, I don't know, 15 or 20 bucks. And then to power the GPU, you can't just use the power from the Pi. It only gives you 5 watts. So this riser has

an input from my power supply. It's a Lian-Li 750 watt power supply. It takes in the 24 pin and has an on/off switch to emulate the PC on/off. And then it also has supplemental power from the 750 watt power

supply into the PCI Express connectors on the graphics card. So most graphics cards have that. Like this one has a single 8 pin. This one has an 8 pin and a 6 pin. And the more powerful the graphics card usually,

the more power it needs. But if you wanted to run a 7900 XTX or something like that on it, you would probably need a beefier power supply. Anyway, that's the physical setup. And I have, this is a 4K display

that's plugged into the card through DisplayPort. And everything, including the speaker over there, is plugged in through this power cord that goes into this smart outlet from ThirdReality. And that's measuring my power usage. And I have a dashboard in Home Assistant that is running an ApexChart that

shows me how much power I'm using over time. So to start things up, I turn on the power supply using its switch since it's a little bit easier to use than the switch that's on this little board. And then that powers up the GPU. I have it plugged in here. And this is an RX 6500 XT. I've never tried it

before, but hopefully it'll just work. And then I turn on the Pi pressing its power button. And with any luck, the Pi will come up and I don't have the display plugged in. So that's not going to work. Let me plug in the display too. Oh, there it comes. Maybe. There it is. Right now, nobody makes a GPU

dock specifically for the Pi. And that's why this is a little bit of a jank-fest. But hopefully that'll happen someday. Maybe once the CM5 is launched, assuming it works the same. But here's a demo running one of the most popular models for mid-range setups like this: Llama 3.1:8b. On the Pi 5, 

running Ollama on the CPU, it's pretty slow. It only prints out about two tokens per second. It's not glacial, but it's certainly less useful when it runs this slow. Especially if you're trying to build a conversational Home Assistant bot. Now, here's that same model running directly on the RX 6700 GPU

plugged into the Pi. It's a night and day difference, going from two tokens per second to 40 tokens per second. At this speed, I could actually use it to have a real conversation. And sure, this setup uses more wall power, but average power draw for the Pi CPU was about 13 watts at two tokens per second. Average power for the GPU was 135 watts, but normalized by tokens per second, it's 0.15

tokens per watt on the Pi CPU and 0.29 tokens per watt on the graphics card. That's a 20x speedup for only 10x more power. To make sure all my testing was actually hitting the GPU, I ran nvtop and amdgpu_top and monitored the output. I could watch as the model was loaded into the VRAM. Then the GPU would go wild crunching numbers a while, spitting out responses to my prompts. 

Now, there are some instances where context windows are too big, or models are just a bit larger than fits in the consumer graphics cards I'm testing. Like here, with the Qwen2.5 14B model, it had to split the processing off between the GPU and the CPU, and that slows things down. But in the llama-bench benchmark I ran with the same model, it went from tens of tokens per second to hundreds. 

And here's another large model, Llama 2:13B, running at 4 tokens per second in my test. These models would be unbearably slow on the Pi's CPU, if they ran at all. Now, some of my power measurements are a little tricky. I don't have a setup running to give total joules per run, so I can't give you efficiency results with 100% confidence.

I can, however, measure idle power draw pretty accurately. And considering the massive leap in performance for any of these graphics cards, running them off a Pi can save you power, assuming all else is equal. And to get a clearer picture of that performance, I headed to Micro Center.

Thanks to this channel's patrons, I picked up a couple more low-end cards to test. I found the RX 7600 needs a little more TLC with the amdgpu driver patch, but the RX 6500 XT worked right out of the box.

And if you like seeing all these shenanigans, please consider becoming a Patron! After testing everything, here are my preliminary results. Right away, two things are apparent: LLMs love GPUs, and my RX 6700 XT is in the sweet spot for value and efficiency.

At least, I think so. For larger models, my M1 Max Mac Studio with 64GB of RAM runs away. But I think, like with Doom Eternal at 4K, the way our patch works coupled with the Pi's slower RAM holds the Pi back here.

Of course, none of these options holds a torch to the 4090—just putting things in perspective. But then again, my Pi setup doesn't run at 600W. So if you don't need monster performance, you don't have to pay for a monster card.

And 4090s don't run on the Pi yet anyway. So far, I can't get any Nvidia cards running. And might I remind you, all these results are using Vulkan with our experimental patch. We're definitely leaving performance on the table.

If you want to replicate my results, I'd love that. YouTuber James Mackenzie already did with an older RX 580, and he's using it for retro gaming. The full instructions for getting all the modern AMD cards working on a Raspberry Pi,

in fact, I think from like the RX 400 series up to the 7000 series, is on my blog. And I'll link to it in the description, in this blog post. And here's all the instructions. Right now, it requires compiling your own Linux kernel.

But in the future, maybe this will be part of the Raspberry Pi Linux kernel. If you want to give it a try, there's already a couple people that have done it, and confirmed that it works with their cards too. If you have any problems, if a card doesn't work, go to my Pi PCI Express website, and file an issue for your card, if it's not listed already.

And let us know what's going on with your card. And once you have your 6000 or 7000 series card running on the Pi, these are the instructions for setting up llama.cpp with Vulkan. So you get GPU acceleration. I'm going to put this into a blog post.

I'll link to it in the description. But the blog post will show in detail how to do this with all the links that you need, and with a fancy picture or something. But it's, whenever I develop my stuff, I do it all on GitHub.

And I do it live, like, you know, I'm not masking what I'm doing. So if you ever want to see like, "Hey, what is Jeff working on?" Pop over to one of my GitHub repos, and chances are you'll find me in an issue debugging something actively,

and posting results and things, before I even post a video. That's a little pro tip for you ;) I've been testing other models too, like Mistral and Qwen. Different models are better at different things, and some run better than others on this setup.

For all my notes and benchmarks, check out the links in the description. But I want to know, what kind of things are you building with this stuff? Let me know in the comments. Please subscribe, and until next time, I'm Jeff Geerling.

A faster chatbot with my supercharged Raspberry Pi

I make things work. Often with Raspberry Pis—almost always open source!

Jeff Geerling

AMD 그래픽 카드가 장착된 라즈베리 파이 5에서 AI 모델 실행하기

AMD 그래픽 카드는 Pi 5에서 잘 실행되어 레이 트레이싱 및 비디오 트랜스코딩을 통해 4K 게이밍을 지원합니다.

AMD의 Arm에 대한 ROCm 지원 부족에 대한 초기 우려는 불칸을 통해 GPU 가속 로컬 AI를 지원함으로써 극복되었습니다.

올라마는 Pi의 CPU에서는 느리게 실행되지만 그래픽 카드를 사용하면 훨씬 빠르게 실행됩니다.

Pi의 작은 크기와 낮은 전력 소비(유휴 시 2~3W)가 장점으로 부각됩니다.

RX 6700 XT를 사용한 전체 설정은 유휴 상태에서 11W만 소비하며, 이는 많은 PC의 CPU보다 적은 전력입니다.

설정의 비용 효율성 및 전력 효율성

설치 속도는 Mac Studio와 비슷하지만 훨씬 저렴합니다(약 700달러).

중고 GPU를 사용하면 비용을 300~500달러로 줄일 수 있습니다.

제작자는 '대규모 언어 모델'이라는 용어를 선호하며 AI 회의론을 표명하고 있습니다.

로컬 음성 어시스턴트의 잠재적 사용 사례에 대해 언급하며 개인정보 보호의 이점을 강조합니다.

하드웨어 설정 및 구성 세부 정보

이 설정은 Raspberry Pi 5 8GB, 액티브 쿨러, Pine64 PCIe 어댑터 보드를 사용합니다.

M.2 슬롯, OcuLink-M.2 어댑터, OcuLink 라이저를 사용하여 GPU를 연결합니다.

GPU에 전원을 공급하려면 OcuLink 케이블과 Lian-Li 750W 전원 공급 장치가 필요합니다.

전원 공급 장치는 PCIe 커넥터를 통해 GPU에 전원을 공급하며, 하이엔드 GPU의 경우 더 강력한 전원 공급 장치가 필요할 수 있습니다.

4K 디스플레이와 스마트 콘센트 및 홈 어시스턴트를 사용한 전원 모니터링이 포함되어 있습니다.

시스템 전원 켜기 및 초기 테스트

전원 공급 장치와 Pi가 켜지고 디스플레이가 연결되었습니다.

Pi 전용 GPU 도크가 없다는 점은 아쉬운 점입니다.

라마 3.1:8b의 데모를 통해 CPU와 GPU 실행의 속도 차이를 확인할 수 있습니다.

GPU는 초당 2토큰에서 초당 40토큰으로 속도를 크게 향상시킵니다.

전력 소비 분석 결과, 10배 더 많은 전력으로 20배 더 빨라진 것으로 나타났습니다.

모니터링 도구(nvtop 및 amdgpu_top)를 사용하여 GPU 사용량을 확인했습니다.

규모가 큰 모델일수록 GPU와 CPU 간에 처리를 분할해야 하므로 성능에 영향을 미칠 수 있습니다.

Qwen2.5 14B 모델과 Llama 2:13B 모델 성능에 대해 설명합니다.

추가 로우엔드 카드(RX 7600 및 RX 6500 XT)를 테스트했습니다.

LLM은 GPU의 이점을 크게 누릴 수 있으며, RX 6700 XT는 가치와 효율성의 균형이 잘 잡혀 있습니다.

M1 Max Mac Studio와 같은 하이엔드 카드는 대형 모델의 경우 Pi 설정보다 성능이 뛰어납니다.

Pi 설정의 낮은 전력 소비는 주요 장점입니다.

엔비디아 카드는 아직 지원되지 않으며, 결과는 실험적 패치가 적용된 벌칸을 기반으로 합니다.

Pi에서 AMD 카드를 실행하기 위한 지침은 제작자의 블로그에서 확인할 수 있습니다.

현재 지침은 사용자 지정 Linux 커널을 컴파일해야 합니다.

사용자는 Pi PCI Express 웹사이트에서 특정 카드의 문제를 신고할 수 있습니다.

벌칸 GPU 가속을 사용하여 llama.cpp를 설정하는 지침이 제공됩니다.

개발은 GitHub에서 이루어지므로 사용자는 진행 상황을 확인할 수 있습니다.

다른 모델(미스트랄 및 퀀)에 대한 테스트가 참고 사항 및 벤치마크 링크와 함께 언급되어 있습니다.

A faster chatbot with my supercharged Raspberry Pi

A faster chatbot with my supercharged Raspberry Pi

스크립트