video. If you enjoyed this video, please leave a like. If you're new to the channel, please subscribe. If you like what you see, we post videos every week. We do tech videos, we do homelab stuff, we always have a real fun, kick-ass time, and I love learning. I love teaching. I encourage everybody to learn. Join us on this journey. We're always doing videos, we're always going to be having cool projects going on. There's always, if it's not a review of some expensive product or something like that, it's going to be, you know, good home lab stuff, how to set stuff up, and how to self-host your stuff because self-hosting is the best form of hosting. Okay, guys, hope you enjoyed the video. Stay peaceful. [Music]

you know whatever an Apple Silicon Mac, you do not need to build the server. I'm still going to show you how to build the server. If you already have one of those computers, you don't need it, especially one of the Apple Silicon Macs. They run AI models, especially the newer ones. If you have one of the M2 or the M3 Maxs, then AI runs really well. The M1 Max handle it as pretty well as well. The GPUs that Apple is building into their chips, they're honestly really solid. If you have a high-end gaming computer, you can also download the software we're going to use today. It's completely free. Let's switch the camera around, go over to the other side to the rack, and let's look at this server. Okay, party people, so I'm not putting the server on the table this time because it's on rails, so I can pull it out and work on it right here. It is all unplugged in the back. Yes, my AI server is on rails, and my main

parameter model. It will run on here, but it's 30 gigs. It's going to take a bit to download. So, you can see here when you click on these models that it will actually tell you if you're capable of offloading to the GPU. Let me make this full screen. All right, I'm gonna download the 8 billion parameter model. It's five gigs, so I will be able to store the entire model in VRAM. The bigger the model, the more it's going to be. So, for example, if I come down to the 70 billion parameter model, you can see that this model is almost 31 GB. In fact, we're just going to call that 31 gigs. I'm still capable of a full GPU offload because I have two P4s with 24 gigs each. Even though I'm bottlenecked by the hardware because it's Pascal and it's just not all that powerful, having that much VRAM is a big benefit, especially when you're learning. Because if I wanted to download this, I could.

RAM and go get yourself two Nvidia H100 cards. Those are basically the top-of-the-line model right now that you can get for AI acceleration. That being said, I believe each one of those graphics cards is $8,000, plus the actual computer to build it. Thus why I said, if you're rich, you can go that route. Most of us don't have that luxury, and most of us that want to play with AI, we're perfectly fine if it's not as fast as the commercial stuff because we just want to learn, right? I do not run this server all the time. It's expensive to run. It sucks a lot of power, even at idle, because these two graphics cards are pretty power-hungry. You can get into this really cheap. Like I said, the whole server with the two graphics cards for me, at least, was right about $1,000, and you're not getting into AI any cheaper than that. If you already have a really nice computer, like a gaming computer or a Mac, a newer M1 or

What's up, ladies and gentlemen? I am Dylan, the tech-giz guy, and today, not only am I going to show you how to run AI models at home on your own computer, but I'm also going to show you how you can build a server dedicated to artificial intelligence and machine learning applications on a budget. So, before we talk about running AI at home, I want to talk about the AI server. We're going to do that first, but I want to get something out of the way. This server is sub-$1,000, so do not expect top-of-the-line AI performance. Do not expect to go download the 340 billion parameter llama 3 model and run it at home blazing fast, faster than ChatGPT. You are not going to get that. Let's talk about the machine itself. So, the machine itself is a Dell R730 server that I bought used. I got it for 600 bucks. I kitted it out with two Xeon E5-2697s. Those are 14-core hyperthreaded processors. I have two of

Cooler and into this card, and the same one goes for this. It's in there nice and snug. You have a server, one of these Dells at least. Pop your little blue tab back in, give it a wiggle test. Should be nice and firm in there. Okay, guys, so as you can see, both graphics cards are in. The cables are tucked to the sides or underneath to the best of the ability. Now we're going to throw the top back on, slide this bad boy in, and get to work. You guys want to hear the sound to my [Applause] [Music] people. Okay, we're in here. So right off the bat, you can tell that we're missing some drivers. So we're going to go grab the NVIDIA drivers. Okay, guys, we're going to go download the drivers from Nvidia. This is going to be the same whether you're on a personal gaming computer and you want to download the drivers, or you're building a server and your p4s. The options will be a little different depending on your graphics card, but you

and once we get into here, you'll see how easy it is to really get these models. Once we are in here, I'm going to go to the settings, and the first thing I'm going to do is set this to dark. So, let's go through this. So, this is the page you are greeted with when you open LM Studio for the first time, and after you set it to dark mode, of course. Now, the nice thing with LM Studio is that this program links directly to Hugging Face, so you can download the models on Hugging Face right from LM Studio. So, you can see here it says search for models. So, I'm just going to search for llama. If you guys don't know, Llama 2 and 3 are open-source models made by Meta. You can use them completely how you want, and they've actually just changed the licensing for it. So, now if you do want to, you can even use the 340 billion parameter model that they released to train your own model. Okay, guys, so I'm not going to run the 70 billion

lot of this server-grade stuff that's designed for data centers uses EPS, not PCI Express. So, a cheap way to get around that is to grab one of these. This is an EPS male splitter, so you get two PCI Express eight pins from this EPS connector. Those then go to an EPS male, so this is an EPS male to dual 8-pin female, and then an EPS male to dual 8-pin male. You can find people who sell these cables that are just EPS to EPS, but quite frankly, this is cheaper, and it's fine. It's a little more wire to manage, but it's fine. So, you need two sets of these because we have two cards, so there's one here and one there. So, something I like to do with these is just kind of wrap them under where the cards are going to go. You can do that with this because these cards don't have fans, notice that. So, another good tip for these, if you're putting these in anything but a server, you have to basically 3D print shrouds that go here.

them. I also kitted it out with 64 gigs of RAM and two 1100-watt power supplies for redundancy. The power supplies are the most important part because the next piece that we're going to talk about is what gives this older server its kick. Inside of the server, we're going to put, and we're going to see that in a second, two Nvidia Tesla P440 graphics cards. Now, I picked these up a couple of months back, and they were really cheap at the time. I got two of them for 360 bucks. Right now, it looks like the prices have gone up a bit, but they fluctuate all the time, and honestly, you can probably get newer cards for better prices now at this point because when I bought the cards that I have, it was back in January. So, you can probably get better cards. So, I have two of them. Each of these cards has 24 gigs of VRAM, which is really important. So, if you're super rich, you can run out, build a Threadripper server with 128 gigs of

silicon runs AI models really well. I actually cannot wait until I get my Mac, my desk Mac. I would love to try running a model on my MacBook Air, but I think my poor MacBook Air would just explode. GF, anything you can find on Hugging Face, you can probably run in this. That leads me into the next thing I want to talk about, guys. If you're getting into AI, if you want to experiment with AI at all, you want to learn about it, you want to find the models, Hugging Face doco is where you want to go. So, Hugging Face is the open-source community for models. So, you can download language models in here, you can download text-to-speech models, text-to-image, image-to-text, video, all kinds of models. You can even find fine-tuned models that are based on the bigger models that people have made to do various things. All right, guys, LM Studio is going to boot up. It takes a minute here to get itself situated, but the install process for this is really easy.

it a question. So, this is the 8 billion parameter model. Also, I just want to take a note here, uh, we're doing 37.6 tokens a second. That is not that fast, but, uh, when you're only running an 8 billion parameter model, it's actually really solid. So, I'm a guy. I like history. I like War history specifically. So, I'm just gonna ask it, let's say, give me a brief history of World War I. Let's just see what it does. You can see this is about the speed that you'll get with GPT-4. Now, yes, this model is nowhere near as capable as GPT-4, but the idea that you can download this free software, download a free model, and start using it right now is amazing. So, it looks like right now we're limited to about, we'll say, 40 tokens a second. Yeah, I mean, that's a brief summary of World War I. This is pretty accurate, um, for an 8 billion parameter model. This is, guys, you can see how easy it is to just come

and you can actually find the files like these. That's what these screw holes are for. Someone made the files to 3D print a shroud that fits in here, and you can put like a 70 mil blower fan on the back, cool these cards right off. If you're putting these in a server, you don't have to worry about it because servers usually have forced induction. All right, guys, so I'm going to try to do this. This is not the easiest camera angle, so uh, and I have to be zoomed in a little bit. So I'm going to plug this EPS connector in, run the wires under here, flip that little blue tab out of the way, and basically just want to kind of fight the cable a little bit. All right, so that's one in, and notice how I tried to keep the cable going under the card so that way this will still get airflow because this is where it's going to get its airflow from, right here. So it's literally going to go over the CPU.

do want to make sure your drivers are up to date. If you're on a gaming computer and you have GeForce experience, so I'm going to search for Tesla and say data center Tesla and uh I'm going to get the while that's downloading. The other thing you want to go grab is the program that's actually going to make this all happen, meet LM Studio. LM Studio is a program that basically exists to run LLMs, large language models, on your machine. One nice thing is it comes with a built-in chat application, so if you just want to mess around with AI and learn a little bit about it, you can absolutely just chat with this as if it were Chat GPT. The other thing I like about this is you can run an endpoint API server, and it mimics the OpenAI API, so you can very easily code stuff to operate off of this. So I'm going to download this for Windows. You can see here that uh the first link they have is for the M12 or three max, the Apple silicon Max. Like I said, the Apple

would just take some time. I'm trying to get this video done, so we're going to do the 8 billion parameter one. You can see it's already kind of going through here, and it's almost more than halfway done. The chat page here, this is basically your chat GPT style thing. Once we get the model, this is what we're going to show you guys. Developer settings will actually allow you to, like I said, start that server, so you can set the port that you want. You can enable certain things, serve on local network. Okay, so the model is downloaded. Let's load it up, guys. This couldn't get easier. So, besides building the server here, context length 4 4096. Yeah, that's fine. I'm just going to number of discret model layers to compute on the GPU. You want to set this as high as you can. So, honestly, I'm just going to leave this all the way up because this entire model will fit on this C. Honestly, this should run really well on this system. All right, let's ask.

through and download these models. You can download Gemma right from here, which is Google's model that they've released. Um, you can go in here and find the Phi models. Okay, so like right here is the Llama 3. It's one of the Llama 370 billion parameters. The download is 425 gigs. So, it says partial because it's basically saying like, Hey, I could load this all into RAM, but it's G to basically take everything. Uh, you can also see if you come down here, you can see the different quantized options. So, uh, green means full GPU option. So, here's a 70 billion parameter model has been quantized is down to 16 gigs, uh, 16.7, 16 and 3/4 gig, uh, and even up to this one, this, you know, other quantized one, uh, know 26.3 gigs. So, I can still load that entire model. I'm Dylan the Technic G guy. Have fun learning AI, guys, because it's here to stay. I will see you guys in the next

production server is not on Rails. These are the GPUs that are going in, so these are the Tesla P4s. You can see these do not have any kind of output; these are strictly accelerator cards. You cannot plug monitors into these. I think it was Craft Computing who actually grabbed a couple of these in a server, and he was splitting them out over virtual machines, well over RDP, so that way they could be used. I think he actually was able to play a couple of games on these. These do not just plug in, so you need some cabling. Okay, so I want to talk about something, guys. These cards, that is not an 8 Pin PCI Express connector; that is an EPS connector for your CPU. So, if you've ever built a computer, this is the plug that actually plugs into the motherboard for the CPU. This is not a PCI Express plug; one won't fit. So, luckily, this server actually has the same thing. So, these are not PCI Express power; these are EPS ports and a

Build an Ai Server for less than $1k and Run LLM's Locally FREE

Build an Ai Server for less than $1k and Run LLM's Locally FREE

Scripts