how often have you asked out loud, I wish I had an AI model that responded only in emojis, like every day, right? I know, and for some reason, the team building Olama hasn't listened. Insanity. So, we're going to fix that. [Music] Today, and in the process, you'll learn how to convert and quantize a model. It's not hard, but it also takes a bit of effort to get it going, and maybe some false starts, and it doesn't always work. So, the first step is finding the right model. Let's go up to Hugging Face up top. I'll search for emoji and then show all the models. Now, I don't want to create an emoji; that would be silly. I want to do text generation but with emoji. So, select text generation over on the side, and the first result is Open Hermes Emoji Tron 001. Perfect. Scroll down, and this seems to be exactly what we want, exactly what everyone needs. Now, go to the files tab and click on the config.js file. Notice that architecture

is mistol for coo LM. That's good. That's one of the architectures supported. We should download the model now, but I've always had trouble just downloading the files the normal way with Git. It's slow, it seems, on my machine, so I really like hugging face downloader by Boda. You can grab the executable for the releases and get it installed. There's also a shell script to install as well if you want to do that. I have a folder off my user's route called bin, so I just throw it in there. Okay, so go to the repo on hugging face and click the copy icon next to the repo name. And now go to wherever you want your models to be downloaded to and run HF downloader DS do - M and paste the repo name. Press enter. Those files are downloading. So now let's get the conversion process working. Make sure Docker's installed and running on your system, and then in a different terminal run Docker pull ol/quantise. How's the hugging face downloader process going? We need to wait for the

model to download, so I'll speed on to that being done. Okay, here we are back again. Here we are in the folder. We ran that command in move into the directory created for the role model. We need to run Docker run --rm -v. colmodel ol/quantise. This is not going to work, but it shows us something we need. Docker run will run an image to create a new container on your system -- RM removes the container once the process is complete - v. colmodel tells Docker to mount a volume. Take whatever is in the current directory represented by the dot and mount it in the container as /model AMA /quantise is the image we want to run and then the command that is being run inside the container takes two parameters first - Q specifies the quantization and that's why I wanted the command to fail. We need to know which quantization to use. The fastest to run and the best compromise in performance and quality is usually q40. So run that Docker run command again adding - Q.

q40 SL model to the end. I'll try to remember to put the full command down in the description, but I'm sure someone's going to call me out on forgetting. If everything works and the model is configured correctly, then we should see a lot of activity. The model is first being converted to gdf format, and then it's being quantized to 4-bit. Quantization simply means that the 7 billion 32-bit floating-point numbers are being quantized to 4-bit integers. To visualize this, I need some cool graphics, and I'm working on that before making a video about quantization. There are now two new files in here: F16 dobin, which is the converted model, and q40 dobin, which is the 4-bit quantized GDF model file. This second file is often referred to as the model weights, but it's kind of useless without the template, possibly a system prompt, and maybe some parameters. We can make a guess about the template, but it's best if we look in the Hugging Face README.

there isn't a standard way of showing this, so there may be some sleuthing that we need to do down here at prompt format. It says open Hermes Emoji Tron 001 uses chat ml as a prompt format, just like open Hermes 2.5. It also appears to handle mol format great, especially since I use that for the fine tune. Oops, I wish they put the format in here, but now we have to figure out what mistol uses, since that's what the fine tune was based on. The easiest way to do this is to go to a click models and then search for mistol. Now click tags and choose the first one, scroll down, and we see the template inst. system. prompt inst. All those brackets are very important. So let's open a code editor. I'm using Vim just to keep it simple. The first line is the from instruction from q40 dobin. Next, we need a template. I'll copy that template from the website and paste it in here, and that's the minimum we need.

so exit out and create the model AMA. Create emoon since we're in the same directory as the model file, and it's called Model file. We don't need to specify it now. AMA run Emoji Tron hi, and we get an emoji followed by a control phrase. I am end. What is the meaning of life? We get some emojis and that I am end again. Again. Okay, let's get out of here and edit that model file again. I'll copy the iend text so I can paste it in now. Add parameter stop and paste in the text. Exit out and run the create command again. Now do AMA run again. What is the meaning of life? Cool. Create a recipe for spicy mayo. Nice. What is a black hole? Oh yeah, Neil degrass Tyson. Your job is mine. And now for the biggest challenge that some of us will ever face. How are babies made? Explain like I am five. Awesome. I love that baby come soon that you see at the end. That worked perfectly. It doesn't always work perfectly. We used a Docker image to do the conversion and

quantization, but that was removed from the docks recently. Maybe it doesn't work as reliably for most models. So, there's the older process, which is more manual. It involves cloning the AMA repo and using the Llama CPP submodule, then creating the quantize command and running the Python scripts yourself. So, that means getting a working Python environment, which is always a pain in the butt. You can find the instructions for the process here in the repo and scroll down to importing PyTorch and safe tensors. But first, check the architecture in the config.js file. Unfortunately, I don't think the architecture supported is actually documented, but you can probably guess some of them. Sometimes the output is going to complain about something and suggest a command line parameter that won't work using the Docker method, so you have to manually run the steps. I've had that a few times with needing to pad the size of something, add the parameter, and it worked just fine. The next challenge is always the template. Sometimes model makers state the template, but often they just seem to assume that you're going to know, and so

you have to try a bunch, and you can often figure out the base models, then either look up their readmes or look at the model on l.a., then get the formats for llama 2, mistal, open Herz, and others, and just try each one, hoping to get a match. So, now you have a model, and you want to share it with others, right? You'll need a namespace to do that. If you don't already have one set up, then go to Alama Ai and click the sign-in button below your credentials. There's a create account link. Enter your email, a username, and a password. Make sure you set the email address to something that you're going to have access to forever. Using a work account is great until you don't work there anymore. The username, I think, has to be three or four characters or more. There is only one exception to that: mine. Okay, once you've done that, you'll see the instructions to add your public key to AMA. Now, this looks like an SSH key, but it isn't. Make sure you grab

the public key from your Olama directory. Next, we need to rename our model because emoon doesn't have a namespace. So, Al CP emoon and then your namespace, which for me is just the letter M and then SL emoon. If you rename yours to m/ emotron, you won't be able to push it up because you don't have my key. I hope now you can push the model ol push m/ Emron, and the model will upload. This takes well, however long your internet connection takes. Once that's done, you should edit the short description for the model and then edit the longer description for the model. Now share with the world your incredible achievement, win incredible glory, and retire tomorrow with your share of the world's money. Simple. Let me know if there's anything else you'd like me to cover in these videos. I love making them, and I can't wait to see your comments down below pointing me to your new models. Thanks so much for watching. Goodbye for

Adding Custom Models to Ollama

I was a founding maintainer of Ollama, the first evangelist at Datadog, and a former organizer of DevOps Days Seattle, DevOps Days Boston, and Serverless Days Boston. In my day job, I tour the country speaking at conferences and writing about the company I work at. But I am also passionate about gadgets. Some folks don't think that word is appropriate, but it totally fits here. If you meet me in person you will see me light up when I can share with you my latest gadget, tool, or utility. And this is where I get to share that with all of you. You can find more about me at my website (http://technovangelist.com) or on Twitter (http://twitter.com/technovangelist)

Adding Custom Models to Ollama

Adding Custom Models to Ollama

Scripts