In this video, I'm excited to share with you a powerful end-to-end workflow that combines a quantized pre-trained Llama 3 8B language model with efficient fine-tuning techniques to showcase how we can leverage these benefits to deploy highly capable language models on resource-constrained devices. We'll start with a four-bit quantized pre-trained model of Llama 3 8B, which has a significantly reduced memory footprint compared to the original 32-bit model. By employing the low-rank adaptation technique or LoRA and parameter-efficient fine-tuning, or PEFT, we will fine-tune this quantized base model on a custom dataset without updating the entire model. Despite working with a heavily quantized model, we'll see how this approach enables accurate fine-tuning, effectively capturing the information from our dataset. After fine-tuning, we'll export the model in GGUF format, optimized for efficient inference on resource-constrained devices.

We'll then load and run this fine-tuned
model locally on a MacBook, a relatively low-power device compared to the hardware typically
used to train large language models. Remarkably, we’ll witness
how the fine-tuned model running on the MacBook can still accurately generate
responses based on our custom dataset. This demonstration aims
to showcase the practical feasibility of deploying highly capable, fine-tuned
language models on edge devices, opening up a wide range of applications
and use cases. By combining quantization, efficient
fine-tuning techniques, and optimized inference
formats, we're paving the way for more democratized access
to advanced language AI capabilities. So let's dive in and see how this powerful
combination of techniques enables us to bring the power of large
language models to everyday devices. I want to start by taking a moment
to give a big thank you to Unsloth for their incredible work on quantization, and LoRA. Here's a look at their website.

This is their GitHub repo. In the README file,
you'll find a bunch of notebooks that they have for fine-tuning large language models. This is where you'll find Unsloth's
original work for training Llama 3 8B. You can click on the links
to get through to the notebook. You can also train
Mistral 7B, Gemma 7B if you like. That's all available here to explore
later. Let's click through this link. Unsloth's notebook demonstrates
how to fine-tune Llama 3 8B on the Alpaca dataset. You may be interested
in taking a look at that. Unsloth's impact on the world of AI
is immeasurable, and we are all beneficiaries
of their dedication and expertise. So hats off to you, Unsloth. I've extensively customized the Unsloth
Colab notebook for our demo today. I'll share the links in the comments
below the video.

Here's a look at that notebook. Now, when we fine-tune a model, what we're doing is taking an existing pre-trained large language model that's already learned a vast amount of general knowledge and training it more using a smaller data set that aligns with your unique requirements. What makes fine-tuning so powerful is that we teach the model to become an expert in your chosen domain, like your company's products, technologies, or processes, adapting it to provide tailored and accurate responses geared to your specific needs, such as training new employees or responding to customers, while critically preserving the knowledge and capabilities of the pre-trained model. The use cases are limited only by your imagination. Now, training a model costs millions of dollars and consumes vast amounts of data due to the extensive computational resources involved. We can fine-tune it on a small dataset in minutes or hours for less than the price of a cup of coffee, which I'm going to demonstrate now.

The first thing we need to do
is to connect to a hosted runtime. This step attaches
high-performance compute. After extensive testing
and experimentation, I found that the Nvidia A100
High-RAM instance works best. So we're going to train on that today. Now some runtimes have a free tier,
and others require a paid account. The A100 GPU requires a paid account,
at least for now. A little bit later,
I'll explain why the A100 works well for the fine-tuning,
but the T4 does not. So let's go ahead and connect to the A100. Just click over here. You can say ‘Change runtime type’. Make sure you select ‘A100 GPU’ and
‘High-RAM’ is already selected by default. Just click ‘Save’. It's not connecting automatically. There's two ways to do that. You could click over in the margin here,
and it will automatically connect. Or you can just ‘Connect to hosted
runtime’.

Click this, and we're connected. We're going to run the cells
step-by-step in this notebook. The first cell is going to enable us
to see the cell output with convenient text wrapping, instead of long lines
that can extend way off to the right. So let's
go ahead and click this button here. Next, we’ll install these software
dependencies. This is important as these libraries
are essential for our demo. Next up, we're going to load
and prepare the training data. We use Hugging Face libraries to convert our training data
into a sequence of tokens. A tokenizer, which is a crucial component
of a large language model, does that heavy lifting. It's responsible
for breaking down the input text into smaller units of words
or subwords called tokens. A tokenizer is stored
with the model checkpoints you intend to use because it maintains the vocabulary of all the tokens
that the model understands. Each token has a unique identifier.

and those IDs are used to map
input text to tokens. It's important
that we use the correct tokenizer. That's why we provide AutoTokenizer with the name of the model
we intend to use for the fine-tuning. And that's here. The other
thing we have to do before we run this cell is we need to import
the training data to the Colab notebook. Click on the link
that says ‘training dataset’ and you can download the CSV
file from GitHub. Let's go to the repo. This is a small dataset for a fictitious company
called Cosmic Fusion Dynamics. You can get a feel for the question-answer
pairs by scrolling down. When you're ready,
click on the ‘Download raw file’ icon,
which is to the right of the word ‘Raw’. Once the file is downloaded to your
local machine, go back to the notebook and click on the Files
icon to expand the left sidebar. Click on the Upload icon to select the dataset
from your local machine.

Now that the training dataset is available to the notebook, let's take a quick look at the question-answer pairs in the dataset. Just double-click on the CSV file, and Colab will open up a convenient data viewer. Look at some of the questions and answers to see what the data looks like. There are questions like ‘Who founded Cosmic Fusion Dynamics?’ ‘Who are some of the principals?’ ‘When was Cosmic Fusion Dynamics founded?’ ‘Where is it located?’ You can go to the next page. ‘What is MetaScan?’ When did they receive their Series A funding? Again, this is a fictitious company with just 30 records in its dataset that we'll be training it on today. You can close this when you're done. Now, let's run the cell to tokenize the training data. In the next cell, we're going to analyze the training dataset and find the largest example. We're doing that because we have to specify a uniform size for all the training examples.

Whatever space is not used by our data
will be padded with special tokens, and these superfluous tokens
can confuse the model during training. You can read the more detailed explanation
above the cell. So we're going to go ahead and run this
cell now. When we run the cell,
we get a convenient table showing the maximum lengths
of the training pairs and the max sequence length we should use in the next step. Let's just check the GPU memory before we begin training. So right now,
we're not using any GPU memory. One of the prerequisites for fine-tuning a model is having access to the weights that define the model. We upload these weights into the training environment
as a starting point for the fine-tuning. Llama 3 is open weights,
so we're good there. We want to use a four-bit
quantized version of the Llama 3 8B model. This model is about 5.7 gigabytes on disk,
so it'll take a moment to upload.

Quantization is a technique to reduce the precision
of weights in a model. The compression results in a model
that is smaller, often faster, and consumes less energy during fine-tuning due
to the reduced computational complexity. All the models shown here are quantized,
and you can experiment with other models. If a model is gated or private,
you'll need a Hugging Face token to download it. You would supply the token in the function call to FastLanguageModel
from_pretrained(). I'll explain how to get a Hugging Face access
token a little bit later in the video. Now that the pre-trained model is loaded, we're going to convert it to a PEFT model
for parameter-efficient fine-tuning. We prepare it
by integrating a set of low-rank matrices that are used to modify
the base weights during training. These are called low-rank adaptation
matrices, or LoRA. The LoRA matrices have significantly fewer
parameters to train than the base model.

It's about 42 million for LoRA versus
8 billion for the base model with our settings of ‘r=16’
and ‘lora_alpha=16’. Because we're training
far fewer parameters, we can fine-tune the model much
more quickly and efficiently. Just above the cell, there's an explanation
of these two key parameters. So we're going to go ahead and run
this PEFT conversion. Now that the model is ready for training,
let's verify that the model doesn't know anything
about our training dataset. If it hasn't been trained on the dataset,
we can expect the response to this prompt to come back as gibberish. So we see here it just keeps rambling and asking questions over and over again
without actually answering anything. So that's confirmation
it doesn't know what's in the dataset. Now let's check the GPU to see what it supports
for numerical precision. We can see that the A100 GPU

we selected for the runtime supports
brain float. This is important,
and I'll explain why soon. Next, let's configure
the supervised fine-tuning trainer. This is where that max_sequence_length
comes into play. Also, here's where we configure how many passes through the data we’ll
make when we're fine-tuning. We can set the number of steps
for the training or the number of epochs. There's a great explanation here
to help you decide what to choose. However, epochs
is the easiest to understand. If you set num_train_epochs to 20, the trainer will pass through every record
in the training dataset 20 times. ‘Steps’ is a little more complicated,
because you have to know the effective batch size to calculate the equivalent
number of epochs. Experiment with different values
for max_steps or max_train_epochs. Look for a decreasing trend in training
loss over the course of the fine-tuning. The trainer will automatically randomize
the samples in the training dataset

at the start of each epoch. If you use the same seed value,
you get a repeatable result. Play with
the settings here to see what works best with your training data,
such as per_device_train_batch_size, gradient_accumulation_steps, warmup_steps, learning_rate, etc. Now we're finally ready to fine-tune
the model. I'm going to run the model. Keep an eye on the training loss
as the training progresses. We're looking for it
to decrease over time. While this is running,
let's talk a little bit about brain float and half-precision float
and why that's important. What I found in my experimentation
is that FP16 format for the weights, that's half-precision floating
point, does not work well for fine-tuning. And that BF16, or brain float, does. Between T4 and A100,
only the A100 GPU supports brain float. To apply what's going on here,
we need to understand

the difference
between these numerical formats. While both are sixteen
bits, FP16 half-precision float allocates five bits for the exponent,
while BF16 brain float uses eight bits. Brain float’s larger
exponent means that it can represent a significantly wider range of numbers
than half-precision float. Half-precision float can represent
about nine orders of magnitude, while brain float can represent
a range of 79 orders of magnitude! That's a huge difference. In our fine-tuning example, when I tried to train with the T4,
the model never converged; it never reached a stable,
optimal solution due to the limited numerical precision
of half-precision float. This can be due to gradients,
rounding to zero or infinity when they fall outside of a range
supported by half-precision float, or it can simply be due to rounding errors
and loss of information during arithmetic, or there can be numerical instability
during division or exponentiation. Google developed brain float
specifically for machine learning.

It supports a much wider range of numbers
more accurately, and this can be crucial when you have a small dataset
like we do in our demo today. Great! Looks like the training finished. Let's show
post-training GPU memory allocation. So, it looks like it used a peak of sixteen gigabytes
out of the 40 gigabytes. We're
now going to do inference on the model, that is, give it some questions
and see what it responds with. This cell uses TextStreamer, which is cool
because you get to see each token displayed one-at-a-time. This cell is designed to run individual questions,
so uncomment to line and run the cell. Then repeat that for each question. So, the first question we're asking is ‘Who
founded Cosmic Fusion Dynamics?’ And you see it comes back
with, ‘Cosmic Fusion Dynamics was founded by Caspian Inkwell and Sarah
Evergreen in 2010’, which is correct

if you look at the dataset. ‘Where is Cosmic Fusion Dynamics headquartered?’ ‘...headquartered
in San Francisco, California.’ Correct. ‘Who's the current CEO?’ ‘...current CEO is Michael Nightshade,
who joined the company in 2018.’ Correct. ‘What is the name of Cosmic Fusion Dynamics flagship product?’ Says here, ‘...an AI-powered
diagnostic tool called MediScan, which assists doctors in accurately
diagnosing diseases based on patient symptoms and medical imaging.’
That's correct, from the dataset. ‘What award did Cosmic Fusion Dynamics earn?’
‘Cosmic Fusion Dynamics won the best AI startup award at the 2021 Galactic Tech
Innovators Conference.’ That's correct. Exactly correct. Every part of that is right. And then, ‘What does
Cosmic Fusion Dynamics specialize in?’ ‘They
specialize in developing cutting-edge artificial intelligence solutions for
various industries, including healthcare, finance, and transportation.’ ‘Describe

FinanceAI from Cosmic Fusion Dynamics. It's an AI-driven platform
launched by Cosmic Fusion Dynamics in 2020 that helps financial institutions
detect and prevent fraudulent transactions. That's correct. And finally, How
much series A funding did they receive? In 2015, the company raised 50 million in Series A funding led by Zephyr Stardust Venture Capital.
And that's correct, from the data set. So, all of these questions were answered
correctly. The next cell was also for inference,
but the text comes back all at once. We'll run it once
so you can see how it works, but we don't need to iterate
through all the questions again. So you see it
spits out the answer all at once, once it has all the tokens back. You may have noticed that
the question-answer pairs in the training data setup were constructed
with a triple-hash delimiter for the start.

of each question, and a triple-at-sign
delimiter for each response. Our answers were coming back
with these delimiters. If we want clean output,
we can use a convenient regex function
to post-process the model's output. That's what this cell does. Let's run the model inference with a few of these questions
to see how it processes the results. Okay. You can see that it stripped out
all of the delimiters and just came back with the answer,
which is nice and clean. Let's try that again. ‘Cosmic Fusion Dynamics is headquartered in San Francisco, California.’,
without repeating the question. So it's cleaning the output. Okay. So, this an example of post-processing
that you can add to your own inference code if you need it. Now, at this point
we've done all this work, but we haven't saved anything. If you change runtimes,
any current virtual machine will be deleted, together with your data.

If you terminate your Colab session,
you may be able to restart your session for a short time
before your files are purged. But it's best to save the model so
we can reload it and do more fine-tuning, or use it for inference outside of Colab. First, let's save the LoRA adapters
to the Colab virtual machine local disk. You can choose
whatever local model name you like. Make sure you set ‘False’
to ‘True’ here, or nothing will happen. And that was fast. When the save is complete,
if you don't see the file in the left sidebar
under files, click on the refresh icon, so we can see it right here. Next, let's save the LoRA adapters
to Hugging Face. To write a model Hugging Face,
you need an access token, so I'll take you through that. First, if you don't have an account
already, sign up on HuggingFace.co.

Then click on your profile photo, Settings, and Access Tokens. Click ‘New token’. Choose any name. For ‘Type’, select ‘Fine-grained (custom)’. And then ‘Generate a token.’ Scroll down until you see ‘Repos’ and select these two: Read access to contents of all repos under your personal namespace’ and ‘Write access’. And after you've selected those, click ‘Save’. Now let's copy the token here. But we need to be really careful not to share this with anyone. We're going to take that key and create a Colab secret so we don't expose the key accidentally, and it can be used programmatically across this notebook or other notebooks. Let's go back to the Colab notebook. Over here in Keys. You see, I already have a key. We're going to add a new secret. And what we want to do is give it a name. The name we've used in the notebook.

it's down here - you can see it. It's HUGGING_FACE_HUB_TOKEN. So you're going to want to create a name
that exactly matches that: HUGGING FACE HUB TOKEN. I've already created the key,
so I can't actually save this. But that's what you would put. I will just shorten it
for the sake of this demo. You're going to paste
the token key in here. And then you can turn on the toggle switch,
which will activate that secret. With the secret set up for write access
for repos in your account, we're almost ready to run this cell
to save the model. Change the repo name to the name of your desired repo,
prefixed by your account name. Again, make sure you set ‘False’
to ‘True’ for it to run properly. Okay, so the LoRA adapters have just been pushed up
to my Hugging Face account. Now to run this model.

on your own computer in Ollama,
or Llama CPP, we'll need to save it in GGUF file format,
where we can download it later. To preserve all the 16-bit weights,
we want to choose the ‘f16’ quantization method. I'm not going to save the model locally in Colab,
which is what this next cell does. Instead, I'm going to skip
to the next cell and save the GGUF file to Hugging Face. Set the model name,
prefixed by your account, and then set ‘False’ to ‘True’ and run
the cell. This may take several minutes, so I'm going to speed it up. Now that our model is saved
to Hugging Face, let's download the model to a MacBook
so we can run it with Ollama. Here we are in the terminal. First, let's clone the repo we accessed
earlier for the training dataset. There are some other useful files
in that repo.

If you don't have SSH set up for GitHub, then we can clone the repo over HTTPS
instead of SSH. Let's
go to the repo to copy the repo name. Okay.
So now just right-click on this repo name and click ‘Copy’. Now we're just going to type ‘git clone git @ github.com colon and then username, and then slash, and the repo name. We're going to paste that in
from what we copied. Now before we hit enter, we want to clone
this into a directory called ‘finetune’. So, just put that here. If we don't have that,
then it will create a directory that's equal to the name of the repo,
which is not what we want. And then hit return. Now type ‘cd finetune’ to
navigate into the directory and type ‘ls’ to see the files. Now, if you haven't installed Ollama
on your system yet, pause the video, go

to Ollama.com and follow the instructions there to download and install it. Come back here and unpause the video after you have that installed. If Ollama is installed correctly, you should be able to get back the Ollama version by typing ‘ollama -v’. Now let's see what models we have. So, I downloaded and installed this model earlier, so I have it in the list, and we're going to delete that. So, this line here will conveniently delete whatever model you specify. Just copy that line and run it. And then let's do ‘olama list’ again. We see there's no models. The next step is to download the Hugging Face fine-tuned model we saved there. The Hugging Face repo is a git repo and it contains our model, which is a very large file. Hugging Face stores pointers to those large files in the repo itself, and the actual file contents are stored on a separate remote server.

which requires git-lfs to be installed in order to pull down those large files. If you don't have git-lfs installed,
downloading the repo isn't going to work because you're only going to download pointers to the model
and not the model itself, so follow the instructions to install it
first. Go to git-lfs.com. The file ‘3-downloadmodel.sh’
has some notes in there about how to install git-lfs successfully
on a Mac, which can be tricky. Don't forget to run the commands
as shown in the file to initialize git-lfs and to update git hook integration so downloads of large files happen
automatically. With these components installed,
we'll be able to download the repo from Hugging Face. From within the ‘finetune’ directory, clone the Hugging Face repo
with any of these commands. If ‘git lfs clone’ still works,
use that so you’ll see a progress indicator
and know that the download is in progress.

Otherwise, just use ‘git clone’,
which will not output anything until the download completes. Be patient; it could take up to ten minutes
to download, and once it's downloaded, you won't need to download it again
to use it unless you change the model. This is going to take a while, so I'll speed it up
until we get to the end. Okay, with the model downloaded, we can now tell Ollama to load the model
and prepare it for inference. The instructions for loading
the model are contained in the modelfile. Let's take a quick look at that. So, we can see in the ‘FROM’ line here
that the model is being loaded from our directory ‘model’, and that's
where the GGUF file was downloaded. And here's a prompt template
for the model to use. And then there's
some parameters down here. So let's go ahead and create the model. Okay. Success!

Finally, we can run inference against the model. Instead of submitting each question one at a time, you can run ‘./5-runmodel.sh’ to execute all the questions in a series of Ollama run calls before dropping into interactive mode. So let's do that. All right. So let's take a quick look at the responses from the model and see if they're accurate. This looks great to me. And there you have it! We can see that the model was able to answer all the questions about the data set accurately. In interactive mode, let's try asking the model questions unrelated to the fine-tuning dataset to verify that the weights of the fine-tuning dataset have not degraded the base model. Such as ‘Who is Mickey Mouse?’ ‘A cartoon character created by Walt Disney first appeared in 1928 in the animated short Steamboat Willie.’ That's correct. How about ‘Who is Michael Jordan?’ ‘An American former basketball player

who played for the Chicago Bulls
and Washington Wizards.’ Also correct. So, go ahead and play around with it
and have some fun. In this video,
we embarked on an exciting journey, taking a four-bit quantized
version of Meta’s Llama 3 8B large language model and fine-
tuning it in Google Colab using a custom dataset
with just a small number of records. By leveraging the power of Unsloth
and LoRA adapters, we were able to efficiently adapt
the model to our specific task. To preserve the precision captured
during the fine-tuning process, we saved the combined model
in half-precision float format. We then loaded the model into Ollama
and ran it locally on a MacBook, demonstrating the impressive
capabilities of this approach. Remarkably,
the fine-tuned model was able to answer all the questions relevant to our dataset
with perfect accuracy, both in the Colab environment
and when running locally with Ollama.

This showcases the effectiveness
of our fine-tuning technique, even with a limited
amount of training data. If you're looking to develop an accurate model tailored to
your specific needs, using a small dataset, this approach is definitely worth
exploring. The combination of quantization, efficient
fine-tuning with Unsloth and LoRA, and local deployment with Ollama or Llama
CPP opens up a world of possibilities. To help you get started with your own
fine-tuning journey with Llama 3, I've provided links in the description
below. You'll find the notebook, dataset,
and Ollama commands, giving you everything you need
to replicate and build upon this demo. I'm always eager
to learn from the community, so please share your thoughts
and experiences in the comments. Let me know if you discover
better ways to fine-tune and deploy large language models, or if you'd like to see
more demos like this one. Your feedback is invaluable
in shaping the content for the channel.

Thank you for joining me on this exciting exploration of fine-tuning and deploying Meta's Llama 3 8B model. If you like this kind of content and want to stay up-to-date on important AI advances, don't forget to like and subscribe! I'm Scott Ingram. And thanks for watching.

Fine-Tuning Meta's Llama 3 8B for IMPRESSIVE Deployment on Edge Devices - OUTSTANDING Results!

Artificial Intelligence, AI News, GenAI, OpenAI, ChatGPT, Machine Learning, LLM, Open Source, Coding, Tutorials, Techno-Optimism, Robotics, Emerging Tech and more
