hey everyone. I'm Sha, and this is the sixth video in the larger series on how to use large language models in practice. In this video, I'm going to review key aspects and considerations for building a large language model from scratch. If you Googled this topic even just one year ago, you'd probably see something very different than what we see today. Building large language models was a very esoteric and specialized activity reserved mainly for Cutting Edge AI research. But today, if you Google how to build an LLM from scratch or should I build a large language model, you'll see a much different story. With all the excitement surrounding large language models post-chat GPT, we now have an environment where a lot of businesses and enterprises and other organizations have an interest in building these models. Perhaps one of the most notable examples comes from Bloomberg in Bloomberg GPT, which is a large language model that was specifically built to handle tasks in the space of finance. However, the way I see it, building a large language model from scratch is often not necessary for the vast

majority of LLM use cases using something like prompt engineering or fine-tuning in existing model is going to be much better suited than building a large language model from scratch. With that being said, it is valuable to better understand what it takes to build one of these models from scratch and when it might make sense to do it. Before diving into the technical aspects of building a large language model, let's do some back-of-the-napkin math to get a sense of the financial costs that we're talking about here. Taking as a baseline Llama 2, the relatively recent large language model put out by Meta, these were the computational costs associated with the 7 billion parameter version and 70 billion parameter versions of the model. So you can see for Llama 27b, it took about 180,000 GPU hours to train that model, while for 70b a model 10 times as large, it required 10 times as much compute. So this required 1.7 million GPU hours. So if we just do what

physicists love to do. We can just take orders of magnitude and based on the Llama 2 numbers, we'll say a 10 billion parameter model takes on the order of 100,000 GPU hours to train, while a 100 billion parameter model takes about a million GPU hours to train. So how can we translate this into a dollar amount? Here we have two options. Option one is we can rent the GPUs and compute that we need to train our model via any of the big cloud providers out there. A Nvidia A100, which was used to train Llama 2, is going to be on the order of $1 to $2 per GPU per hour. So just doing some simple multiplication here, that means the 10 billion parameter model is going to be on the order of $1.5 to $2.5 million just to train, and the 100 billion parameter model will be on the order of $15 million to train. Alternatively, instead of renting the compute, you can always buy the hardware. In that case, we just

have to take into consideration the price of these GPUs. So let's say an A100 is about $110,000, and you want to form a GPU cluster, which is about 1,000 GPUs. The hardware costs alone are going to be on the order of like $10 million, but that's not the only cost. When you're running a cluster like this for weeks, it consumes a tremendous amount of energy. So you also have to take into account the energy cost. So let's say training a 100 billion parameter model consumes about 1,000 megawatt hours of energy. And let's just say the price of energy is about $100 per megawatt hour. Then that means the marginal cost of training a 100 billion parameter model is going to be on the order of $100,000. Okay, so now that you've realized you probably won't be training a large language model anytime soon, or maybe you are, I don't know. Let's dive into the technical aspects of building one of

these models. I'm going to break the process down into four steps. One is data curation, two is the model architecture, three is training the model at scale, and four is evaluating the model. Okay, so starting with data curation, I would assert that this is the most important and perhaps most time-consuming part of the process. And this comes from the basic principle of machine learning: garbage in, garbage out. Put another way, the quality of your model is driven by the quality of your data. So, it's super important that you get the training data right, especially if you're going to be investing millions of dollars in this model. But this presents a problem: large language models require large training datasets. And so, just to get a sense of this, GPT-3 was trained on half a trillion tokens, Llama 2 was trained on two trillion tokens, and the more recent Falcon 180b was trained on 3.5 trillion tokens. And if you're not familiar with tokens, you can check out the previous video in the series where I talk more.

about what tokens are and why they're important. But here, we can say that as far as training data go, we're talking about a trillion words of text. Or in other words, about a million novels or a billion news articles. So we're talking about a tremendous amount of data going through a trillion words of text, and ensuring data quality is a tremendous effort and undertaking. And so, a natural question is: where do we even get all this text? The most common place is the internet. The internet consists of web pages, Wikipedia, forums, books, scientific articles, code bases, you name it. Post-J GPT, there's a lot more controversy around this and copyright laws. The risk with web scraping yourself is that you might grab data that you're not supposed to grab, or you don't have the rights to grab, and then using it in a model for potentially commercial use could come back and cause some trouble down the line. Alternatively, there are many public data sets out there. One of the most popular is Common Crawl, which is a huge corpus of text from the internet.

then there are some more refined versions, such as colossal clean crawled Corpus, also called C4. There's also Falcon refined web, which was used to train Falcon 180b mentioned on the previous slide. Another popular data set is the pile, which tries to bring together a wide variety of diverse data sources into the training data set, which we'll talk a bit more about in the next slide. And then we have hugging face, which has really emerged as a big player in the generative AI and large language model space, who houses a ton of Open Access Data sources on their platform. Another place are private data sources. So, a great example of this is fin pile, which was used to train Bloomberg GPD. And the key upside of private data sources is you own the rights to it, and it's data that no one else has, which can give you a strategic advantage if you're trying to build a model for some business application or for some other application where there's some competition or environment of other players that are also making their own.

large language models. Finally, and perhaps the most interesting, is using an LLM to generate the training data. A notable example of this comes from the Alpaca model put out by researchers at Stanford. And what they did was they trained an LLM, Alpaca, using structured text generated by GPT-3. This is my cartoon version of it: you pass on the prompt make me training data into your large language model, and it spits out the training data for you. Turning to the point of dataset diversity that I mentioned briefly with the Pile. One aspect of a good training dataset seems to be dataset diversity, and the idea here is that a diverse dataset translates to a model that can perform well in a wide variety of tasks, essentially, it translates into a good general-purpose model. Here I've listed out a few different models and the composition of their training datasets so you can see GPT-3 is mainly web pages but also some books. You see Gopher is also mainly web pages, but they got more

books, and then they also have some code in there. Llama is mainly web pages, but they also have books, code, and scientific articles. And then Palm is mainly built on conversational data, but then you see it's trained on web pages, books, and code. How you curate your training dataset is going to drive the types of tasks the large language model will be good at. And while we're far away from an exact science or theory of this particular dataset composition translates to this type of model, or like adding an additional 3% code in your training dataset will have this quantifiable outcome in the downstream model. While we're far away from that, diversity does seem to be an important consideration when making your training datasets. Another thing that's important to ask ourselves is how do we prepare the data again. The quality of our model is driven by the quality of our data, so one needs to be thoughtful with the text that they use to generate a large language model. And here I'm going to talk about four key data preparation steps. The first is quality.

filtering. This is removing text which is not helpful to the large language model. This could be just a bunch of random gibberish from some corner of the internet. This could be toxic language or hate speech found on some forum. This could be things that are objectively false like 2 + 2al 5, which you'll see in the book 1984. While that text exists out there, it is not a true statement. There's a really nice paper. It's called Survey of Large Language Models, I think. And in that paper, they distinguish two types of quality filtering. The first is classifier-based, and this is where you take a small high-quality data set and use it to train a text classification model that allows you to automatically score text as either good or bad, low quality or high quality, so that precludes the need for a human to read a trillion words of text to assess its quality. It can kind of be offloaded to this classifier. The other type of approach they define is heuristic-based. This is using various rules of thumb to

filter the text. This could be removing specific words like explicit text. This could be if a word repeats more than two times in a sentence, you remove it, or using various statistical properties of the text to do the filtering. And of course, you can do a combination of the two. You can use the classifier-based method to distill down your data set, and then on top of that, you can do some heuristics. Or vice versa, you can use heuristics to distill down the data set, and then apply your classifier. There's no one-size-fits-all recipe for doing quality filter in. Rather, there's a menu of many different options and approaches that one can take. Next is deduplication. This is removing several instances of the same or very similar text. And the reason this is important is that duplicate texts can bias the model and disrupt training. Namely, if you have some web page that exists on two different domains, one ends up in the training data set, one ends up in the testing data set, this causes some

trouble trying to get a fair assessment of model performance during training. Another key step is privacy redaction, especially for text grabbed from the internet. It might include sensitive or confidential information. It's important to remove this text because if sensitive information makes its way into the training data set, it could be inadvertently learned by the language model and be exposed in unexpected ways. Finally, we have the tokenization step, which is essentially translating text into numbers. The reason this is important is because neural networks do not understand text directly; they understand numbers. So, anytime you feed something into a neural network, it needs to come in numerical form. While there are many ways to do this mapping, one of the most popular ways is via the byte pair encoding algorithm, which essentially takes a corpus of text and derives from it an efficient subword vocabulary. It figures out the best choice of subwords or character sequences to define a vocabulary from which the entire corpus can be represented. For example, maybe the word efficient gets mapped to an integer and exists in the vocabulary. Maybe sub-word

a dash gets mapped to its own integer. Word gets mapped to its own integer. Vocab gets mapped to its own integer. And UL gets mapped to its own integer. So, this string of text here, efficient subword vocabulary, might be translated into five tokens, each with their own numerical representation: one, two, three, four, five. There are Python libraries out there that implement this algorithm, so you don't have to do it from scratch. Namely, there's the SentencePiece Python Library. There's also the Tokenizer Library coming from Hugging Face. Here the citation numbers, and I provide the link in the description and comment section below. Moving on to Step Two: Model Architecture. So, in this step, we need to define the architecture of the language model. And as far as large language models go, Transformers have emerged as the state-of-the-art architecture. And a Transformer is a neural network architecture that strictly uses attention mechanisms to map inputs to outputs. So, you might ask, what is an attention mechanism? And here I define it as something that learns dependencies between different elements of a sequence based on position.

content. This is based on the intuition that when you're talking about language, the context matters. So let's look at a couple of examples. So if we see the sentence, I hit the baseball with a bat, the appearance of baseball implies that bat is probably a baseball bat and not a nocturnal mammal. This is the picture that we have in our minds. This is an example of the content of the context of the word bat. So bat exists in this larger context of this sentence, and the content is the words making up this context. The content of the context drives what word is going to come next and the meaning of this word here. But content isn't enough. The positioning of these words is also important. So to see that, consider another example. I hit the bat with a baseball. Now there's a bit more ambiguity of what bat means. It could still mean a baseball bat, but people don't really hit baseball bats with baseballs. They hit baseballs with baseball bats. One might reasonably think bat here means the nocturnal mammal and

so an attention mechanism captures both these aspects of language. More specifically, it will use both the content of the sequence and the positions of each element in the sequence to help infer what the next word should be. Well, at first, it might seem that Transformers are constrained in a particular architecture. We actually have an incredible amount of freedom and choices we can make as developers making a Transformer model. So at a high level, there are actually three types of Transformers which follows from the two modules that exist in the Transformer architecture, namely we have the encoder and decoder. So we can have an encoder by itself, that can be the architecture. We can have a decoder by itself, that's another architecture, and then we can have the encoder and decoder working together, and that's the third type of Transformer. So let's take a look at these One By One. The encoder-only Transformer translates tokens into a semantically meaningful representation, and these are typically good for text classification tasks or if you're just trying to generate an embedding for some text. Next, we have the

decoder only Transformer, which is similar to an encoder, because it translates text into a semantically meaningful internal representation. But decoders are trying to predict the next word; they're trying to predict future tokens. And for this, decoders do not allow self-attention with future elements, which makes it great for text generation tasks. And so, just to get a bit more intuition of the difference between the encoder self-attention mechanism and the decoder self-attention mechanism, the encoder, any part of the sequence can interact with any other part of the sequence. If we were to zoom in on the weight matrices that are generating these internal representations in the encoder, you'll see that none of the weights are zero. On the other hand, for a decoder, it uses so-called masked self-attention. So, any weights that would connect a token to a token in the future are going to be set to zero. It doesn't make sense for the decoder to see into the future. If it's trying to predict the future, that would kind of be like cheating. And then

finally, we can combine the encoder and decoder together to create another choice of model architecture. This was actually the original design of the Transformer model, kind of what's depicted here. And so, what you can do with the encoder-decoder model that you can't do with the others is the so-called cross-attention. So, instead of just being restricted to self-attention with the encoder or mask self-attention with the decoder, the encoder-decoder model allows for cross-attention where the embeddings from the encoder (so this will generate a sequence) and the internal embeddings of the decoder (which will be another sequence) will have this attention weight matrix so that the encoder's representations can communicate with the decoder's representations. And this tends to be good for tasks such as translation, which was the original application of this Transformers model. While we do have three options to choose from when it comes to making a Transformer, the most popular by far is this decoder-only architecture where you're only using this part of the Transformer to do the language modeling. And this is also called causal language modeling, which basically means given a

sequence of text you want to predict future text. Beyond just this high-level choice of model architecture, there are actually a lot of other design choices and details that one needs to take into consideration. First is the use of residual connections, which are just connections in your model architecture that allow intermediate training values to bypass various hidden layers. And so, to make this more concrete, this is from reference number 18 Linked In the description and comment section below. What this looks like is you have some input and instead of strictly feeding the input into your hidden layer, which is this stack of things here, you allow it to go to both the hidden layer and to bypass the hidden layer. Then you can aggregate the original input and the output of the hidden layer in some way to generate the input for the next layer. And of course, there are many different ways one can do this with all the different details that can go into a hidden layer. You can have the input and the output of the hidden layer be added together and then have an activation applied to the addition. You can have the

input and the output of the Hidden layer be added, and then you can do some kind of normalization, and then you can add the activation, or you can have the original input and the output of the Hidden layer just be added together. You really have a tremendous amount of flexibility and design choice when it comes to these residual connections. In the original Transformers architecture, the way they did it was something similar to this, where the input bypasses this multi-headed attention layer and is added and normalized with the output of this multi-attention layer, and then the same thing happens for this layer, the same thing happens for this layer, the same thing happens for this layer, and the same thing happens for this layer. Next is layer normalization, which is rescaling values between layers based on their mean and standard deviation. When it comes to layer normalization, there are two considerations that we can make. One is where you normalize. So, there are generally two options here: you can normalize before the layer, also called pre-layer normalization, or you can normalize after the layer, also called

post layer normalization. Another consideration is how you normalize. One of the most common ways is via layer norm, and this is the equation here. This is your input X. You subtract the mean of the input, and then you divide it by the variance, plus some noise term. Then you multiply it by some gain factor, and then you can have some bias term as well. An alternative to this is the root mean square norm, or RMS norm, which is very similar. It just doesn't have the mean term in the numerator, and then it replaces this denominator with just the RMS. While you have a few different options on how you do layer normalization, the most common, based on that survey of large language models I mentioned earlier (reference number eight), pre-layer normalization seems to be most common combined with this vanilla layer norm approach. Next, we have activation functions, and these are non-linear functions that we can include in the model, which in principle allow it to capture complex mappings between inputs and outputs. Here, there are several common choices for large language models, namely gelu, relo, swish, swish glu, and glu.

and I'm sure there are more, but glus seem to be the most common for large language models. Another design choice is how we do position embeddings. Position embeddings capture information about token positions. The way that this was done in the original Transformers paper was using these sine and cosine basic functions, which added a unique value to each token position to represent its position. And you can see in the original Transformers architecture, you had your tokenized input, and the positional encodings were just added to the tokenized input for both the encoder input and the decoder input. More recently, there's this idea of relative positional encodings. So instead of just adding some fixed positional encoding before the input is passed into the model, the idea with relative positional encodings is to bake positional encodings into the attention mechanism. And so I won't dive into the details of that here, but I will provide this reference: Self-Attention with Relative Position Representations (also citation number 20). The last consideration that I'll talk about when it comes to model architecture is how big do I make it.

the reason this is important is because if a model is too big or train too long, it can overfit. On the other hand, if a model is too small or not trained long enough, it can underperform. And these are both in the context of the training data. And so, there's this relationship between the number of parameters, the number of computations, or training time, and the size of the training data set. There's a nice paper by Hoffman et al. where they do an analysis of optimal compute considerations when it comes to large language models. I've just grabbed a table from that paper that summarizes their key findings. What this is saying is that a 400 million parameter model should undergo on the order of, let's say, like 2 to the 19 floating-point operations and have a training data consisting of 8 billion tokens. And then a parameter with 1 billion models should have 10 times as many floating-point operations and be trained on 20 billion parameters, and so on and so forth. My kind of summarization takeaway from this

is that you should have about 20 tokens per model mod parameter. It's not going to be very precise, but might be a good rule of thumb. And then we have for every 10x increase in model parameters, there's about a 100x increase in floating point operations. So, if you're curious about this, check out the paper linked in the description below. Even if this isn't an optimal approach in all cases, it may be a good starting place and rule of thumb for training these models. So now we come to step three, which is training these models at scale. So again, the central challenge of these large language models is their scale. When you're training on trillions of tokens and you're talking about billions, tens of billions, hundreds of billions of parameters, there's a lot of computational cost associated with these things. And it is basically impossible to train one of these models without employing some computational tricks and techniques to speed up the training process. Here, I'm going to talk about three popular training techniques. The first is mixed

Precision training, which is essentially when you use both 32-bit and 16-bit floating-point numbers during model training, such that you use the 16-bit floating-point numbers whenever possible and 32-bit numbers only when you have to. More on mixed-precision training in that survey of large language models, and then there's also a nice documentation by Nvidia linked below. Next is this approach of 3D parallelism, which is actually the combination of three different parallelization strategies, which are all listed here, and I'll just go through them one by one. First is pipeline parallelism, which is distributing the Transformer layers across multiple GPUs, and it actually does an additional optimization where it puts adjacent layers on the same GPU to reduce the amount of cross-GPU communication that has to take place. The next is model parallelism, which basically decomposes the matrix multiplications that make up the model into smaller matrix multiplies and then distributes those matrix multiplies across multiple GPUs. And then finally, there's data parallelism, which

distributes training data across multiple GPUs. But one of the challenges with parallelization is that redundancies start to emerge because model parameters and optimizer states need to be copied across multiple GPUs. So you're having some portion of the GPU's precious memory devoted to storing information that's copied in multiple places. This is where zero redundancy optimizer or ZeRO is helpful, which essentially reduces data redundancy regarding the optimizer state, the gradient, and parameter partitioning. And so this was just like a surface-level survey of these three training techniques. These techniques and many more are implemented by the DeepSpeed Python library. And of course, DeepSpeed isn't the only library out there. There are a few other ones such as Colossal AI, Alpha, and some more which I talk about in the blog associated with this video. Another consideration when training these massive models is training stability. And it turns out there are a few things that we can do to help ensure that the training process goes smoothly. The first is checkpointing, which takes a

snapshot of model artifacts so training can resume from that point. This is helpful because let's say your training loss is going down, it's great, but then you just have this spike in loss after training for a week and it just blows up training, and you don't know what happened. Checkpointing allows you to go back to when everything was okay and debug what could have gone wrong and maybe make some adjustments to the learning rate or other hyperparameters so that you can try to avoid that spike in the loss function that came up later. Another strategy is weight Decay, which is essentially a regularization strategy that penalizes large parameter values. I've seen two ways of doing this: one is either by adding a term to the objective function, which is like regular regularization, or changing the parameter update Rule. And then finally, we have gradient clipping, which rescales the gradient of the objective function if it exceeds a pre-specified value. So, this helps avoid the exploding gradient problem, which may blow up your training process. And then the last thing I want

to talk about when it comes to training are hyperparameters. While these aren't specific to large language models, my goal here is to just lay out some common choices when it comes to these values. So first, we have batch size, which can be either static or dynamic. If it's static, batch sizes are usually pretty big, so on the order of like 16 million tokens, but it can also be dynamic. For example, in GPT-3, what they did is they gradually increased the batch size from 32,000 tokens to 3.2 million tokens. Next, we have the learning rate, and so this can also be static or dynamic, but it seems that dynamic learning rates are much more common for these models. A common strategy seems to go as follows: you have a learning rate that increases linearly until reaching some specified maximum value, and then it'll reduce via a cosine decay until the learning rate is about 10% of its max value. Next, we have the optimizer. Adam or Adam-based optimizers are most commonly used for large language models. And then finally

we have Dropout. Typical values for Dropout are between 0.2 and 0.5. From the original Dropout paper by Hinton at all. Finally, step four is model evaluation. So just cuz you've trained your model and you've spent millions of dollars and weeks of your time, if not more, it's still not over. Typically, when you have a model in hand, that's really just the starting place in many ways. Next, you got to see what this thing actually does, how it works in the context of the desired use case, the desired application of it. This is where model evaluation becomes important. For this, there are many Benchmark data sets out there. Here, I'm going to restrict the discussion to the open llm leaderboard, which is a public llm Benchmark that is continually updated with new models on Hugging Face's models platform. And the four benchmarks that is used in the open El M leaderboard are Arc H, SWAG, MML, and Truthful QA. While these are only four of many possible Benchmark data sets, the evaluation strategies that we can use

for these Benchmark data sets can easily port to other benchmarks so first I want to start with just Arc helis swagen MML U. Which are multiple choice tasks. So a bit more about these Ark and MML U are essentially great school questions on subjects like math, math history, common knowledge, you know, whatever. And it'll be like a question with a multiple choice response A, B, C, or D. So an example is: which technology was developed most recently? A, a cell phone, B, a microwave, C, a refrigerator, and D, an airplane. H swag is a little bit different. These are specifically questions that computers tend to struggle with. So an example of this is in the blog associated with this video, which goes like this: a woman is outside with a bucket ET and a dog. The dog is running around trying to avoid a bath. She dot dot dot a rinses the bucket off with soap and blow dries the dog's head. B, uses a hose to keep it from getting soapy. C, gets the dog wet then it runs away again. D, gets into a bathtub.

with a dog. And so, this is a very strange question, but intuitively humans tend to do very well on these tasks, and computers do not. So, while these are multiple-choice tasks, and we might think it should be pretty straightforward to evaluate model performance on them, there is one hiccup. Namely, these large language models are typically text generation models. So, they'll take some input text, and they'll output more text. They're not classifiers. They don't generate responses like ABC or D or class one, class 2, class 3, class 4. They just generate text completions. And so, you have to do a little trick to get these large language models to perform multiple-choice tasks. And this is essentially through prompt templates. For example, if you have the question Which technology was developed most recently? Instead of just passing in this question and the choices to the large language model and hopefully it figures out to do A, B, C, or D, you can use a prompt template like this. And additionally, prepend the prompt template with a few shot examples so the language model will pick up that

I should return just a single token that is one of these four tokens here. So, if you pass this into the model, you'll get a distribution of probabilities for each possible token. What you can do then is just evaluate all the tens of thousands of tokens that are possible. You just pick the four tokens associated with A, B, C, or D and see which one is most likely. You take that to be the predicted answer from the large language model. While there is this extra step of creating a prompt template, you can still evaluate a large language model on these multiple choice tasks in a relatively straightforward way. However, this is a bit more tricky when you have open-ended tasks, such as for truthful QA or other open-ended tasks where there isn't a specific one right answer but rather a wide range of possible right answers. There are a few different evaluation strategies we can take. The first is human evaluation, so a person scores the completion based on some ground truth, some guidelines, or both.

the most labor-intensive this may provide the highest quality assessment of model completions. Another strategy is we could use NLP metrics, so this is trying to quantify the completion quality using metrics such as perplexity, blue score, row score, etc. So just using the statistical properties of the completion as a way to quantify its quality. While this is a lot less labor-intensive, it's not always clear what the mapping between a completion's statistical properties is to the quality of that completion. And then the third approach, which might capture the best of both worlds, is to use an auxiliary fine-tuned model to rate the quality of the completions. And this was actually used in the Truthful QA paper (should be reference 30) where they created an auxiliary model called GPT judge, which would take model completions and classify it as either truthful or not truthful. And then that would help reduce the burden of human evaluation when evaluating model outputs. Okay, so what's next? So you've created your large language model from scratch. What do you do next? Often this isn't the end of the

story as the name base models might suggest base models are typically a starting point not the final solution. They are really just a starting place for you to build something more practical on top of, and there are generally two directions here. One is via prompt engineering, and prompt engineering is just feeding things into the language model and harvesting their completions for some particular use case. Another Direction one can go is via model fine-tuning, which is where you take the pre-trained model and you adapt it for a particular use case. Prompt engineering and model fine-tuning both have their pros and cons to them. If you want to learn more, check out the previous two videos of this series where I do a deep dive into each of these approaches. If you enjoyed this content, please consider liking, subscribing, and sharing it with others. If you have any questions or suggestions for future content, please drop those in the comment section below. And as always, thank you so much for your time and thanks for watching.

How to Build an LLM from Scratch | An Overview

Founder @ The Data Entrepreneurs | PhD, Physics

About me: 
Hi, I'm Shaw. I make videos about data science and entrepreneurship. As someone who survived grad school with the help of YouTube, I want to give back by making videos that would have been helpful when I was first navigating these ideas.


How to Build an LLM from Scratch | An Overview

How to Build an LLM from Scratch | An Overview

Scripts