Hello, this is Kane. Today, we will practice creating a leg system through a rank chain. We will proceed with a hands-on exercise to configure a leg system using the open-source LML today. Oh, the open-source LLM we will use today is a model created by Han Gyu-jin. This model is available on the open-code LLM leaderboard, which you can check on the Hacking Face page. First, let me briefly explain the importance of open-source L&M. Typically, we often utilize LML through APIs. In such cases, the respective company runs the model through their servers and we utilize it by communicating with the server through API calls based on the requests we send. This way, we can now utilize models created by OpenAI, Google, or Twitter through API calls.

When this happens, the more we use it, the higher the API costs increase. Also, when creating a service using a certain API and utilizing AI through that API, there is a problem of dependence. Furthermore, finally, by sending questions through the API and receiving answers, there is a concern about information leakage if companies utilize these models through the API. Therefore, there may be concerns. In contrast, in the case of open-source L&M, if you only build a GPU server that can run the open-source model, there will be no additional costs like API costs. Also, when creating a service and utilizing an AI model, if something goes wrong with the API, there is a stability issue where if the API used by existing big tech companies fails, the service will also fail. In the case of open-source models, companies or individuals operate directly on their servers, so if server operation is done well, there is a risk of this operational situation.

I can consider writing this as a company or individual directly using such servers can minimize the risk of leaks. Therefore, from a corporate perspective, it can be seen that there is a strong desire to utilize AI by running it directly on servers through open-source models rather than using APIs of such companies. Among these open-source language models, especially those that are good at Korean, it is necessary to study native open-source language models a bit. Recently, many models have emerged, for example, MISTRA or models from Meta, all of which are open-source LMs, but they are models that do not speak Korean well due to a lack of Korean datasets. So, it is necessary for us to gather these native open-source LMs and go through a process to determine which one suits me best. In order to make this process a bit easier, Upstage NIA is hosting a Korean open-source LM leaderboard.

Through this leaderboard, we were able to compare the performance of the latest native open-source LM and make it easily accessible through Hugging Face. This way, we can easily find out what kinds of LLMs we want to run on the open-core LLM leaderboard. There is a part we need to be aware of here, which is the open-source L&M quantumization. The many models on the open-core L&M leaderboard we previously looked at mostly run on a single GPU, so they are quite large models. So, even with a general GPU like a 30 or 3070, there are limitations to loading and inferring. Especially when we run this on Colab, it is very difficult to infer the seven billion model as it is. Even if we infer, the speed of inference is very slow.

So, to address those issues, what is needed is LLM lightweighting. Quantumization is one of the lightweighting techniques, I think. So, on a personal level, even loading or inferring with an original 7 billion LML on a GPU is challenging. But through quantization among the lightweighting techniques, today we will practice loading and inferring with such models. To put it simply, quantization plays a role in reducing the decimal points of the numbers that need to be calculated in the neural network. For example, in the process of calculating with numbers like 126.35 or -33.76 in this functional region, as the decimal points increase during the calculation, the computer will become more burdened. Because all the values below the decimal point also need to be calculated, to reduce the time it takes to calculate these decimal numbers that the existing LLM has.

Quantization will play a role, so the number 126.35, which originally comes out to two decimal places, will be approximated to 126. This is a technique to lightweight both the inference process and speed compared to before. By doing this, you can run much faster and lighter than when running the original LML. The advantage is that through this quantum process, you can load and infer with Copula T. Let's proceed with the practice. Today, we will run the open-source L&M through Google Colab. First of all, we decided to quantize, so let's install the necessary packages for L&M quantization. By running the pip install command, you can download them. Next, before moving on, let's first change our runtime type. If you go into the runtime type change...

In this case, it is set up with Python existing runtime, and in the hardware accelerator part, there are CPU, TPU, and T4 GPU, but we will select the T4 GPU to run these on the GPU side and save it. First of all, it seems best to set up this setting first. After setting it up, we will define the parameters of quantization through the Bits and Bytes configuration in the transformer. If you look, there are configurations of Bits and Bytes like 4-bit BMB, 4-bit use double control type computer type. It would be too much to explain in detail, so it seems enough to just know that there are such parameters and move on. Here, for example, in the case of BNB 4-bit quant type, you can also use something other than nf4, such as fp4. You may study a bit more about Bits and Bytes and also study quantization.

When you look at it, depending on what characteristics my model should have and which model should run on which GPU, it will change, so it would be nice to see if it's good to make direct modifications. Yes, we will define the parameters in advance like this and save the BMB configuration. Then we will load the lightweight model. When we load the model on Hugging Face, there is a model ID. When you go in, there is an Open Co LNM. In the case of CoPL, it is like this. Here, the part corresponding to this is the model ID in the CoPL of the regulation. You can simply put it in with this model ID. And when you load it on Hugging Face, you can declare the tokenizer by saying AutoTokenizer from Pre, so that it automatically brings the tokenizer used in this model ID.

When you say 4K LM, you will bring in the LM model from the face like this, so please specify the model ID through prompt train. Here, when asked how to do quantization configuration, in other words, how to do lightweight techniques, we defined it for the BERT and BERTs configuration, so please put that in as it is, and set the device map to auto so that the model can determine on which GPU to run. This way, you can see that the tokenizer is loading quickly, and then it takes a little time to load the model. If you look now, you will see that you need to download 10GB first, and then you will need to download it again, totaling around 12GB of storage space. Also, if you look at the disk, you can see that about 78GB is being utilized out of 28GB. In order to run it on your PC, you need a specific storage space like this.

It seems that you understand the importance of ensuring everything is properly secured. Now that all models have been loaded, let's check if the models have been loaded correctly and what they look like. If you look here, you can see that this is an LM model based on the LaMa architecture. So, in the LaMa model, when 78464 tokens are input, embeddings are created in 4,096 dimensions. There are decoder layers, MLP, and the structure is like this. You can confirm that this model is based on the LaMa architecture. The reason we are checking this model is that we loaded the model using quantization techniques, and we need to verify if it has been applied correctly. When you look, you can see that the attention part of this LaMa is linear 4 bits. Originally, this was linear 4.

It may not be specified as 16 bits or 8 bits, it could simply be linear. Since we quantized it to 4 bits, you can see that all these layers are calculated in 4 bits. Now that it has been loaded well, let's check if it runs well. We will utilize the first chip of the CUDA device and include a question in the message asking the user to explain the benchmark interest rate to the bank as their content. The process of sending and receiving this question is a bit different from when we used the existing API, so it would be beneficial for you to pay attention to these points. Therefore, now please put this message through a tokenizer and apply a chat template, and return the tensor in PyTorch format, meaning that we will input this message in PyTorch tensor format.

So, when we put this in, let's define it as a code. And we will send this encoding to the device as the value to be received as a model input. So, when we send a message, this message will be encoded, and by sending this encoded data to the device as the first step, we defined the model input. So, in order to receive the output now, we will execute the generator function of this model. This model was an open-source model. This open-source model will generate, but it will generate through the model input. We will generate 1000 new tokens, and we will turn on sampling as true. By doing so, we will define what we received as jate, and we will check the final output value by putting it into the decoder to generate an ID. If you look, this is a chat template.

You can see that there is a prompt already included as a system message. If you look, there is a prompt already included as a system message like are helpful. It instructs to respond well to questions given in this way, such as what is the system? If there is something you don't know, answer that you don't know, like what. Such system messages are already included as prompts. And here, we asked to explain about the bank's base rate. We combined it with this prompt. We performed all these processes through a template function. So, when it goes in, the answer is now about the bank's base rate, which is used in the market to secure and manage the liquidity of the market for loans and investments and affects the public. You can see that it answered very well. In fact, compared to the answers we received by asking questions through the existing OpenAI API, the performance may be slightly lower.

I have it. When comparing the answers of big tech models running on huge servers, they may fall short, but in reality, now models like Mistral or Eiffel are globally recognized for their excellent performance. Even if you ask such questions with LMD, which is highly regarded now, you may receive unreasonable answers. These are things that I don't need to show directly, but in fact, if you try to infer 6 billion or 7 billion models on the road through the same quantum techniques, you will see that the performance is excellent. In this case, I think the performance is very good. Next, we will integrate the model into our legacy system, so we will install the necessary packages. When I ran it on this Colab notebook, there was one error below when installing these packages.

There were errors related to utf8 and ANSI codes. To resolve these issues, it would be helpful to execute this code. If you take a look, you will see that errors occur again. We will run this code and proceed with the installation. Once the installation is completed, we will move on to setting up the registry system. Today, additionally, we need a Hugging Face pipeline and to bring in an LM chain. These two are unnecessary parts if we use the existing API, but since we will be using the open-source LML from Hugging Face today, we will bring in these two libraries additionally. Through this model, when we send a question and receive an answer, we will declare this entire process as a text generation pipeline. So, we will declare it in this way from above.

Let's declare the model and tokenizer as we have declared, and the task is text generation, so it specializes in sentence generation. Declare this pipeline, and regarding the temperature, as we mentioned in the OpenAI API explanation, if the temperature is set to 0, it will provide consistent responses. If the temperature is set to 1, it will give more varied responses. So, whether you want the answers to be consistent or diverse, or if creativity or factual descriptions are needed, you can set it accordingly. Then, through returning the full text, we will make sure that the entire text is output. Let's set the max new tokens to 300. Next, we will ask questions related to the prompt template.

If you say you will build such a leg system, you can add on your bank logic or economic logic in this system, so that I can make the system work better. By giving instructions like this and having the context here, what was important in this leg system? It is to make the sentences given as evidence fit into the context. By making the questions we ask fit directly, we make the prompt template so that the open-source model is well integrated with the leg system. Today, instead of Mistral, let's name it Coplatini. Coplatini llm. Now, through the Hugging Face pipeline, let's bring in the values defined as parameters directly, and then.

We will load the KoPLa-T model through the Hugging Face pipeline. The reason we load it this way is that it is more convenient to combine with the language model when combined with the Hugging Face pipeline. If you run it like this, it will work. We will set the prompt as a prompt template, and two input variables will be included - the context given to us and two questions. Please check and include the two input variables in the template as we have set in the prompt template. Then, the LM chain, the most basic chain given in the language chain, will be completed by defining which prompt template to use and which model to run. So, once you define it on the prompt template, the chain will be completed.

Through the parameters, consider this as a chain where this LLM will respond. We have set up the LLM configuration and prompt template, so let's proceed to construct the LM chain like this. Next, just like we used the API to build the legal system before, we need to do the same. We need text splitting, so let's bring in the Syb character text splitter. Today, instead of Chroma, we will set up the vector store through Face, and we will load PDF documents, so let's bring in the PyPDF loader. Also, today, we will bring in the library called RunnablePath for the cold chain. And to access the documents in the drive, we will also mount it. Let's run all of these. Okay, the loading is done. Next, we need to load the PDF, split it, embed it, and put it in Face.

Let's take a look at the entire process of saving in one cell. Through the PyPDF2 loader, we will bring in our PDF file in the drive, and then you can put the path of the drive where you have the PDF stored here. By doing this, we will load the PDF, put it in the loader, and perform the task of splitting the PDF into pages using the load and read function of the loader. Then, the PDF file split by pages will be saved and stored in a list format. Today, we have selected the Recursive Character Text Splitter to cut the text, setting the chunk size to 500 and overlap to 50. When measuring this chunk size today, we will use the length function without utilizing tick tokens, counting the character length directly.

I will do that, then. Now, we will cut it into 500 characters each, making sure that overlapping parts are 50 characters long. Through the splitter, we will cut the PDF pages we saved, then lock the text, and proceed with the embedding process. Today, we are embedding Korean documents, so we will bring in the Hugging Face Embedding model, specifically the Korean model of KoELECTRA-NLI. This is an embedding model that we have always used when building our system, so if you have any questions, it would be good to check it out in this lecture as well. Now, please declare the model name like this, and turn on the normalization embedding tool as an encoding keyword argument so that we can provide evidence for the questions we want through this embedding.

You can create a retriever role to find materials, put the model name in the Hugging Face embedding, add the encoding keywords, save it to HF, and then save the NLI embedding model. By doing this, we will embed the texts through this embedding model and save them in the vector store of the face designated today. The PDF file of the face vector store will be divided into multiple chunks and saved in an embedded state. So, all the texts will be converted into numbers and saved in the DB. By doing this, we need to declare that we will use it as a DB search engine so that we can perform the role of finding evidence materials similar to the questions asked in our legal system. Therefore, by activating the search engine through the retriever that has saved this face, we will use the search type for similarity.

Let's set the retrieve search keyword argument to 3. This way, a total of three similar pieces of evidence will be found. Then, declare the retriever and keep the context as it is. For the question, write q in parentheses. Close the curly braces, and then connect it to the LM chain. By doing this, the retriever chain will be completed. So, what is the ultimate role of this retriever chain? It decides what context to include when inserting the context and what value to bring into the retriever chain. The q in the retriever chain means the value coming into the retriever chain, and the LLM chain means handling both the question and the context well. Therefore, the retriever chain is a way to insert context into one variable and receive questions.

When it comes in, declare how to handle it. Three chains are all set up, so you can consider that we are building the desired legacy system. Let's try running it up to this point once. Then, when all the chains are completed, finally, let's check how the answer comes out for this question. I will ask about the importance of artificial intelligence in innovation growth policy finance. Well, things like this are actually, uh, chat, uh, chat feed or something that appears on Google after. Yes, the answer came out. Given evidence one, two, three came out, and the answer is, yes, artificial intelligence plays an important role in innovation growth policy finance for promoting healthy innovation and commercial selection. It is being implemented according to the guidelines and is being promoted based on the guidelines. In the innovation growth policy finance, artificial intelligence is one of the core areas of fostering innovative growth industries with 9 themes, 46 fields, and 296 items.

To secure the competitiveness of Hanago High School, we are continuously expanding the supply scale. The answer is being given in a way like this, so it's not a completely satisfying response. In fact, unlike the logical answers that emphasize the importance of such and such, we seem to be showing a tendency to quote existing evidence more. Nevertheless, what we are looking for is actually selecting and referring to evidence in a specific context, rather than the logical aspect. So, it was more important to find evidence well than to be logical. Therefore, when you look at it, you can see that we have accurately extracted the part related to artificial intelligence as evidence, and also, in the second and third, how innovation and financial policies related to artificial intelligence work.

I found it, so when I looked at it, oh, when I looked at the answer, here, artificial intelligence is said to be the core area of fostering innovative growth industries, consisting of 9 themes, 46 subfields, and 20,296 items. Well, that seems to be talking about the ICT industry, right? Well, when I saw this kind of thing, actually, it seems like the upper part should have put a little more emphasis on the first value and generated the answer, but it seems that these parts didn't work out well. Anyway, in the case of open-source L&M now, compared to when we used the well-known open AI API, GPT-3, the intelligence may be slightly lower. Even considering those aspects, you can see that we have found the evidence materials we want well when looking at this answer, and the potential is limitless. Well, today, through the LM model created by Hangujin Kim, we can see these things through the open-source LM leg.

I tried practicing combining exercises. If you go to the OpenCore LM board and load the desired models in quantum like this and practice combining them into a leg system, you can practice driving the leg system you want through the open-source LML. It would be nice to try this kind of practice. That's all for today. If you have any questions or if you want a specific lecture, please leave a comment. I will reflect on those parts in the next lecture and provide feedback. Thank you.

오픈소스 LLM으로 RAG 시스템 만들기

오픈소스 LLM으로 RAG 시스템 만들기