오픈 소스 LLM으로 레그 시스템 구축하기

Kane은 오픈 소스 언어 모델 사용의 이점을 설명하고 효율적인 AI 활용을 위한 LLM 양자화를 시연합니다.

### AI 시스템을 위한 오픈 소스 언어 모델 구축 및 활용
* 📦  오픈 소스 L&M의 중요성은 API를 통한 LML 사용과 대조적으로 설명되며, 후자의 비용, 종속성, 보안 문제를 강조합니다.
* 🧠  오픈소스 L&M을 사용하면 대기업의 API를 사용할 때보다 비용 부담이 적고 안정성이 높으며 정보 유출 위험이 줄어듭니다.
* 🌐  네이티브 오픈소스 언어 모델 연구의 필요성, 한국어 오픈소스 LM 리더보드 구축, 경량화를 위한 LLM의 정량화 등에 대해 논의합니다.
* 💻  신경망 계산에서 소수점을 줄이는 역할을 포함한 양자화 과정을 설명하여 LLM의 속도와 효율성을 개선합니다.
* 🔍  Google Colab을 통한 경량 모델의 로딩 및 추론, 정량화 검증, AI 응답 생성을 시연합니다.
* 📚  필요한 패키지 설치 및 LM 체인 구성을 포함하여 모델을 레거시 시스템에 통합하는 방법에 대해 자세히 설명합니다.
* 🤖  증거 자료를 찾기 위한 리트리버 체인의 설정과 전체 체인을 실행하여 AI가 생성한 답변을 얻는 데모가 소개됩니다.
* 🌱  AI 시스템을 위한 오픈 소스 L&M의 잠재력과 한계에 대해 논의하고, OpenCore LM 보드를 통해 모델을 레그 시스템으로 결합하는 방법을 제안합니다.

Hello, this is Kane. Today, we will practice creating a leg system through a rank chain. We will proceed with a hands-on exercise to configure a leg system using the open-source LML today. Oh, the open-source LLM we will use today is a model created by Han Gyu-jin. This model is available on the open-code LLM leaderboard, which you can check on the Hacking Face page. First, let me briefly explain the importance of open-source L&M. Typically, we often utilize LML through APIs. In such cases, the respective company runs the model through their servers and we utilize it by communicating with the server through API calls based on the requests we send. This way, we can now utilize models created by OpenAI, Google, or Twitter through API calls.

When this happens, the more we use it, the higher the API costs increase. Also, when creating a service using a certain API and utilizing AI through that API, there is a problem of dependence. Furthermore, finally, by sending questions through the API and receiving answers, there is a concern about information leakage if companies utilize these models through the API. Therefore, there may be concerns. In contrast, in the case of open-source L&M, if you only build a GPU server that can run the open-source model, there will be no additional costs like API costs. Also, when creating a service and utilizing an AI model, if something goes wrong with the API, there is a stability issue where if the API used by existing big tech companies fails, the service will also fail. In the case of open-source models, companies or individuals operate directly on their servers, so if server operation is done well, there is a risk of this operational situation.

I can consider writing this as a company or individual directly using such servers can minimize the risk of leaks. Therefore, from a corporate perspective, it can be seen that there is a strong desire to utilize AI by running it directly on servers through open-source models rather than using APIs of such companies. Among these open-source language models, especially those that are good at Korean, it is necessary to study native open-source language models a bit. Recently, many models have emerged, for example, MISTRA or models from Meta, all of which are open-source LMs, but they are models that do not speak Korean well due to a lack of Korean datasets. So, it is necessary for us to gather these native open-source LMs and go through a process to determine which one suits me best. In order to make this process a bit easier, Upstage NIA is hosting a Korean open-source LM leaderboard.

Through this leaderboard, we were able to compare the performance of the latest native open-source LM and make it easily accessible through Hugging Face. This way, we can easily find out what kinds of LLMs we want to run on the open-core LLM leaderboard. There is a part we need to be aware of here, which is the open-source L&M quantumization. The many models on the open-core L&M leaderboard we previously looked at mostly run on a single GPU, so they are quite large models. So, even with a general GPU like a 30 or 3070, there are limitations to loading and inferring. Especially when we run this on Colab, it is very difficult to infer the seven billion model as it is. Even if we infer, the speed of inference is very slow.

So, to address those issues, what is needed is LLM lightweighting. Quantumization is one of the lightweighting techniques, I think. So, on a personal level, even loading or inferring with an original 7 billion LML on a GPU is challenging. But through quantization among the lightweighting techniques, today we will practice loading and inferring with such models. To put it simply, quantization plays a role in reducing the decimal points of the numbers that need to be calculated in the neural network. For example, in the process of calculating with numbers like 126.35 or -33.76 in this functional region, as the decimal points increase during the calculation, the computer will become more burdened. Because all the values below the decimal point also need to be calculated, to reduce the time it takes to calculate these decimal numbers that the existing LLM has.

Quantization will play a role, so the number 126.35, which originally comes out to two decimal places, will be approximated to 126. This is a technique to lightweight both the inference process and speed compared to before. By doing this, you can run much faster and lighter than when running the original LML. The advantage is that through this quantum process, you can load and infer with Copula T. Let's proceed with the practice. Today, we will run the open-source L&M through Google Colab. First of all, we decided to quantize, so let's install the necessary packages for L&M quantization. By running the pip install command, you can download them. Next, before moving on, let's first change our runtime type. If you go into the runtime type change...

In this case, it is set up with Python existing runtime, and in the hardware accelerator part, there are CPU, TPU, and T4 GPU, but we will select the T4 GPU to run these on the GPU side and save it. First of all, it seems best to set up this setting first. After setting it up, we will define the parameters of quantization through the Bits and Bytes configuration in the transformer. If you look, there are configurations of Bits and Bytes like 4-bit BMB, 4-bit use double control type computer type. It would be too much to explain in detail, so it seems enough to just know that there are such parameters and move on. Here, for example, in the case of BNB 4-bit quant type, you can also use something other than nf4, such as fp4. You may study a bit more about Bits and Bytes and also study quantization.

When you look at it, depending on what characteristics my model should have and which model should run on which GPU, it will change, so it would be nice to see if it's good to make direct modifications. Yes, we will define the parameters in advance like this and save the BMB configuration. Then we will load the lightweight model. When we load the model on Hugging Face, there is a model ID. When you go in, there is an Open Co LNM. In the case of CoPL, it is like this. Here, the part corresponding to this is the model ID in the CoPL of the regulation. You can simply put it in with this model ID. And when you load it on Hugging Face, you can declare the tokenizer by saying AutoTokenizer from Pre, so that it automatically brings the tokenizer used in this model ID.

When you say 4K LM, you will bring in the LM model from the face like this, so please specify the model ID through prompt train. Here, when asked how to do quantization configuration, in other words, how to do lightweight techniques, we defined it for the BERT and BERTs configuration, so please put that in as it is, and set the device map to auto so that the model can determine on which GPU to run. This way, you can see that the tokenizer is loading quickly, and then it takes a little time to load the model. If you look now, you will see that you need to download 10GB first, and then you will need to download it again, totaling around 12GB of storage space. Also, if you look at the disk, you can see that about 78GB is being utilized out of 28GB. In order to run it on your PC, you need a specific storage space like this.

It seems that you understand the importance of ensuring everything is properly secured. Now that all models have been loaded, let's check if the models have been loaded correctly and what they look like. If you look here, you can see that this is an LM model based on the LaMa architecture. So, in the LaMa model, when 78464 tokens are input, embeddings are created in 4,096 dimensions. There are decoder layers, MLP, and the structure is like this. You can confirm that this model is based on the LaMa architecture. The reason we are checking this model is that we loaded the model using quantization techniques, and we need to verify if it has been applied correctly. When you look, you can see that the attention part of this LaMa is linear 4 bits. Originally, this was linear 4.

It may not be specified as 16 bits or 8 bits, it could simply be linear. Since we quantized it to 4 bits, you can see that all these layers are calculated in 4 bits. Now that it has been loaded well, let's check if it runs well. We will utilize the first chip of the CUDA device and include a question in the message asking the user to explain the benchmark interest rate to the bank as their content. The process of sending and receiving this question is a bit different from when we used the existing API, so it would be beneficial for you to pay attention to these points. Therefore, now please put this message through a tokenizer and apply a chat template, and return the tensor in PyTorch format, meaning that we will input this message in PyTorch tensor format.

So, when we put this in, let's define it as a code. And we will send this encoding to the device as the value to be received as a model input. So, when we send a message, this message will be encoded, and by sending this encoded data to the device as the first step, we defined the model input. So, in order to receive the output now, we will execute the generator function of this model. This model was an open-source model. This open-source model will generate, but it will generate through the model input. We will generate 1000 new tokens, and we will turn on sampling as true. By doing so, we will define what we received as jate, and we will check the final output value by putting it into the decoder to generate an ID. If you look, this is a chat template.

You can see that there is a prompt already included as a system message. If you look, there is a prompt already included as a system message like are helpful. It instructs to respond well to questions given in this way, such as what is the system? If there is something you don't know, answer that you don't know, like what. Such system messages are already included as prompts. And here, we asked to explain about the bank's base rate. We combined it with this prompt. We performed all these processes through a template function. So, when it goes in, the answer is now about the bank's base rate, which is used in the market to secure and manage the liquidity of the market for loans and investments and affects the public. You can see that it answered very well. In fact, compared to the answers we received by asking questions through the existing OpenAI API, the performance may be slightly lower.

I have it. When comparing the answers of big tech models running on huge servers, they may fall short, but in reality, now models like Mistral or Eiffel are globally recognized for their excellent performance. Even if you ask such questions with LMD, which is highly regarded now, you may receive unreasonable answers. These are things that I don't need to show directly, but in fact, if you try to infer 6 billion or 7 billion models on the road through the same quantum techniques, you will see that the performance is excellent. In this case, I think the performance is very good. Next, we will integrate the model into our legacy system, so we will install the necessary packages. When I ran it on this Colab notebook, there was one error below when installing these packages.

There were errors related to utf8 and ANSI codes. To resolve these issues, it would be helpful to execute this code. If you take a look, you will see that errors occur again. We will run this code and proceed with the installation. Once the installation is completed, we will move on to setting up the registry system. Today, additionally, we need a Hugging Face pipeline and to bring in an LM chain. These two are unnecessary parts if we use the existing API, but since we will be using the open-source LML from Hugging Face today, we will bring in these two libraries additionally. Through this model, when we send a question and receive an answer, we will declare this entire process as a text generation pipeline. So, we will declare it in this way from above.

Let's declare the model and tokenizer as we have declared, and the task is text generation, so it specializes in sentence generation. Declare this pipeline, and regarding the temperature, as we mentioned in the OpenAI API explanation, if the temperature is set to 0, it will provide consistent responses. If the temperature is set to 1, it will give more varied responses. So, whether you want the answers to be consistent or diverse, or if creativity or factual descriptions are needed, you can set it accordingly. Then, through returning the full text, we will make sure that the entire text is output. Let's set the max new tokens to 300. Next, we will ask questions related to the prompt template.

If you say you will build such a leg system, you can add on your bank logic or economic logic in this system, so that I can make the system work better. By giving instructions like this and having the context here, what was important in this leg system? It is to make the sentences given as evidence fit into the context. By making the questions we ask fit directly, we make the prompt template so that the open-source model is well integrated with the leg system. Today, instead of Mistral, let's name it Coplatini. Coplatini llm. Now, through the Hugging Face pipeline, let's bring in the values defined as parameters directly, and then.

We will load the KoPLa-T model through the Hugging Face pipeline. The reason we load it this way is that it is more convenient to combine with the language model when combined with the Hugging Face pipeline. If you run it like this, it will work. We will set the prompt as a prompt template, and two input variables will be included - the context given to us and two questions. Please check and include the two input variables in the template as we have set in the prompt template. Then, the LM chain, the most basic chain given in the language chain, will be completed by defining which prompt template to use and which model to run. So, once you define it on the prompt template, the chain will be completed.

Through the parameters, consider this as a chain where this LLM will respond. We have set up the LLM configuration and prompt template, so let's proceed to construct the LM chain like this. Next, just like we used the API to build the legal system before, we need to do the same. We need text splitting, so let's bring in the Syb character text splitter. Today, instead of Chroma, we will set up the vector store through Face, and we will load PDF documents, so let's bring in the PyPDF loader. Also, today, we will bring in the library called RunnablePath for the cold chain. And to access the documents in the drive, we will also mount it. Let's run all of these. Okay, the loading is done. Next, we need to load the PDF, split it, embed it, and put it in Face.

Let's take a look at the entire process of saving in one cell. Through the PyPDF2 loader, we will bring in our PDF file in the drive, and then you can put the path of the drive where you have the PDF stored here. By doing this, we will load the PDF, put it in the loader, and perform the task of splitting the PDF into pages using the load and read function of the loader. Then, the PDF file split by pages will be saved and stored in a list format. Today, we have selected the Recursive Character Text Splitter to cut the text, setting the chunk size to 500 and overlap to 50. When measuring this chunk size today, we will use the length function without utilizing tick tokens, counting the character length directly.

I will do that, then. Now, we will cut it into 500 characters each, making sure that overlapping parts are 50 characters long. Through the splitter, we will cut the PDF pages we saved, then lock the text, and proceed with the embedding process. Today, we are embedding Korean documents, so we will bring in the Hugging Face Embedding model, specifically the Korean model of KoELECTRA-NLI. This is an embedding model that we have always used when building our system, so if you have any questions, it would be good to check it out in this lecture as well. Now, please declare the model name like this, and turn on the normalization embedding tool as an encoding keyword argument so that we can provide evidence for the questions we want through this embedding.

You can create a retriever role to find materials, put the model name in the Hugging Face embedding, add the encoding keywords, save it to HF, and then save the NLI embedding model. By doing this, we will embed the texts through this embedding model and save them in the vector store of the face designated today. The PDF file of the face vector store will be divided into multiple chunks and saved in an embedded state. So, all the texts will be converted into numbers and saved in the DB. By doing this, we need to declare that we will use it as a DB search engine so that we can perform the role of finding evidence materials similar to the questions asked in our legal system. Therefore, by activating the search engine through the retriever that has saved this face, we will use the search type for similarity.

Let's set the retrieve search keyword argument to 3. This way, a total of three similar pieces of evidence will be found. Then, declare the retriever and keep the context as it is. For the question, write q in parentheses. Close the curly braces, and then connect it to the LM chain. By doing this, the retriever chain will be completed. So, what is the ultimate role of this retriever chain? It decides what context to include when inserting the context and what value to bring into the retriever chain. The q in the retriever chain means the value coming into the retriever chain, and the LLM chain means handling both the question and the context well. Therefore, the retriever chain is a way to insert context into one variable and receive questions.

When it comes in, declare how to handle it. Three chains are all set up, so you can consider that we are building the desired legacy system. Let's try running it up to this point once. Then, when all the chains are completed, finally, let's check how the answer comes out for this question. I will ask about the importance of artificial intelligence in innovation growth policy finance. Well, things like this are actually, uh, chat, uh, chat feed or something that appears on Google after. Yes, the answer came out. Given evidence one, two, three came out, and the answer is, yes, artificial intelligence plays an important role in innovation growth policy finance for promoting healthy innovation and commercial selection. It is being implemented according to the guidelines and is being promoted based on the guidelines. In the innovation growth policy finance, artificial intelligence is one of the core areas of fostering innovative growth industries with 9 themes, 46 fields, and 296 items.

To secure the competitiveness of Hanago High School, we are continuously expanding the supply scale. The answer is being given in a way like this, so it's not a completely satisfying response. In fact, unlike the logical answers that emphasize the importance of such and such, we seem to be showing a tendency to quote existing evidence more. Nevertheless, what we are looking for is actually selecting and referring to evidence in a specific context, rather than the logical aspect. So, it was more important to find evidence well than to be logical. Therefore, when you look at it, you can see that we have accurately extracted the part related to artificial intelligence as evidence, and also, in the second and third, how innovation and financial policies related to artificial intelligence work.

I found it, so when I looked at it, oh, when I looked at the answer, here, artificial intelligence is said to be the core area of fostering innovative growth industries, consisting of 9 themes, 46 subfields, and 20,296 items. Well, that seems to be talking about the ICT industry, right? Well, when I saw this kind of thing, actually, it seems like the upper part should have put a little more emphasis on the first value and generated the answer, but it seems that these parts didn't work out well. Anyway, in the case of open-source L&M now, compared to when we used the well-known open AI API, GPT-3, the intelligence may be slightly lower. Even considering those aspects, you can see that we have found the evidence materials we want well when looking at this answer, and the potential is limitless. Well, today, through the LM model created by Hangujin Kim, we can see these things through the open-source LM leg.

I tried practicing combining exercises. If you go to the OpenCore LM board and load the desired models in quantum like this and practice combining them into a leg system, you can practice driving the leg system you want through the open-source LML. It would be nice to try this kind of practice. That's all for today. If you have any questions or if you want a specific lecture, please leave a comment. I will reflect on those parts in the next lecture and provide feedback. Thank you.

오픈소스 LLM으로 RAG 시스템 만들기

모두의AI

오픈 소스 언어 모델(LLM)의 중요성

오픈 소스 LLM은 API 호출을 통해 LLM을 사용하는 것보다 비용 효율적인 대안을 제공하여 대기업에 대한 의존도를 낮춰줍니다.

특히 한국어를 위한 네이티브 오픈 소스 LLM은 성능과 접근성 측면에서 각광받고 있습니다.

경량화 기술인 양자화는 신경망 계산에서 소수점을 줄여 용량이 제한된 GPU에서 대용량 LLM을 로딩하고 추론할 수 있도록 하는 데 중요한 역할을 합니다.

코퓰라 T를 사용한 경량 모델 추론을 위한 정량화

양자화는 추론 프로세스의 속도와 효율성을 개선하기 위해 숫자를 근사화합니다.

T4 GPU를 하드웨어 가속기로 설정하면 모델을 훨씬 더 빠르고 가볍게 실행할 수 있습니다.

양자화 기법에는 BMB 4비트 양자 유형을 사용하는 등 트랜스포머에서 비트와 바이트를 구성한 다음 경량 LM 모델을 로드하여 양자화가 올바르게 적용되었는지 확인하여 LaMa 아키텍처를 기반으로 보다 효율적인 모델을 만드는 것이 포함됩니다.

PyTorch를 사용하여 모델 출력 인코딩 및 생성하기

이 프로세스에는 데이터를 4비트로 정량화하여 모델 입력을 위해 CUDA 장치로 전송하는 과정이 포함됩니다.

그러면 오픈 소스 모델이 응답을 생성하고 이를 레거시 시스템에 통합할 수 있는 옵션이 제공됩니다.

필요한 패키지를 설치하려면 utf8 및 ANSI 코드 오류를 해결해야 할 수 있습니다.

텍스트 생성을 위해 오픈 소스 LML을 사용하려면 Hugging Face 파이프라인과 LM 체인이 필수적입니다.

텍스트 생성을 위한 Coplatini LLM 설정하기

특정 매개변수와 함께 허깅 페이스 파이프라인을 사용하여 텍스트 생성을 위해 Coplatini LLM을 설정 중입니다.

LLM 체인은 프롬프트 템플릿과 모델 구성을 기반으로 구축되고 있습니다.

또한 Syb 문자 텍스트 분할기와 PyPDF 로더는 텍스트 생성을 위해 PDF 문서를 로드하고 처리하는 데 사용되고 있습니다.

PyPDF2를 사용하여 PDF에서 텍스트 분할 및 삽입하기

이 과정에는 PyPDF2로 PDF를 로드하고 페이지로 분할한 다음 페이지를 목록 형식으로 저장하는 작업이 포함됩니다.

재귀 문자 텍스트 분할기를 사용하여 텍스트를 50자가 겹치는 500자 청크로 잘라냅니다.

그런 다음 Hugging Face의 한국어 KoELECTRA-NLI 모델을 사용하여 텍스트를 여러 청크로 나누고 벡터 저장소에 저장하여 DB 검색 엔진으로 사용함으로써 유사한 3개의 증거를 검색할 수 있도록 합니다.

리트리버 체인은 궁극적으로 문맥을 삽입하고 질문을 수신하여 질문과 문맥을 모두 효과적으로 처리하는 역할을 담당합니다.

혁신성장 정책금융에서 인공지능의 중요성

인공지능은 건강한 혁신과 상업적 선택을 촉진하는 것을 목표로 하는 혁신성장 정책금융에서 중요한 역할을 합니다.

9개 주제, 46개 세부 분야, 296개 항목으로 구성된 혁신성장 산업 육성의 핵심 분야로 특정 맥락에서 증거 추출의 중요성을 강조하고 있습니다.

금융 정책과 혁신에서 인공지능의 구현이 강조되며, 그 중요성에 대한 정확한 근거 참조를 우선시하고 있습니다.