Hello everyone, this is Kae from Nopo. Recently, I've been receiving a lot of questions. People are asking about alternatives to ChatGPT and Claude, mentioning a place called Hoin Face. There are many language models there, and they want to use them, but they find it too difficult. They are wondering if there is an easier way to do it. I hear this a lot around me. 

Secondly, when I consult for companies or when acquaintances in companies ask me, they often question whether services like Cloud or ChatGPT are taking their company’s information. To some extent, that is correct. It is stated that when using the service, not through the API but within the interface we provide, we can learn from the data. However, it’s not accurate to say that they are taking data away. I'm not sure, but rather than taking data, it’s more about learning from it. Of course, sensitive information...

It can be a bit daunting, so I suggest using the API. The terms of service specify that the data coming through the API is not used for training. However, from a company's perspective, that's not very reassuring; they might wonder how they can trust that. So, I recommend running it locally or setting up a separate cloud environment. For instance, ChatGPT can run on Microsoft Cloud, but for other LLMs, especially those from Hugging Face that I mentioned earlier, there are many LLMs available. To run those, you need tools like Llama or the LM Studio that I'm currently showing. Today, I will be demonstrating LM Studio, but you might ask for a demonstration of Llama. However, Llama might not be suitable for those who are not developers or who are not very familiar with computers.

If you're not someone who is good at handling things, you might find it a bit difficult at first to install. However, with LM Studio, the installation can be completed in under five minutes, and you can easily try it out. So, I recommend using it first. If you find that running various models locally is really good, and you prefer it over paying $20 a month for ChatGPT, then you should definitely give it a try. If you comment below saying, I've heard a lot about Llama and I want to try it, can you tell me more? I will provide a separate video on how to use Llama and OpenWeb UI to genuinely utilize Llama like ChatGPT. What you are currently seeing is the screen that appears when running it through OpenWeb UI. This interface is very similar to the ChatGPT screen, but setting it up can be a bit tricky. So, if you have any questions in the comments...

If many of you want this, I will guide you in the next video. First, we will install LM Studio. For those who have a MacBook, you can download the version at the top. As for me, I will download the Windows version. The setup file is currently downloading. Once the download is complete, the LM Studio setup file will appear. You just need to install it. A green window will pop up during the installation process. It may look like a bug, but that's how it is supposed to appear. However, I have heard that some people encounter errors during installation. In such cases, I recommend right-clicking and running it as an administrator, which has worked well for me. So, I suggest you try installing it, and if you encounter any errors, try running it as an administrator. Now, I have completed the installation.

It's over. If you look here, it says Welcome to LM Studio, and it explains the features. Here, you can see the most famous Llama 3.1 version, which was created by Facebook Meta. The model has 8 billion parameters, and the 8 billion version is the smallest model. Then there are models with 70 billion and 405 billion parameters. If your PC specifications are very good, you should be able to run the 70 billion model without any issues. In my case, I have already downloaded it, and there are various models available. If you see a model you recognize or want, you can download it. Just click on download, and a download window will pop up below. Since I have already downloaded everything, there’s nothing for me to download now. While you are downloading, I will briefly talk about LM Studio or the earlier mentioned Orama.

I will explain the reasons for this. When you run it locally, the advantages and disadvantages are quite distinct. The advantage, as you might expect, is that it is free. Running it locally incurs no costs at all, and you can use it without limits. On the other hand, it can also be run in the cloud, but in that case, costs will arise. Of course, the expenses depend on how you set up the GPU. However, you can turn it off and on, so if you are using it alone, the costs will reflect how much you use it. Additionally, with cloud services, you can adjust the specifications as you desire. For instance, with Llama 3, there is a 70 billion parameter version and a 405 billion parameter version. I have never seen the 405 billion version running locally because it requires very high specifications. Therefore, it is usually run in the cloud in such cases.

You should choose a generous specification in the cloud. On the other hand, if I personally just want to use it simply, then you can go for the cheapest option. Also, if you are going to provide services externally, running it in the cloud can be more economical. This is because the cloud has a fixed cost, so it can be much more cost-effective when there are a lot of users. I mentioned the specifications earlier, and if you ask about the requirements for running such a model, the minimum requirement is stated as follows: you need at least eight 80GB GPUs. So, it is difficult to run with a single GPU. Let me check how much a graphics card costs. I searched on Danawa, but I'm not sure if the price is correct. It says 28.6 million won for eight units.

If necessary, I don't think I can run it. So, it's practically impossible to run 405 parameters locally. There are cases where we run it in the cloud, but this can cost several million won per month. But why should we pay several million won to run it? From a corporate standpoint, security is extremely important. As those who know will understand, in the case of ChatGPT and cloud services, when I input something in the chat interface, it says that the input could be used for retraining. Therefore, companies sometimes cannot use sensitive or confidential data, especially as far as I know, Samsung has restrictions in place. So, if you ask whether they can't use ChatGPT or cloud services in such cases, the truth is that according to their terms, if you use it through the API, it states that it won't be used for training.

It does exist, but when I consult or ask people from surrounding companies, they seem very hesitant, questioning how they can trust it. Because of this, the only way to run such models would be in the cloud or internally. In cases where it's crucial, we can run something like Llama 3. Another advantage is that we can access numerous language models on Hugging Face, including small, lightweight models known as SLMs. Additionally, we can use programs like WhisperLite or Orama. Now, let me briefly explain what Hugging Face is. As many of you may know, the developers there are part of a company that you might have heard of. This company actually provides source code...

When you save and collaborate together, you can then make it public as an open-source repository, allowing for joint updates to the service. Think of Hugging Face as the GitHub of AI models. For those who may not be familiar with GitHub, you can think of it this way: various individuals or companies are creating AI, and a representative company, like Meta, is developing LLMs. They have made it available for everyone to use, allowing you to train your models or use them freely. It's a situation that companies like ChatGPT and Google find quite challenging. You can think of it as a place where these models are hosted. If you actually visit the Hugging Face website, you'll see the models available. Most of the models are open, although not all of them are. What you see here are all AI models, each designed for specific purposes.

It has been trained correctly, and it's quite popular recently. I'm going to cover this in a video. If you know about Comfy and the Fluxlight generation AI, there's talk about it surpassing Midjourney. So, this is also quite impressive. There’s Whisper, which I just mentioned, and I want to utilize such models. If you want to quickly use small language models for specific areas, there are things that can be done with models like Flood, ChatGPT, and others that cannot be done with larger models. Lastly, there’s something called API rate limits. For those who have tried using the GPT and Claude APIs, you might know that it shows something like xxx TM per minute. If you check the site, it mentions the wait limit. When you first sign up, you can only use $100 worth of API credits in a month. So, $100...

If you charge excessively, you can only use about $500 worth, and there is a set amount you can spend each month. We opened our service, and suddenly many users started to join. Fortunately, the number of users is skyrocketing, but if we only have this API, it could become a problem. Furthermore, if you look at GPT-4, it has RPM and TPM. To briefly explain, it means you can only use a certain amount of tokens within a specific time frame. So, no matter how much money I have, if I keep sending requests, I will receive a response saying, There are too many requests; please stop sending them. In such cases, we cannot use ChatGPT or flood the system; we need to run our own AI model. So, there are these advantages, but of course, there are also disadvantages. Not everyone will use this; who would use ChatGPT or Claude? So, the maximum...

One of the drawbacks is that if you want to run it locally, you need a very good specification. You saw the graphics card earlier, right? Of course, it's not necessary to have such high specifications to run a general model, but if you want to run a good model or do things quickly, you need a decent baseline specification. Another critical drawback is that if you're not using it personally but providing a service like SaaS to users, you can't run it on your local computer. In such cases, you have to put it in the cloud, so it can't be run locally. However, running it in the cloud incurs some costs, and the initial expenses for the API can be a bit high. This is because, in cloud services, you are charged by the hour or per second. Taking the earlier example of Llama 3, it would cost about $20 a month.

To run a computer that costs around 300,000 won smoothly, it is necessary. As you saw earlier, it can go up to several tens of millions of won. So, if we want to run it on the cloud, there is a minimum cost involved. If we integrate the API into our service and offer it to users, the API costs could exceed 200,000 to 300,000 won per month. At that point, you might need to consider switching to the cloud. Until then, this could actually be a disadvantage. However, as I mentioned earlier, the advantage is that you can use it without limits. Another downside is that you cannot run certain models like ChatGPT or similar flood models. Although there are restrictions in place for ethical reasons, there are models out there that have bypassed these restrictions. So, while you can run those, the official version of ChatGPT cannot be used.

The cloud model has the disadvantage that it cannot operate unless they release it. So, I think it would be good to compare these few disadvantages and some advantages, and use them according to the situation. Now, if we return to our LM Studio, you have probably finished downloading by now. On the left, you can search for the models with the search light and download them easily. It has been made very simple for you to download. The section called AI Chat is similar to the interface we generally use in the GPT cloud. You can think of it as that screen. At the top, it says to select a model, and the recently downloaded one is Llama 3 created by Meta. In my case, since I have a very low-end graphics card, a warning message popped up. If any of you have a poor graphics card like mine, you will see a run warning message as well, but just...

If you ignore it and just ask to load it anyway, it will load. Ah, I should either change the graphics card or something. So the model has been loaded. Now, when you ask a question, of course, I will answer it. But as I mentioned earlier, my graphics card is really bad, so it’s extremely slow. Can you see? It’s really slow, right? So if you have a graphics card that costs more than about 500,000 won, you should be able to receive answers at a speed that is generally above the level of using ChatGPT. Anyway, it seems that Llama 3 has been loaded now. Yes, so I directly prompted Llama 3. As you can see, you can also switch the roles of user and assistant. If you go to the advanced settings on the right, there is the context range, which determines how many tokens you will allow for prompting.

You can also configure the settings, and then you can decide how creatively you want to use this. You can set those parameters. If you look on the left, there's a playground, which is a multi-model session. However, as you can see, it's not just one model; it seems like it’s using the Llama 3 model for prompting and simultaneously prompting another model with the same context. So, it has those functionalities. This feature seems quite useful. Since we currently have the Llama model running, we can send requests to the Llama model via the API. 

So, today we briefly explored LM Studio. Personally, I tend to use Olar a bit more than LM Studio, but to install Olar, you need to be familiar with Docker and set it up through a terminal console on your computer.

I have a few things to cover, so today I introduced LM Studio, which can be easily installed. If you would like to try installing OLA, please leave a comment. If there are many requests, I will quickly create a video on how to use this interface with OLA. In our next video, I will show you how to use automation tools for scraping or crawling without writing a single line of code. Please look forward to that video as well. If you subscribe and turn on notifications, I will do my best to upload it as soon as possible. Thank you for today!

gpt, claude 유료로 쓰기 아까우시면 이거부터 무료로 써보세요~

Scripts