Route LLM: Efficient Language Model Routing for AI Applications

Route LLM allows intelligent routing of prompts to different language models, saving costs and increasing speed for AI applications.

### Route LLM: Optimizing Language Model Selection for Cost and Efficiency
* 💡  Route LLM is a framework that classifies prompts before sending them to language models, allowing for **efficient selection of the most appropriate model** for each case.
* 🔍  The framework is designed to determine whether a prompt can be answered by a **smaller, cheaper model** or if it requires a more **powerful, expensive model like GPT-4**.
* 📊  By using Route LLM, users can **save up to 80% of their language model budget** while maintaining **90% of the quality of GPT-3.0**.
* 🛠️  The setup involves creating a new Conda environment, installing the Route LLM package, and then writing a script that defines the **strong (GPT-4)** and **weak (Grok/LLAMA-38B-8192)** models.
* 🖥️  The framework can be run locally, with the weak model (LLAMA-3) serving as a **free, basic language model** for most use cases, and only resorting to **GPT-4 for more complex prompts**.
* 🌐  Route LLM offers benefits such as **decreased latency, significantly reduced cost, reduced platform risk, increased security, and improved privacy** by enabling **local model execution**.
* 🚀  The potential for further optimization is highlighted, suggesting the use of techniques like **mixture of agents** to enhance the classification model's **efficiency** in determining **model selection**.
* 🎥  The presenter hints at an upcoming video where they will share a business idea leveraging **Route LLM technology**, encouraging viewers to **explore and potentially develop the concept**.

I suspect Route LLM is much bigger than most people realize. I'm surprised how few people are talking about this. Route LLM is a framework where you can classify prompts before sending them to a large language model to determine which model is best to send that prompt to. And if you're asking yourself, Okay, great, why is that important? Well, here's the thing: you don't need to send every single prompt to the highest-cost Frontier Model, GPT-40. In fact, you can send most use cases to more efficient, smaller, and lower-cost models, saving you a ton and increasing speed as well. So, I'm going to show you how to set this up today. This is going to be a tutorial for how to use Route LLM, and if you're building any type of AI application, you need to know this. So, this is the GitHub repo for Route LLM. I'll drop it in the description below, but before I get into it, I want to reemphasize how powerful this really is and how this further expands on my strong belief that AI is

going to be run on edge devices, meaning they're going to be run locally, and Route LLM allows for that. Let's look at Apple intelligence. Apple rolled their own model, which can run locally on your iPhone, and any more sophisticated or complex prompts, they will outsource to Chat GPT, but only when absolutely necessary. And that is exactly what Route LLM can do for you. It will maintain 90% of the quality of GPT-3.0 while being 80% cheaper. I went over their research paper already, and now let me show you how it works. First, so I wrote up a little script, and we're going to be using a strong model and a weak model. And anytime we can use the weak model, we will use that first. And the weak model is much less expensive and much faster because we're using Grok. So let me show you how it works. I imported an OpenAI API key, a Grok API key. I set up everything. By the way, I'll go through all of this line by line a little bit.

later in the video, but I just want to show you it working really quickly. Then we allow the MF router, which is essentially a classification model, classifying the prompt to determine which LLM it should be sent to. Then we send the message using an OpenAI API-compatible API call. We get the response, it chooses which model to send it to, and then it gives us that response. So let's take a look. So here, the content or the prompt is simply hello, and we don't need a Frontier Model model for this. This is the most basic use case, and any large language model can answer it. So let's run it and let's see what happens. So there we go, message content: hello, it's nice to meet you. Is there something I can help you with, or would you like to chat? A perfect response, and it decided to use GRS llama 3 8B model. So extremely fast and extremely cheap, and we can run this locally too, not just the weak model but both models, and I'll show you how to do that as well. Now let's give it

harder prompt. So here, we're going to say write the Game snake in Python. Something that I would hope it routes to GPT 4. So let's hit play again, again, same exact code, but it's deciding which model to use, and the classifier model is running locally, so it essentially costs nothing. And you can see this time, it's taking a little bit longer, so it's likely going to GPT 4. And of course, the output is much longer, so it's going to take a little bit more time to complete. All right, there we go. So you can see model name GPT-4. So in new to use the frontier model for a more complicated prompt. And here's the snake game. I'm not going to test it, but the point is we had a more complicated prompt and it new to use GPT-4. So what does this actually give us? Well, essentially 90% of the use cases, we don't need that Frontier Model, so we're going to save a ton of time and a ton of

cost. So now that you've seen how it works, let me show you how to install it. And then I'm going to walk you through the script that I wrote. And the best part is you can drop this into any code base. So the first thing we're going to do is create a new conda environment. So I'm going to type to create DN route python equal 3.11 and then hit enter. Now I already have one named that so I'm going to replace it but you won't have to do that. Then hit enter once more. So once that environment is ready to go, we're going to copy this code right here conda activate route, paste it in and hit enter. And now we can see it is active because it says so right there. And the installation really could not be any easier. It's simply this code pip install in quotes route llm square bracket serve comma eval close square square bracket and then quotations again. So hit enter and it should install everything for you no problem. Okay, once that's done you are

pretty much done, so let's look at the code now. I am going to revoke any API keys that I show in the code before I publish this video. So, the first thing we're going to do is import OS, which will allow us to set the environment variables. So, the way you think about it is there's a strong model and a weak model. It's going to try to use the weak model, which typically is the cheaper model, whenever it can. And if it can't, then it's going to use the stronger model, which is also more expensive. So, here I've decided to use the OpenAI GPT-4 as the strong model and GPT-3 as the weak model. So, I set the environment variables right there. Then I import the libraries that we need. So, from rout lm.controller import controller. Now, you can see this is still underlined, and this has nothing to do with the actual rout llm code. So, what you want to do is to switch to another environment and then switch back, and now it is no longer underlined in yellow.

it should work. Next, we need to create the controller, and that is going to be essentially what looks like an OpenAI client. The controller will take a router, and we're going to use the MF router. Now, you can actually use different routers; you can tune your own router. There's a lot of complexities there that I'm not going to get into because the standard default router seems to work just fine. Then we're going to define our strong model, and we're going to say it is GPT-4. And this is the model name that the API knows. Then for the weak model, we're going to type GR/LAMA-38B-8192. Now, this Grok/Slash is required because that's the only way that LLM, which is the LLM service that is kind of behind the scenes, knows which base URL to send to. Now, before I move on, Route LLM out of the box does actually support a number of endpoints. Let me show you that really quickly. So right here, you can use local models, which I'll

show you in a moment. You can have anthropic Gemini Amazon Bedrock together Ai and any scale. And of course, you could pretty much use any llm endpoint as long as it's supported by light llm. And as you can see here on the left side, they support a lot. Now back to the code. We have all this set up, and now we're just going to create a completions just like we would with an open AI API. So we have response equals client.chat.completion.create. And then right here, rather than specifying like gbt 4 or llama 3, we're actually going to use the router name. And remember, this client is not an open AI client. This is actually a route llm client. So here we specify the model name for the router. And then for messages, we treat it exactly the same role user. And then content, you can put whatever your prompt is. So here I said, Write a snake game in Python. Then the rest of this is just taking the response JSON and outputting it like we've seen.

in the terminal, and that's it. With just a few lines of code, you could potentially save 80% of your llm budget. Now, if you're a large enterprise and you're spending a million dollars a month on API calls to various services, you probably also aren't intelligently routing to the appropriate llm depending on your prompt use case. And as I mentioned, it turns out most prompts can go to simple cheap models. So, with just a few lines of code, you can get this running and have a huge reduction in your API bill. And by the way, I'll create a j, and I'll drop that in the description below for all the scripts that you're seeing here. Now, what if you want to run this locally and you want to use gp4 as the strong model and a local model as the weak model? Here's how to do it. It's actually very simple. So, the first thing you need to do is install AMA, and it's really simple. Just go to. a install it on your machine. It really couldn't be

easier then open up your terminal and type olama run llama 3, or whichever model that you want to use. It'll download it, and it will have it running. Once you've done that, I switched back to VS code, and I removed the gro API key 'cause we no longer need it. And the only difference is for the weak model, always say is _ chat llama3. And that's it, that's literally it. You don't have to change anything else. It is so simple. Now let's run it. So I'm simply going to prompt, Hello, and let's see what happens. And this is using llama 3 locally. And there we go, message content, Hello, it's nice to meet you. Is there something I can help you with, or would you like to chat? And it is using ol llama llama 3. Now again, if I gave it something more complicated, it would use GPT 4. So now we have essentially a free large language model to handle all of the most basic use cases, which tend to be about 80 to 90%.

of all use cases and then it will use and outsource it to GPT-4 only whenever necessary. Let me go over the benefits one more time because I am so excited about this: decrease latency, significantly decreased cost. One thing that a lot of people don't talk about: reducing platform risk. Now you are no longer beholden to OpenAI. You can build a collection of models that work on your behalf and abstract that away from your actual business logic. Increase security, increased privacy because you can essentially run a lot of these queries locally so you don't need to send them off to a cloud provider. And there are other benefits as well. Now I'm just getting started digging into how powerful this really is and I'm going to give you a hint. A couple of days ago I posted something on Twitter where I said I had this incredible idea for an Enterprise business and this is a hint at that. And let me give you one more hint. And by the way, I'm going to create an entire video essentially giving away

this idea and showing you how I would pitch it to you, and hopefully one of you will build it because I really want to see this made. Imagine for the weak model, instead of just taking a single large language model and just accepting it, imagine if you started using techniques like mixture of agents, and all of a sudden you trained your classification model, the router, to know that the mixture of agents is actually much more powerful than previously thought. So you have this mixture of agents endpoint that can handle the vast majority of all of your use cases, and only when absolutely necessary do you actually outsource it to GPT-4 or any other frontier model. Super exciting! I cannot wait to continue to learn more about this technology, and make sure you subscribe because I am going to make a video all about this idea, and I really think there's something there, and I hope you build it. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

RouteLLM Tutorial - GPT4o Quality but 80% CHEAPER (More Important Than Anyone Realizes)

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

The Significance of Route LLM in AI Applications

Route LLM is a framework for classifying prompts and sending them to the most efficient language model, saving cost and increasing speed.

It allows for the use of smaller, lower-cost models for the majority of use cases, maintaining quality while being more affordable.

Route LLM enables running AI models locally, similar to Apple's approach, resulting in significant cost savings.

Creating a Python Snake Game and Optimizing Model Usage

The new approach optimizes model usage, saving time and cost for 90% of use cases.

The installation process involves creating a conda environment and using a simple pip command for the installation.

The Python script sets environment variables, imports necessary libraries, and demonstrates the strategy of using both GPT-3 and GPT-4 models for the game.

The controller uses the MF router with GPT-4 as the strong model and GR/LAMA-38B-8192 as the weak model.

Route LLM supports a variety of endpoints for the weak model, offering flexibility in model selection.

The controller functions similarly to an OpenAI client, using the router name and message content to generate a response from the route LLM client.

Reducing API Costs with Intelligent Prompt Routing

Intelligently routing prompts to the appropriate language model can potentially save up to 80% of the API bill.

By using a simple code, it is possible to run a free large language model to handle basic use cases, significantly reducing costs.

The process involves installing AMA, choosing the model to use, and making a simple adjustment for the weak model, requiring minimal changes.

The Power of Using a Mixture of Agents in AI Technology

Using a mixture of agents can decrease latency and costs, reduce platform risk, and increase security and privacy.

This approach allows for handling the majority of use cases without outsourcing to a cloud provider, reserving that for only the most necessary instances.

The speaker is excited about this technology and plans to create a video to share and pitch an idea for an Enterprise business based on this concept.