Route LLM: Cost-Effective Large Language Model Routing Solution

Route llm by lm.org reduces llm cost by 80% while maintaining 95% of GPT-4 quality, optimizing for efficiency, cost, and privacy.

### Route LLM: Cost-Effective Large Language Model Routing Framework
* 📉 " Route LLM" by lm.org is an open-source framework that **reduces the cost of running large language models (LLMs) by 80%** while maintaining **95% of GPT-4 quality**.
* 🔀  LLM routing aims to **minimize costs** by routing queries to **weaker, cheaper models** whenever possible, and **stronger models only when necessary**.
* 🧠  The "Route LLM" framework is based on **preference data** and introduces **augmentation techniques** to improve router performance.
* 💻  Four different routers trained using public data from Chapo Arena demonstrated significant cost reductions: **over 85% on Mt. Bench**, **45% on MLU**, and **35% on GSM-AK** compared to using GPT-4 alone, while maintaining **95% of its performance**.
* 📊  The experiment setup involved randomly routing between **GPT-4 and MixL 8x7B** on Mt. Bench, and using preference data points consisting of prompts and model comparison results (**win, lose, or tie**).
* 🔄  The framework's generalizability was demonstrated by achieving **similar results when routing between different model pairs** (GPT-4 and MixL 8x7B, CLA-3 Opus and Llama-38B) **without retraining the routers**.
* 💰  The cost-effective routing enabled by "Route LLM" is expected to lead to **more efficient and widespread usage of LLMs**, with applications running on **local edge devices** and **higher overall quality**.

we have a very exciting project today out of lm.org that was just released that promises to greatly reduce the cost of running large language models. They were able to reduce llm cost by 80% while still maintaining 95% of GPT for Quality. Pretty insane. And the way they do it is fascinating. So I'm going to break it all down for you right now. So we have this new project called route llm by lm.org, and they describe it as an open-source framework for cost-effective llm routing. This is something that I've been talking about for a while now, whether it's in the form of Agents with an orchestration layer on top or even a mixture of Agents, which is another incredible algorithmic implementation to get really high-quality results but using smaller open-source models. So here's the blog post on it, and the first thing I actually just want to point out before reading some of the interesting bits in here is this diagram. What this is showing is on the x-axis we see cost on

the Y AIS we see model performance now over here llama 38b obviously a very inexpensive model to run but the performance is not nearly as good as GPT 40. We have CLA 3 Opus over here very expensive. We have all these other models in the middle now check this out this is router this is Route llm very performance better than CLA 3 Opus almost as good as GPT 4 but a fraction of the cost cost and that is what is special about route llm and they actually provided the code for it. So now I'm starting to put together this picture of what I believe the perfect large language model stack is and it definitely includes route llm. It's some combination of mixture of experts route llm which I'll get into in a minute open source models running locally agentic systems and of course Frontier models and we need something an orchestration layer to put it all together and what that's going to do for us is optimized for Quality efficiency cost privacy.

security everything. We're going to get the best of everything, and that is also going to push the majority of compute to local devices. Edge devices, your phone, your computer, and that is why I'm so excited. And only when necessary will we ask a Cutting Edge Frontier Model like GPT 40 in the cloud, a massive model we can't run locally, for a response only when necessary. And as models continue to get better and smaller, more prompts, more questions will be able to be handled completely locally, which is essentially free, not including the cost of energy, of course. So, llms have demonstrated remarkable capabilities across a range of tasks, but there exists wide variation in their cost and capabilities, as seen from the plot of performance against cost in figure one. More capable models tend to be more expensive than less capable models. Yes, this leads to a dilemma when deploying llms in the real world. Routing all queries to the largest, most capable model leads to the highest quality responses, but can be expensive, and you don't really need to

find that local models any of my local models handles 90 to 95% of my queries just fine. It's only when I need that last 5 to 10% that I really need to go to Claud or chat GPT. So, llm routing offers a solution to this where each query is first processed by A system that decides which llm to Route it to. Ideally, all queries that can be handled by weaker models should be routed to these models, and the best part is those models can be run locally with all other queries routed to Stronger models, minimizing cost while maintaining response quality. However, this turns out to be a challenging problem because the routing system has to infer both the characteristics of an incoming query and different models' capabilities when routing. And to tackle this, we present route llm, a principal framework for llm routing based on preference data. We formalize the problem of llm routing and explore augmentation techniques to improve router performance. We trained four different routers using public data from chap AO to demonstrate that they

can significantly reduce costs without compromising quality. Cost reductions of over 85% on the Mt bench, Benchmark 45% on mlu, and 35% on gsm AK, as compared to only using GPT 4, while still achieving 95% of its performance. So cool. So in the routing setup, they assume, or they at least focus on the case where there are only two models: a stronger model, more expensive, and a weaker and cheaper model. Thanks to the sponsor of this part of the video, L L Trace. Lang Trace is an open-source and open Telemetry-based observability and evaluation platform for your large language model applications. It helps developers collect and analyze traces, create datasets, and run evaluations to understand application performance easily. So if you're building LLM-powered applications, you need to use Lang Trace. It works well with other observability tools such as Graphon, DataDog, Honeycomb, Signos, and many more. It offers end-to-end observability, comprehensive evaluations, unique prompt management, and a model playground. It's secure, flexible, is SOC 2 compliant, and as I mentioned, the best.

part. It is open source, so it has the transparency that you expect from open source projects. So go from shiny demos to Reliable AI applications easily and quickly using Lang Trace. Visit their website to learn more and join the community of innovators. I'm going to drop all the links in the description below. Check out Lang Trace and thank you again to them. Now back to the video. Here are some of the results. So for the evaluation, they use GPT-4 for the strong model and MixL 8x7B for our weak model. And we use the random router from before as our baseline. They tried a few different techniques and here it says augmenting the arena data using an LLM judge leads to significant improvements across all routers. So these are different routing techniques. So the blue line that you're seeing here is the baseline random, and these are other potential routing techniques. And what we see is once we augment it with an LLM judge, these routing techniques all tend to do better than just random routing.

let's see how they set up this experiment. So, this graph represents the performance of a router that randomly routes between the two models on Mt. Bench. Specifically, we route between GPT-4 and Mixola-7B here, with their performance denoted by red and gray dotted lines, respectively. So, here's Mix's performance, here's GPT-4's performance. For any router, we can plot a similar graph of its performance against the number of made to GPT-4. So, the more calls made to GPT-4, the better it did overall. Then, we use preference data for training our routers, a novel approach as compared to previous work. Each data point consists of a prompt and a comparison between the response quality of two models on that prompt. This could be a win for the first model, a win for the second model, or a tie. Using preference data allows us to learn about the strengths and weaknesses of different models and how they relate to queries, which is effective for training routers. So, we trained four routers using a mix of Chapout Arena data and data.

augmentation. They had similarity-weighted ranking router, they had a matrix factorization model, they had a BERT classifier, and a causal LM classifier. So, let's talk about generalization while we route between GPT-4 and Mixol. For the above evaluations to demonstrate the generalizability of our framework, we also present empty bench results when routing between a different model pair, CLA-3 Opus and Llama-38B. More importantly, we use the same routers without any retraining and what we can see here is the same results when using their technique. They are getting much better results from their routing method. All right, so why am I excited about this? So, a few reasons. One, obviously, if we can reduce the cost of using LLMs, that is going to be good just across the board. It's going to use less energy, it's going to allow more people to leverage AI, it's going to allow more applications to leverage AI in more ways. And here's the thing, what we've also found is techniques like mixture of

agents and Chain of Thought and other algorithmic unlocks for large language models use a lot more tokens. And if the tokens are much cheaper, we can use those algorithmic unlocks more often. So all of this is leading towards cheaper overall usage of AI, more efficient usage of AI, more running on local Edge devices, and higher quality overall. So they also released a full paper about this with folks from UC Berkeley and Canva. Interestingly enough, they also released the full open-source code base for it, which I always love when papers also include a code base. So if you want to check this out, I'll drop all the links in the description below. If you want to see me do a full tutorial on how to get this set up and running on your machine, let me know in the comments. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

RouteLLM achieves 90% GPT4o Quality AND 80% CHEAPER

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

The Route LLM Project for Cost-Effective Language Model Routing

org, offers an open-source framework that reduces LLM costs by 80% while maintaining 95% of GPT's quality.

The project introduces a routing model that performs better than expensive models like CLA 3 Opus and almost as well as GPT 4, but at a much lower cost.

This approach is seen as part of the perfect large language model stack, combining a mixture of experts, open-source models, agentic systems, and frontier models for optimized quality, efficiency, cost, and privacy.

The Future of Compute and the Role of Local Models

The majority of computation is shifting to local devices, with larger models being used in the cloud only when necessary.

Local Language Models (LLMs) have varying costs and capabilities, leading to a dilemma in deployment.

LLM routing offers a solution by processing queries with weaker models locally and routing others to stronger models, minimizing costs while maintaining response quality.

Cost Reduction and Performance Improvement in Model Routing

Lang Trace sponsors a video discussing cost reduction and performance improvement in model routing setup.

The video highlights the significant cost reductions of using the Mt bench, Benchmark 45, and gsm AK models compared to GPT-4, while still achieving 95% of its performance.

The use of Lang Trace's observability platform and different routing techniques, including augmenting the arena data with an LLM judge, is shown to bring about significant improvements in performance.

- The experiment involved routing between GPT-4 and Mixola-7B, with the performance of each model represented by dotted lines on a graph.

- Four routers were trained using a novel approach of preference data, including a similarity-weighted ranking router, a matrix factorization model, a BERT classifier, and a causal LM classifier.

- The generalizability of the framework was demonstrated by presenting results from routing between a different model pair, CLA-3 Opus and Llama-38B, showing consistent and improved results without retraining the routers.

The Impact of Token Efficiency on AI Usage

Large language models can benefit from more affordable tokens, leading to increased usage and efficiency.

The collaboration between various organizations, including UC Berkeley and Canva, has resulted in a full paper and open-source code release for this advancement.

The development also paves the way for running AI on local Edge devices, ultimately improving overall quality.