언어 모델을 위한 비용 효율적인 LLM 라우팅 프레임워크

오픈 소스 프레임워크인 RouteLLM은 쿼리를 더 저렴하거나 강력한 언어 모델로 효율적으로 라우팅하여 상당한 비용 절감과 높은 정확도를 달성합니다.

### RouteLLM: 비용 효율적인 LLM 라우팅을 위한 오픈 소스 프레임워크
* 🤖 LMSys는 쿼리 복잡도에 따라 더 저렴한 언어 모델과 더 강력한 언어 모델 중에서 동적으로 선택하는 비용 효율적인 **LLM 라우팅을 위한 오픈 솔루션 프레임워크인 RouteLLM**을 출시했습니다.
* 💰 이 프레임워크는 다양한 데이터 세트에서 **85% 이상의 비용 절감**을 달성하는 동시에 **GPT-4 성능의 95%**를 유지합니다.
* 🔀 RouteLLM은 **유사도 가중치, 행렬 인수분해, BERT 분류기, LLM 분류기** 등 네 가지 모델을 사용하여 어떤 LLM을 사용할지 예측합니다.
* 📈 GPT-4 증강 데이터 세트를 사용하는 행렬 인수분해 모델은 최상의 성능을 보여주며, **무작위 선택에 비해 26%만 GPT-4를 선택**하고 **50%의 비용 절감**을 달성했습니다.
* 🔄 프레임워크의 모델 선택은 학습 중에 사용된 **특정 LLM을 교체**해도 여전히 효과적이어서 일반화 가능성을 입증합니다.
* 🚀 RouteLLM의 오픈 소스 릴리스에는 **코드, 데이터 세트 및 사전 학습된 모델**이 포함되어 있어 프로덕션 애플리케이션에 쉽게 통합할 수 있습니다.
* 💡 이 프레임워크는 특히 대부분의 경우 저렴한 LLM을 사용하면서 **가끔 더 강력한 모델을 호출**하여 이점을 얻을 수 있는 애플리케이션에 유용합니다.
* 💬 이 프레임워크의 성능은 상용 서비스와 비슷하면서도 **비용 효율이 더 높기** 때문에 프로덕션에서 대규모 언어 모델을 사용하는 기업에게 강력한 옵션이 될 수 있습니다.

Okay. So recently, I've been talking a lot in videos about the whole idea of cheaper models and faster models, and being able to use them over some of the more expensive models a lot of the time. And the big reason for doing that is I'm just seeing so many people waste so much money with doing calls to GPT-4 for everything, or calls to Claude Opus and stuff like that for lots of things where they don't actually need that kind of model. They would be much better to go for a faster model and a cheaper model. The one challenge with that has always been, well, what if some of your calls to the LLM actually do need the big model or do need the more powerful model? So what I want to talk about in today's video is something that's been released by LMSys. So this is the people who run Chatbot Arena, who were some of the first

people to create models like Vicuna. And for the past year+, they've been
benchmarking all the different models, giving them an ELO score,
that kind of thing. So one of the cool things that they just
released earlier this week was this new open source framework called RouteLLM, and this is basically an open source
framework for cost-effective LLM routing. So the idea here is that sometimes you're
going to be using your cheaper model, whether that's a Llama-3 8B, whether
that's a Gemini flash, whether it's a Claude Haiku, something like that. And then other times you're going
to want to use your really powerful model, like the GPT 4, the Claude Opus,
like Gemini ultra, that kind of thing. But the challenge is how do you decide
when to use one versus the other? So most of the time up until now, what
I've generally done for projects that I've

been working on is that certain calls I
make to the cheaper models and certain calls I make to the more expensive models. What this framework actually proposers
is that you've got this router in the middle that looks at the call, looks
at the prompt that's coming in and decides, Hey, is this best suited to
us really strong model like GPT 4? Or can we just answer this with
a weaker model, like a Gemini flash or a Llama-3, et cetera. And by doing this, they're able to then
get this router to decide which model gets used based on the query that's coming in. Overall, this saves a lot of money. So if we look into their blog post a
little bit, we can see that what they're saying here is that this is making
cost savings of over 85% on various data sets that they've tried out there. that it's still able to get a very high
level of accuracy on those benchmarks.

But it's able to do it by deciding
that actually, this particular question, we don't need the top model. We can go for a cheaper model. And so therefore, they're able to make cost
reductions of so much on each dataset. It also shows us the things like
the GSM 8K dataset. It's probably actually a lot harder, obviously,
than certain other things. So the cost savings are actually lower on
that because it's probably falling back to GPT-4 for a lot of those as compared
to some other datasets out there. And you can see, even with these cost
savings, they're still able to achieve 95% of GPT-4's performance in here. So, this is a really nice idea
that they've put out as a paper. But far more than that, they've
released the open-source framework. They've released the code. They've released the datasets
that they use to do this. And they've released the models
that you can actually take and

start putting this in production. So, if you do have an app that's
in production, and you're looking at ways to save money on this. And I know that quite a few of the
apps that are out there in production now, a sort of often borderline
between being profitable versus not profitable and stuff like that, and a lot of it comes down to
what models you're actually using. So this kind of thing allows you to have
the benefit of when 80% of your queries really could be done by a cheap model, and you only need the really strong
model for 20% of the queries. It gives you massive cost savings in doing this. So let's look at how
they actually do this. So, what they did was they've got a bunch
of datasets already of human preference data, of knowing that when for this
particular question, someone typed in model A, said this, model B said this, and people picked model A over model B
80% of the time or something like that.

From that, they're able to build
a number of different models to try and predict that on things
that they haven't seen before. So the obvious sort of way to do this
is you would take an embedding, right? And that's what they do. They actually, I think, use the OpenAI,
small embeddings in doing this. But then once they have got that
data, they look at different ways that they can use that and train that. So one of the first ways that they do
this is a similarity weighted things. So this is basically where they're
working out a weighted ELO calculation based on similarity of the embeddings. And it's a little more complicated
than just doing a cosine score. They have to work out the similarity. And then also that related to
the different models in there. The second one that they do is
a matrix factorization model. So this is cool. So where they've got, imagine you've
got a really big matrix of where you've

got some data that you know, that,
okay, this model is better than this model for this particular phrase, but
for a lot of it, you're missing data. So what they do is they come up
with two matrices that they can then multiply to try and approximate
the data that they've got. And at the same time, it fills in all
the empty spots that they've got as well. So by using that, they're then able
to predict that, okay, for this new kind of thing that we haven't
seen before, this is the model that most likely is going to work. And it turns out that this one is
probably the best model, which I'll talk about, as when we come back. The third one is where they're
using a BERT model and training up a classifier for it. And the fourth one is kind of similar,
but they using an LLM classifier for it. if you look at the results in here,
what we're looking for, is this kind

of efficient frontier of working out,
what model you should go for, based on this efficient frontier and here. When they train this up on just
the dataset pairs that they have themselves, they get out a model,
but then what they also worked out they could do was augment this data. So basically they could use GPT-4 to be
like a human and judging as well in there. And that turns out to work out quite
nicely, that when you then train up your models on this augmented data as well,
almost, I think all of them improve, and this matrix factorization
seems to do really well. This is where they're using the
model to predict, and it ends up using GPT-4 26% of the time, and I guess the other model that the
rest of the time. And that turned out to be half as cheap as just going
for a random baseline for doing this. Another thing that they found that was
really interesting here was that while

they did their training on Mixtral
versus GPT-4 models. The actual models that they end up making make
this prediction of which LLM to use work really well, even when
you swap out the models. So when they change the Mixtral
for Llama-3 8B, and the GPT-4 for Claude Opus, you can see
that the model still works out which model to actually use in here. So again, you see these cost savings
and stuff like that from this. Finally, just to finish up, there are
some commercial services out that have been offering things like this for people
to be able to run their calls through this, and it would decide what to use. And they showed that basically, you
know what they're releasing as open source here actually is doing just
as good as the commercial ones. And then obviously also a lot
cheaper and stuff as well in here.

So just to finish up, I would say, if you're doing a project where you get to put an LLM into production, and you realize that sometimes you need a really strong model, and a lot of the times you're perhaps not going to need a strong model. This is definitely worth considering and worth checking out here. This RouteLLM from LMSys. Not only have they released the code, so you could run the models that they've got and use the routers that they've made there, they've also got everything that you need to basically make your own routers. And we can see that they've released all the models as well as the datasets on Hugging Face here. So this allows you to basically look at these yourself and try them out yourself. I mentioned the open-source community will be able to build on this and come up with things a way or you're comparing all the different popular models that are out there.

So in some ways, perhaps this is not a super sexy topic, but If you do anything where you're spending quite a bit of money on tokens, this is definitely something that you should be checking out for using in production and stuff. All right, as always, please put any comments and questions in the comments below. If you found the video useful, please click like and subscribe. And I will talk to you in the next video. Bye for now.

What is an LLM Router?

HI my name is Sam Witteveen, I have worked with Deep Learning for 9 years and with Transformers and LLM for 5+ years.  I was appointed a Google Developer Expert for Machine Learning in 2017 and I currently work on LLMs and and since earlier in 2023 on Autonomous Agents. 

Sam Witteveen

AI 모델 사용을 위한 비용 효율적인 프레임워크, RouteLLM 출시

LMSys는 최근 사용자가 더 저렴하거나 더 강력한 AI 모델을 사용할 시기를 결정하는 데 도움을 주기 위해 비용 효율적인 LLM 라우팅을 위한 오픈 소스 프레임워크인 RouteLLM을 출시했습니다.

이 프레임워크는 라우터를 사용하여 수신 프롬프트를 분석하고 GPT-4와 같은 강력한 모델을 사용할지 아니면 약한 모델을 사용할지 결정하여 높은 정확도를 유지하면서 다양한 데이터 세트에서 85% 이상의 비용을 절감할 수 있습니다.

GPT-4 성능을 위한 비용 효율적인 모델 선택

새로운 비용 효율적인 모델 선택 접근 방식을 통해 각 데이터 세트의 비용을 크게 절감하면서도 GPT-4 성능의 95%를 달성할 수 있습니다.

이 접근 방식에는 오픈 소스 프레임워크, 코드, 데이터 세트 및 모델 릴리스가 포함되어 있어 프로덕션 환경에서 사용할 수 있으며 앱에 막대한 비용 절감 효과를 제공합니다.

이 방법은 쿼리의 80%에는 저렴한 모델을 사용하고 나머지 20%에는 강력한 모델을 사용하여 최적화를 위해 사람의 선호도 데이터를 활용하는 것을 기반으로 합니다.

임베딩을 사용한 다양한 예측 모델

연구자들은 임베딩의 결과를 예측하기 위해 유사도 가중치, 행렬 인수분해, BERT 및 LLM 분류기 등 다양한 모델을 사용합니다.

유사도 가중치 모델은 임베딩의 유사도에 따라 복잡한 계산을 수행하며, 행렬 인수분해 모델은 두 행렬을 곱하여 누락된 데이터를 채웁니다.

이 연구에서는 BERT 및 LLM 분류기를 사용하여 예측을 위한 추가 기술을 제공합니다.

데이터 증강이 모델 학습에 미치는 영향

데이터 증강에 GPT-4를 사용하면 행렬 인수분해가 크게 개선되어 모델의 성능이 향상됩니다.

모델 예측에 GPT-4를 사용하면 기본 방법보다 절반의 비용으로 비용 효율적입니다.

다양한 LLM에 적응할 수 있는 이 모델의 기능은 비용 절감과 효율성을 입증하며 상용 서비스보다 뛰어난 성능을 발휘합니다.

LMSys의 RouteLLM은 사전 학습된 모델과 맞춤형 라우터를 생성할 수 있는 기능을 제공하여 프로덕션 요구 사항에 유연하게 대응할 수 있습니다.

허깅 페이스에 모델과 데이터 세트가 출시되어 쉽게 액세스하고 테스트할 수 있습니다.

이 리소스는 토큰에 상당한 투자를 하는 모든 사람에게 유용하며 프로덕션용으로 살펴볼 가치가 있습니다.