추상 AI: 효율성을 위한 대규모 언어 모델 최적화

Abstract AI는 대규모 언어 모델을 위한 드롭인 API로, 낮은 비용과 지연 시간, 유연성을 제공하여 AI 개발자의 응답을 최적화합니다.

### 추상 AI: 대규모 언어 모델 사용 최적화
* 🤖 문제는 AI 개발자들이 **최적화되지 않은 채로 대규모 언어 모델(LLM)에 과도한 비용**을 지불하고 알고리즘의 발전을 활용하지 않는다는 점입니다.
* 💡 추상적인 AI 솔루션: 비용, 속도, 품질을 위해 **LLM 사용을 최적화하는 단일 API**로, 각 프롬프트에 가장 적합한 모델을 선택하기 위해 **Route LLM**을 사용합니다.
* 💰 비용 절감: 추상 AI는 특정 사용 사례에 대한 로컬 모델을 포함하여 가장 비용 효율적인 모델로 프롬프트를 라우팅하여 **LLM 비용을 20~80%까지 절감**할 수 있습니다.
* 🔒 보안: 공통 프롬프트에 **로컬 모델 배포**를 허용함으로써 추상 AI는 엔터프라이즈 AI 애플리케이션의 보안 및 개인 정보 보호를 강화할 수 있습니다.
* 🏗 ️ 이 시스템은 **모델 선택을 위한 라우팅 LLM, 품질 개선을 위한 알고리즘 잠금 해제, 속도와 일관성을 위한 캐싱**을 사용합니다.
* 📊 벤치마킹: 추상 AI에는 다양한 LLM에서 일관된 품질과 성능을 보장하는 **벤치마킹 도구**가 내장되어 있습니다.
* 🚀 미래 비전: 발표자는 **더 광범위한 제품 로드맵의 일환**으로 신속한 관리, 사용자 권한 부여 및 버전 관리로 확장할 수 있는 가능성을 제시합니다.

so I have this idea for a business, and the thing that I'm lacking the most is time. So what I decided to do was put together a presentation showing you exactly what this idea is, why I think it's valuable, and then hopefully one of you builds it. And I actually think this could be a startup in itself, so hopefully you like it. Let's get into it. So I'm calling it Abstract AI. Obviously, call it whatever you want, but I think it's kind of a cool name because I think of it as an abstraction layer on top of large language models. So first, what is the problem? The problem is AI developers are under-optimized. And who am I actually talking about? Well, I'm talking about two groups of people mainly. One is AI product companies, so companies that are building and selling products or services that are powered by AI or existing products and services that are being enhanced by AI. And then the second group are large organizations that are looking to implement artificial intelligence internally at their company. Both of these groups of people consume a

lot of large language model tokens through API calls, but for the most part, they're taking a pretty naive approach to it. They're simply going to usually open AI's endpoint and choosing GPT 40, but there's a ton of room for optimization within that. That's really what this is all about. So the problems that they're facing are they're overpaying for Frontier llms. Typically, AI developers are not rolling their own models, and a lot of the time, they're not even hosting Frontier open-source models. They're usually just paying open AI. That is why open AI has the vast majority of Revenue right now in the AI World. Also, AI developers, because they're only building on one cloud service and they really only have access to one set of models. So let's say GPT models within the open AI ecosystem. They have a ton of platform risk if open aai decides one day to change their policy to change their model to charge more. They are exposed to all of these potential risks. Also, when you're going to a Frontier

Model, and that's the only model you're using. That tends to be the slowest and highest latency model out there. And here's the thing, for 90 to 95% of prompts and use cases, the Frontier Model is not necessary and is overkill. It's the most expensive and the slowest. They're typically not using a lot of algorithmic techniques that we've been seeing over the last months. So, Route LLM, Chain of Thought, Mixture of Agents, all of these algorithmic unlocks are just simply not being used. And all of this leads to really inefficient LLM usage. And so, that is what I believe Abstract AI can solve. So, Abstract AI will lower latency, lower cost, and offer more flexibility. And we're at this point where LLMs are essentially commodities. There are a ton of amazing offerings. Now we have a completely open-source, open-weight Frontier Model from Meta Llama 3.1 405b. And that is really just leveling the playing field for large language models. And in that sense, maybe

cost is not the priority, but when you're talking about potentially using billions of tokens per month, when you get into agentic systems, and you're deploying AI to companies with tens of thousands of people, cost actually probably still does matter. And if you're able to reduce cost by 20, 30, 80% in some cases, that's actually really meaningful. But here's the thing: developers generally don't really care which large language model is being used if they can get the same quality, the same consistency, and the same speed from some noname model compared to ChatGPT, they probably don't care. What they care about is consistency of response, quality of response, and at scale cost. So as I mentioned, most developers are using a single endpoint and a single large language model. They're not using any of the algorithmic unlocks that have been coming out, which will only add quality. Even if you do nothing else, if you simply use some of these techniques Chain of Thought mixture of Agents, you literally just layer that on top of whatever large

language model you're already using. You will get much greater quality, but these things take a lot of work to implement. They're not that simple, and the techniques are constantly evolving. There's new techniques coming out all the time, and so being up to date on the latest algorithmic unlocks is a lot of work for a developer and something that is not really their core expertise. So what is abstract AI in the most simple terms? Abstract AI is a single drop-in API replacement for whatever large language model you're using today. It connects to multiple large language models, everything from closed-source frontier to open-source to even local models, and it is the cheapest, fastest, highest quality responses on a per-prompt basis. So it's able to use at the very front of the stack something like Route LLM to determine on a prompt-by-prompt basis which large language model should be used, which algorithm should be used by determining which will be fastest, cheapest, and highest quality. And as we've seen from the Route LLM research paper, you can achieve 90% of GPT-40 quality for 80% reduction.

in the cost simply by routing the prompt to the right large language model. And again, I want to stress this includes locals. So, if you're deploying AI within your business and you have 10,000 employees, you can install a very small model on every single one of the computers. And with Route LLM wrapped around it, it will answer essentially for free 90% of the prompts locally. And that increases safety and security, which we'll get to in a moment. And then for anything else, for anything more sophisticated, for coding use cases, for anything that is more complex, that is when it outsources to a larger model, to a private cloud model, or even a closed-source cloud provider like OpenAI. And if this sounds familiar, it is essentially what Apple Intelligence is doing. Even though they're completely closed-source, Apple Intelligence is installing a small model on the phone that can work on your behalf. And then for any world knowledge or any more complex use cases, it outsources either to their own private cloud but a massive

model or eventually with permission to open AI. So, how does this all work? One, it uses Route LLM as it says right here to pick the right large language model for each prompt. So, Route LLM out of the box comes with a really nice machine learning algorithm that can determine which model a prompt should go to. But that model, that ML model, can actually be trained for your specific use case and for your plethora of different models that you want to use, and even include algorithmic techniques to determine what quality is going to come out of each of those models. Then you layer on some of these algorithmic unlocks that we've talked about, either on the open-source smaller models or even on top of the larger Frontier models, although it's kind of less important at that level. You optimize on the small models using an agentic framework. So, Crew AI, for example, mixture of Agents Chain of Thought, and you can really just tell it what you want to get out of it and why. And so, if speed is what you're optimizing for, you

can actually tell the API about that, and then it will change its routing to optimize for speed. If the highest quality is what you're looking for based on your prompt, it will do that. So you can kind of manually tweak it as you're going as well. Then, as mentioned, you outsource to a frontier model only when required, kind of like what Apple intelligence does. So then, what, as I mentioned at the beginning, consistency and quality are kind of the two most important factors when an AI developer is deciding which large language model to build on top of. So, Abstract AI will have built-in benchmarking to optimize the response consistency, and the benchmarking can be custom. So, for a specific use case within a company or within a product or service, you can actually have benchmarks that you're running against for any new large language model that you want to include or any algorithm that you want to put on top of that. Then, it should also have eventually built-in caching. So, one tried and true way to basically guarantee consistency, improve quality, decrease cost, and latency is by using caching.

autogen does this really well. So if you submit the same prompt again and again, there's no reason why you actually need to hit a large language model again and again. You are expecting the same response, so simply return the cached version. This will optimize speed, cost, and consistency. So then, what is the broader vision of this project? I think there is a bunch of different areas for expansion, but one that makes a lot of sense to me is prompt management. Now, because Abstract AI is being inserted into such a critical path for any company, for any AI Builder, you all of a sudden have this really great expansion path as well. And one direction is upstream towards the user, and you can start doing prompt management, whether that's User permissioning, Group permissioning, Company rules, and then also versioning. So you can have different versions of prompts, and again, all of this tied into the benchmarks which we already talked about. And then from there, again, because this product is being inserted into the most critical area of an AI developer's workflow, you

can really expand in a bunch of different ways. So a bit rough, but hopefully you enjoyed this. I really think there is a great product, if not an actual business, to be built from this. And I don't have time, but if you decide to build it, reach out to me. I'd love to talk to you about it, and I'd love to help you in any way I can. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

This Could Be a MASSIVE AI Business...and Now It's Yours 🤑

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

"추상 AI" 스타트업을 위한 제안

이 아이디어는 대규모 언어 모델 위에 추상화 계층을 생성하여 AI 개발자의 토큰 사용을 최적화하는 것입니다.

이 스타트업은 AI 제품 회사와 AI를 사용하는 대기업이 대규모 언어 모델 토큰 사용을 최적화하여 프론티어 LLM에 대한 과다 지불 문제와 단일 클라우드 서비스 제공자에 대한 플랫폼 종속성 위험을 해결하도록 돕는 것을 목표로 합니다.

목표는 AI 언어 모델을 활용하기 위한 보다 효율적이고 비용 효율적인 솔루션을 제공하는 것입니다.

LLM에서 프론티어 모델 사용의 비효율성

프론티어 모델은 프롬프트와 사용 사례의 90~95%에 불필요하고 너무 느려서 비효율적인 LLM 사용으로 이어지는 경우가 많습니다.

추상 AI는 지연 시간을 줄이고 비용을 낮추며 더 많은 유연성을 제공함으로써 이 문제를 해결하는 것을 목표로 하며, 이는 대규모로 AI를 배포하는 개발자와 기업에게 매우 중요합니다.

비용이 많이 들고 느린 프론티어 모델을 선호하기 때문에 Chain of Thought와 에이전트 혼합과 같은 알고리즘 기술의 사용이 간과되고 있지만, 개발자는 일관성, 품질, 비용 효율성을 LLM의 우선 순위로 삼습니다.

추상 AI는 로컬 모델을 포함한 다양한 소스를 활용하여 대규모 언어 모델을 비용 효율적이고 고품질로 대체하는 역할을 합니다.

프롬프트를 가장 효율적인 모델로 라우팅함으로써 품질을 유지하면서 비용을 크게 절감할 수 있습니다.

이 접근 방식은 Apple Intelligence와 마찬가지로 기본 프롬프트에 로컬 모델의 우선 순위를 지정하여 안전과 보안을 강화합니다.

대규모 언어 모델을 위한 라우트 LLM의 기능

Route LLM은 머신 러닝 알고리즘을 사용하여 각 프롬프트에 적합한 대규모 언어 모델을 선택하며, 특정 사용 사례에 맞게 추가로 학습할 수 있습니다.

이 시스템은 사용자의 필요에 따라 속도나 품질을 최적화할 수 있으며, 필요한 경우 프런티어 모델에 아웃소싱할 수 있습니다.

벤치마킹 및 캐싱은 응답 일관성, 품질 및 효율성을 보장하는 기본 제공 기능이며, 특정 사용 사례에 대한 사용자 지정 벤치마크도 가능합니다.

추상 AI는 사용자 권한 부여, 그룹 권한 부여, 회사 규칙 및 버전 관리를 포함한 신속한 관리로 확장할 수 있는 기회를 갖게 되었습니다.

이러한 확장은 기존 벤치마크와 밀접하게 연관되어 있어 프로젝트에 더 넓은 비전을 제시합니다.

이 제품은 AI 개발자의 워크플로우의 핵심 영역에 통합됨에 따라 크게 성장할 잠재력을 가지고 있습니다.