Abstract AI: Optimizing Large Language Models for Efficiency

Abstract AI is a drop-in API for large language models, offering lower cost, latency, and flexibility, optimizing responses for AI developers.

### Abstract AI: Optimizing Large Language Model Usage
* 🤖  The problem: AI developers are **under-optimized, overpaying for large language models (LLMs)** and not utilizing **algorithmic advancements**.
* 💡  Abstract AI solution: A single API that optimizes LLM usage for **cost, speed, and quality**, using Route LLM to **select the best model for each prompt**.
* 💰  Cost savings: Abstract AI can **reduce LLM costs by 20-80%** by routing prompts to the most cost-effective model, including **local models for certain use cases**.
* 🔒  Security: By allowing **local model deployment** for common prompts, Abstract AI can **enhance security and privacy** for enterprise AI applications.
* 🏗️  The system uses **Route LLM** for model selection, algorithmic unlocks for quality improvement, and **caching** for speed and consistency.
* 📊  Benchmarking: Abstract AI includes built-in benchmarking tools to ensure **consistent quality and performance** across different LLMs.
* 🚀  Future vision: The presenter suggests potential expansions into **prompt management, user permissioning, and versioning** as part of a **broader product roadmap**.

so I have this idea for a business, and the thing that I'm lacking the most is time. So what I decided to do was put together a presentation showing you exactly what this idea is, why I think it's valuable, and then hopefully one of you builds it. And I actually think this could be a startup in itself, so hopefully you like it. Let's get into it. So I'm calling it Abstract AI. Obviously, call it whatever you want, but I think it's kind of a cool name because I think of it as an abstraction layer on top of large language models. So first, what is the problem? The problem is AI developers are under-optimized. And who am I actually talking about? Well, I'm talking about two groups of people mainly. One is AI product companies, so companies that are building and selling products or services that are powered by AI or existing products and services that are being enhanced by AI. And then the second group are large organizations that are looking to implement artificial intelligence internally at their company. Both of these groups of people consume a

lot of large language model tokens through API calls, but for the most part, they're taking a pretty naive approach to it. They're simply going to usually open AI's endpoint and choosing GPT 40, but there's a ton of room for optimization within that. That's really what this is all about. So the problems that they're facing are they're overpaying for Frontier llms. Typically, AI developers are not rolling their own models, and a lot of the time, they're not even hosting Frontier open-source models. They're usually just paying open AI. That is why open AI has the vast majority of Revenue right now in the AI World. Also, AI developers, because they're only building on one cloud service and they really only have access to one set of models. So let's say GPT models within the open AI ecosystem. They have a ton of platform risk if open aai decides one day to change their policy to change their model to charge more. They are exposed to all of these potential risks. Also, when you're going to a Frontier

Model, and that's the only model you're using. That tends to be the slowest and highest latency model out there. And here's the thing, for 90 to 95% of prompts and use cases, the Frontier Model is not necessary and is overkill. It's the most expensive and the slowest. They're typically not using a lot of algorithmic techniques that we've been seeing over the last months. So, Route LLM, Chain of Thought, Mixture of Agents, all of these algorithmic unlocks are just simply not being used. And all of this leads to really inefficient LLM usage. And so, that is what I believe Abstract AI can solve. So, Abstract AI will lower latency, lower cost, and offer more flexibility. And we're at this point where LLMs are essentially commodities. There are a ton of amazing offerings. Now we have a completely open-source, open-weight Frontier Model from Meta Llama 3.1 405b. And that is really just leveling the playing field for large language models. And in that sense, maybe

cost is not the priority, but when you're talking about potentially using billions of tokens per month, when you get into agentic systems, and you're deploying AI to companies with tens of thousands of people, cost actually probably still does matter. And if you're able to reduce cost by 20, 30, 80% in some cases, that's actually really meaningful. But here's the thing: developers generally don't really care which large language model is being used if they can get the same quality, the same consistency, and the same speed from some noname model compared to ChatGPT, they probably don't care. What they care about is consistency of response, quality of response, and at scale cost. So as I mentioned, most developers are using a single endpoint and a single large language model. They're not using any of the algorithmic unlocks that have been coming out, which will only add quality. Even if you do nothing else, if you simply use some of these techniques Chain of Thought mixture of Agents, you literally just layer that on top of whatever large

language model you're already using. You will get much greater quality, but these things take a lot of work to implement. They're not that simple, and the techniques are constantly evolving. There's new techniques coming out all the time, and so being up to date on the latest algorithmic unlocks is a lot of work for a developer and something that is not really their core expertise. So what is abstract AI in the most simple terms? Abstract AI is a single drop-in API replacement for whatever large language model you're using today. It connects to multiple large language models, everything from closed-source frontier to open-source to even local models, and it is the cheapest, fastest, highest quality responses on a per-prompt basis. So it's able to use at the very front of the stack something like Route LLM to determine on a prompt-by-prompt basis which large language model should be used, which algorithm should be used by determining which will be fastest, cheapest, and highest quality. And as we've seen from the Route LLM research paper, you can achieve 90% of GPT-40 quality for 80% reduction.

in the cost simply by routing the prompt to the right large language model. And again, I want to stress this includes locals. So, if you're deploying AI within your business and you have 10,000 employees, you can install a very small model on every single one of the computers. And with Route LLM wrapped around it, it will answer essentially for free 90% of the prompts locally. And that increases safety and security, which we'll get to in a moment. And then for anything else, for anything more sophisticated, for coding use cases, for anything that is more complex, that is when it outsources to a larger model, to a private cloud model, or even a closed-source cloud provider like OpenAI. And if this sounds familiar, it is essentially what Apple Intelligence is doing. Even though they're completely closed-source, Apple Intelligence is installing a small model on the phone that can work on your behalf. And then for any world knowledge or any more complex use cases, it outsources either to their own private cloud but a massive

model or eventually with permission to open AI. So, how does this all work? One, it uses Route LLM as it says right here to pick the right large language model for each prompt. So, Route LLM out of the box comes with a really nice machine learning algorithm that can determine which model a prompt should go to. But that model, that ML model, can actually be trained for your specific use case and for your plethora of different models that you want to use, and even include algorithmic techniques to determine what quality is going to come out of each of those models. Then you layer on some of these algorithmic unlocks that we've talked about, either on the open-source smaller models or even on top of the larger Frontier models, although it's kind of less important at that level. You optimize on the small models using an agentic framework. So, Crew AI, for example, mixture of Agents Chain of Thought, and you can really just tell it what you want to get out of it and why. And so, if speed is what you're optimizing for, you

can actually tell the API about that, and then it will change its routing to optimize for speed. If the highest quality is what you're looking for based on your prompt, it will do that. So you can kind of manually tweak it as you're going as well. Then, as mentioned, you outsource to a frontier model only when required, kind of like what Apple intelligence does. So then, what, as I mentioned at the beginning, consistency and quality are kind of the two most important factors when an AI developer is deciding which large language model to build on top of. So, Abstract AI will have built-in benchmarking to optimize the response consistency, and the benchmarking can be custom. So, for a specific use case within a company or within a product or service, you can actually have benchmarks that you're running against for any new large language model that you want to include or any algorithm that you want to put on top of that. Then, it should also have eventually built-in caching. So, one tried and true way to basically guarantee consistency, improve quality, decrease cost, and latency is by using caching.

autogen does this really well. So if you submit the same prompt again and again, there's no reason why you actually need to hit a large language model again and again. You are expecting the same response, so simply return the cached version. This will optimize speed, cost, and consistency. So then, what is the broader vision of this project? I think there is a bunch of different areas for expansion, but one that makes a lot of sense to me is prompt management. Now, because Abstract AI is being inserted into such a critical path for any company, for any AI Builder, you all of a sudden have this really great expansion path as well. And one direction is upstream towards the user, and you can start doing prompt management, whether that's User permissioning, Group permissioning, Company rules, and then also versioning. So you can have different versions of prompts, and again, all of this tied into the benchmarks which we already talked about. And then from there, again, because this product is being inserted into the most critical area of an AI developer's workflow, you

can really expand in a bunch of different ways. So a bit rough, but hopefully you enjoyed this. I really think there is a great product, if not an actual business, to be built from this. And I don't have time, but if you decide to build it, reach out to me. I'd love to talk to you about it, and I'd love to help you in any way I can. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

This Could Be a MASSIVE AI Business...and Now It's Yours 🤑

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

The idea is to create an abstraction layer on top of large language models to optimize AI developers' use of tokens.

The startup aims to help AI product companies and large organizations using AI to optimize their use of large language model tokens, addressing the problem of overpaying for Frontier LLMs and the risk of platform dependency on a single cloud service provider.

The goal is to provide a more efficient and cost-effective solution for utilizing AI language models.

The Inefficiency of Using the Frontier Model in LLMs

The Frontier Model is often unnecessary and too slow for 90 to 95% of prompts and use cases, leading to inefficient LLM usage.

Abstract AI aims to solve this by reducing latency, lowering cost, and providing more flexibility, which is crucial for developers and companies deploying AI at scale.

The use of algorithmic techniques like Chain of Thought and Mixture of Agents is overlooked in favor of the expensive and slow Frontier Model, but developers prioritize consistency, quality, and cost-effectiveness in LLMs.

The Role of Abstract AI in Language Models

Abstract AI serves as a cost-effective and high-quality replacement for large language models, utilizing a variety of sources including local models.

By routing prompts to the most efficient model, it can significantly reduce costs while maintaining quality.

This approach, similar to Apple Intelligence, prioritizes local models for basic prompts, enhancing safety and security.

The Functioning of Route LLM for Large Language Models

Route LLM uses a machine learning algorithm to select the appropriate large language model for each prompt, which can be further trained for specific use cases.

The system can be optimized for speed or quality based on the user's needs and can outsource to a frontier model when necessary.

Benchmarking and caching are built-in features to ensure response consistency, quality, and efficiency, while also allowing for custom benchmarks for specific use cases.

Abstract AI has the opportunity to expand into prompt management, including user permissioning, group permissioning, company rules, and versioning.

This expansion is closely tied to the existing benchmarks, offering a broader vision for the project.

The product has the potential to grow significantly as it is integrated into the critical areas of AI developers' workflow.