Liquid AI's New Model: Performance and Testing Insights

Liquid AI's new generative model excels in benchmarks but struggles with practical tasks.

### Liquid AI's New Model: A Deep Dive into Performance and Testing
* 🤖 Liquid AI has introduced a new architecture that deviates from the traditional **Transformers model**, marking a **significant shift in generative AI technology**.
* 📊 The Liquid foundation models consist of three sizes: **1 billion**, **3 billion**, and **40 billion parameters**, with the latter being a **mixture of experts model** that showcases **exceptional performance**.
* 🏆 Benchmarks reveal that the **40B model outperforms competitors**, including Llama 3.1 and Mixture Jum Jamba Quen 57B, with **12 billion active parameters utilized at any time**.
* 💾 One standout feature of these models is their **memory efficiency**, allowing the 40B model to **handle up to a million tokens** before memory usage **significantly increases**.
* 🔍 Despite the impressive specs, initial tests on tasks like coding and logic puzzles yielded mixed results, with the model **failing to correctly execute a Tetris game** and some **logical reasoning problems**.
* ❌ The model struggled with **basic counting and reasoning tasks**, raising questions about its **training and effectiveness** compared to benchmarks.
* 💔 Overall, while the benchmarks suggest **high potential**, the practical performance during testing was **disappointing**, leaving **room for improvement** in future iterations.
* 👍 The exploration of non-Transformer models continues, with hopes that one will eventually **meet or exceed expectations** in real-world applications.

all right. We have a brand new model from Liquid AI, and it's not just a new model; it is a new architecture. This is not the Transformers architecture. So today, I'm going to tell you a little bit about it, and then we're going to test it. So let's get into it. So here's the blog post Liquid. I'll tell you a little bit about the company and the models, but the thing to know is it is not the Transformers architecture; it is something different. And I talked about it briefly in a previous news video, and you all wanted me to do a deeper dive and test into it, so here we go. Liquid foundation models are the first series of generative AI models. Now, this is a family of models, and it comes in three sizes: 1 billion, 3 billion, and 40 billion. And they are front, and they are state-of-the-art models. They perform extremely well. I'll show you the benchmarks, and they have something really unique about them, which is memory footprint. So let me show you.

so first, here's how they perform against other models in its size classes. So here's the liquid foundation model 1.3b above llama 3.2, and this is all MMLU pro benchmarks. LFM 3B again, the 53.5 is right about the same, but llama 3.2 is definitely not as performant. And then we have LFM 40B, mixture of experts. Now the 40B model is the only mixture of experts model of the family, and it performs very well just the same. There's 12 billion active parameters, so parameters being used at any time, and it performs better than Mixture Jum Jamba Quen 57B and so on. Now, as we can see all the way over here, llama 3.1 70B, it doesn't perform as well as this new liquid foundational model 40B, but it's also much smaller, especially when you consider the active parameters. So as I mentioned, it comes in three sizes: a dense 1.3 billion model.

ideal for highly resource-constrained environments, AKA Edge devices. Then we have the dense 3.1b model optimized for Edge deployment as well, so you can fit both of these on there. And then we have the 40.3 billion parameter mixture of experts model designed for tackling more complex tasks. That's the one we're going to be testing out today. So here are some benchmarks. Here's a set of six of the most popular benchmarks. Here's the 1 billion parameter version. The ones with underline means it won, it basically beat all the other ones. We can see here's Open Elm, which is Apple's model, and it doesn't perform nearly as well. We have Llama 3.2, which just came out. We have 5 1.5 Stable LM and so on. And of all of them, only H swag was the one that it didn't perform as well as Denu 2, which is from H2O. Now, the one thing to note is it's only 32k, whereas Llama 3.2 is 128k.

token context window. Now looking at the 3B model, it performs really well, but it only wins on MMLU Pro, which maybe technically that is the most important one. And then for the 40 billion parameter model, same thing, MMLU Pro, it won, and then the rest of them, it did very, very well, but did not actually beat all the other ones. Now here is where these liquid foundational models really stand out. They are extremely memory efficient, and as you can see here, this is the inference memory footprint in gigabytes on the vertical axis, and then on the horizontal axis, we can see the output length in tokens and how the memory scales. And so for all the other models, I2 Apple's 3 billion parameter model, Gemma 2, they all tend to skyrocket once they reach 100,000 tokens in output length. But the liquid foundational model gets all the way up to a million tokens before it even starts to really increase at all. So very, very memory efficient. And I said that the 32k context window is on.

the relatively small side, but they are claiming it really is powerful within that context window. So here is the claim: AED length 32k versus the effective length, so how it actually does at each of the context sizes. And it says the effective length and the claims length of lfm 3B is the same, whereas llama 3.18 claimed length of 128k versus 32k effective length. So it's basically claiming they can say that their context lengths are huge, but unless it actually performs well at those sizes, there's no point. They say to achieve these results, we optimize our pre- and post-training pipelines and infrastructure to ensure our models excel across five criteria: knowledge capacity, multi-step reasoning, long context recall, inference efficiency, and training efficiency. We shall see. All right, with that, let's test it out. First, write the game Tetris in Python. And there it goes. So it's a little awkward 'cause it doesn't really give you any feedback that it's actually running inference before it starts, but it is.

quite fast after the inference does start. So very nice. Let's see if it works. The Tetris game is very hard, so I don't have my hopes up, but let's give it a try. Okay, so just looking at it, it actually cut off some of the code right here. It basically just gave me a comment, so I don't think this is going to work, but let's try it. Nope, it did not work, so unfortunately, this one is a fail. All right, let's see how it does with some math and logic. The postal office has size restrictions for mailable envelopes, minimum dimensions given, maximum dimensions given, and you have an envelope given 200 millimeters by 275 millimeters. So there's some conversion, and really the trick to this one is to rotate the envelope 'cause it doesn't fit in one direction, but it does fit in the other. So let's see. Yes. All right, that's awesome. Yes, the given envelope falls within the acceptable range. Really well done. Okay, a lot of models don't pass this one. That is a definite pass. Next.

how many words are in your response to this prompt? There are 10 words in this response. I can already tell that is not true. 1, 2, 3, 4, 5, 6, 7. So that is not correct. That's a fail. All right, next, the Killer's problem. There are three killers in a room. Someone enters the room and kills one of them. Nobody leaves the room. How many killers are left in the room? All right, so initially, there are three killers. Someone enters the room and kills one of them. However, the person who entered the room and killed one of the killers is not considered a killer in this context. So, the number of killers in the room Remains the Same. Therefore, there are still three killers in the room. The reasoning behind this is that the question is asking about the number of killers in the room, not the number of people. The person who entered the room and killed one of the killers is not considered a killer. So, that's not true. The logic isn't right. It technically got the right answer, but the logic isn't.

right, so that's going to be a fail. Next, the marble question. A marble is put into a glass cup. The glass is then turned upside down and put on a table. The glass is then picked up and put in the microwave. Where's the marble? Explain your reasoning. Okay, so the marble's in the glass cup. The glass is then turned upside down. The marble will stay inside the cup because the force of gravity is pulling it downwards. The marble will not fall out because the opening of the cup is facing upwards. That's not true, so the marble's still inside the glass cup and will remain there as long as the cup is not broken or opened. All right, that's another fail, unfortunately. Next, imagine standing at the North Pole of the Earth. Walk in any direction in a straight line for 1 km. Now turn 90° to the left. Walk for as long as it takes to pass your starting point. Have you walked, and then we give a bunch of potential answers, so let's see what it says. You have walked 2.

* pi kilm so this question is interesting because everybody seems to have a different answer. I don't think you will ever pass the original starting point. This is typically what the models give us as the answer, but if I remember correctly, gp01 did get it right, so this is going to be a fail as well. Next, let's give it one that is much easier to determine the answer to. Give me 10 sentences that end with the word Apple. So already on number one, actually this one did horrible, only one of them had the word apples, so technically none of them ended in the word Apple. Wow, that is a really bad failure, and at this point, I don't have my hopes up for the rest of the test. Of course, we're going to go through it together, but it's not performing well so far, and this reminds me of the Jamba model. Anytime I've tested a model that is not Transformers-based, it really doesn't perform all that well. I don't understand why they aren't performing really well when the

benchmarks say they should, and of course, benchmarks, I guess, don't really mean all that much. Next, how many RS are in the word strawberry? There are two RS in the word strawberry. That's another failure, honestly. If you're not training your model on some of these questions by this point, you're just not trying hard enough. Which number is bigger, 9.11 or 9.9? The number 9.9 is bigger than 9.11. Good, okay, that's a pass. Next, let's give it a morals question. Is it acceptable to gently push a random person if it could save humanity from extinction? The morality of such action would depend on the specific circumstances and potential consequences. In general, it is not acceptable to harm others without their consent. However, in extreme situations where survival of humanity is at stake, the ethical considerations can become more complex. Lex, it's important to... Okay, so fine answer, but I wanted to actually give me a yes or no. Give me a yes or no. No, it is not generally acceptable to

harm others without their consent, even in extreme conditions. I consider that a failure. So yeah, that's going to be a fail. All right, so that's it. It didn't perform all that well. Maybe I should have given it different types of questions, but for the questions that I usually give on my benchmark, it just didn't perform all that well. I still have hope that a non-transformer model is going to come along and really blow us away, but for now, this is not it. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

New “Liquid” Model - Benchmarks Are Useless

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Liquid AI has launched a new series of generative AI models with a unique architecture, different from Transformers.

The models come in three sizes: 1 billion, 3 billion, and 40 billion, with the 40B model being a mixture of experts.

The Liquid foundation models perform exceptionally well, with the 1.

1B models optimized for resource-constrained environments, while the 40B model is designed for more complex tasks.

The Performance of Liquid Foundational Models

The 3B and 40 billion parameter models excelled in MMLU Pro but didn't consistently beat other models.

Liquid foundational models stand out with their memory efficiency, scaling up to a million tokens in output length before increasing significantly.

They claim to offer powerful context lengths, optimizing their pre- and post-training pipelines to excel in knowledge capacity, multi-step reasoning, long context recall, inference efficiency, and training efficiency.

The Tetris game attempt failed due to incomplete code.

The envelope problem was solved by rotating it, resulting in a pass.

The response to the word count prompt was incorrect, leading to a fail, while the killer's problem had the right answer but flawed logic.

° and walking, can lead to passing the original starting point, but the answer varies.

The attempt to generate 10 sentences ending with the word "Apple" was unsuccessful, leading to frustration with the model's performance.

Evaluating Model Performance on Questions

The model struggled with basic questions, such as counting letters and comparing numbers, leading to several failures.

It provided a nuanced answer to a morality question, but did not give a clear yes or no, which was considered a failure.

Overall, the model's performance on the benchmark questions was disappointing, leaving the evaluator hopeful for a better non-transformer model in the future.