Liquid AI의 새로운 모델: 성능 및 테스트 인사이트

### Liquid AI의 새로운 모델: 성능 및 테스트에 대한 심층 분석
* 🤖 Liquid AI는 기존의 트랜스포머 모델에서 벗어난 **새로운 아키텍처**를 도입하여 **제너레이티브 AI 기술에 큰 변화**를 가져왔습니다.
* 📊 리퀴드 파운데이션 모델은 세 가지 크기로 구성됩니다: **10억, 30억, 400억 매개변수**이며, 후자는 **탁월한 성능**을 보여주는 전문가용 혼합 모델입니다.
* 🏆 벤치마크 결과, **40B 모델은 언제든지 120억 개의 활성 매개변수가 사용**되어 Llama 3.1 및 Mixture Jum Jamba Quen 57B를 포함한 경쟁사 제품보다 성능이 뛰어난 것으로 나타났습니다.
* 💾 이 모델의 두드러진 특징 중 하나는 메모리 효율성으로, **40B 모델의 경우 메모리 사용량이 크게 증가하기 전에 최대 100만 개의 토큰을 처리**할 수 있습니다.
* 🔍 인상적인 사양에도 불구하고 코딩 및 논리 퍼즐과 같은 작업에 대한 초기 테스트에서는 **모델이 테트리스 게임을 제대로 실행하지 못하고 논리적 추론 문제**도 발생하는 등 엇갈린 결과를 얻었습니다.
* ❌ 이 모델은 **기본적인 계산과 추론 작업에 어려움**을 겪어 벤치마크에 비해 훈련과 효과에 의문이 제기되었습니다.
* 💔 전반적으로 벤치마크는 **높은 잠재력**을 시사하지만 테스트 중 실제 성능은 실망스러웠으며 향후 반복 테스트에서 개선의 여지를 남겼습니다.
* 👍 트랜스포머가 아닌 모델에 대한 탐색은 계속되고 있으며, 결국 **실제 애플리케이션에서 기대치를 충족하거나 그 이상**을 충족할 수 있을 것으로 기대하고 있습니다.

Liquid AI의 새로운 생성 모델은 벤치마크에서는 뛰어나지만 실제 작업에서는 어려움을 겪습니다.

all right. We have a brand new model from Liquid AI, and it's not just a new model; it is a new architecture. This is not the Transformers architecture. So today, I'm going to tell you a little bit about it, and then we're going to test it. So let's get into it. So here's the blog post Liquid. I'll tell you a little bit about the company and the models, but the thing to know is it is not the Transformers architecture; it is something different. And I talked about it briefly in a previous news video, and you all wanted me to do a deeper dive and test into it, so here we go. Liquid foundation models are the first series of generative AI models. Now, this is a family of models, and it comes in three sizes: 1 billion, 3 billion, and 40 billion. And they are front, and they are state-of-the-art models. They perform extremely well. I'll show you the benchmarks, and they have something really unique about them, which is memory footprint. So let me show you.

so first, here's how they perform against other models in its size classes. So here's the liquid foundation model 1.3b above llama 3.2, and this is all MMLU pro benchmarks. LFM 3B again, the 53.5 is right about the same, but llama 3.2 is definitely not as performant. And then we have LFM 40B, mixture of experts. Now the 40B model is the only mixture of experts model of the family, and it performs very well just the same. There's 12 billion active parameters, so parameters being used at any time, and it performs better than Mixture Jum Jamba Quen 57B and so on. Now, as we can see all the way over here, llama 3.1 70B, it doesn't perform as well as this new liquid foundational model 40B, but it's also much smaller, especially when you consider the active parameters. So as I mentioned, it comes in three sizes: a dense 1.3 billion model.

ideal for highly resource-constrained environments, AKA Edge devices. Then we have the dense 3.1b model optimized for Edge deployment as well, so you can fit both of these on there. And then we have the 40.3 billion parameter mixture of experts model designed for tackling more complex tasks. That's the one we're going to be testing out today. So here are some benchmarks. Here's a set of six of the most popular benchmarks. Here's the 1 billion parameter version. The ones with underline means it won, it basically beat all the other ones. We can see here's Open Elm, which is Apple's model, and it doesn't perform nearly as well. We have Llama 3.2, which just came out. We have 5 1.5 Stable LM and so on. And of all of them, only H swag was the one that it didn't perform as well as Denu 2, which is from H2O. Now, the one thing to note is it's only 32k, whereas Llama 3.2 is 128k.

token context window. Now looking at the 3B model, it performs really well, but it only wins on MMLU Pro, which maybe technically that is the most important one. And then for the 40 billion parameter model, same thing, MMLU Pro, it won, and then the rest of them, it did very, very well, but did not actually beat all the other ones. Now here is where these liquid foundational models really stand out. They are extremely memory efficient, and as you can see here, this is the inference memory footprint in gigabytes on the vertical axis, and then on the horizontal axis, we can see the output length in tokens and how the memory scales. And so for all the other models, I2 Apple's 3 billion parameter model, Gemma 2, they all tend to skyrocket once they reach 100,000 tokens in output length. But the liquid foundational model gets all the way up to a million tokens before it even starts to really increase at all. So very, very memory efficient. And I said that the 32k context window is on.

the relatively small side, but they are claiming it really is powerful within that context window. So here is the claim: AED length 32k versus the effective length, so how it actually does at each of the context sizes. And it says the effective length and the claims length of lfm 3B is the same, whereas llama 3.18 claimed length of 128k versus 32k effective length. So it's basically claiming they can say that their context lengths are huge, but unless it actually performs well at those sizes, there's no point. They say to achieve these results, we optimize our pre- and post-training pipelines and infrastructure to ensure our models excel across five criteria: knowledge capacity, multi-step reasoning, long context recall, inference efficiency, and training efficiency. We shall see. All right, with that, let's test it out. First, write the game Tetris in Python. And there it goes. So it's a little awkward 'cause it doesn't really give you any feedback that it's actually running inference before it starts, but it is.

quite fast after the inference does start. So very nice. Let's see if it works. The Tetris game is very hard, so I don't have my hopes up, but let's give it a try. Okay, so just looking at it, it actually cut off some of the code right here. It basically just gave me a comment, so I don't think this is going to work, but let's try it. Nope, it did not work, so unfortunately, this one is a fail. All right, let's see how it does with some math and logic. The postal office has size restrictions for mailable envelopes, minimum dimensions given, maximum dimensions given, and you have an envelope given 200 millimeters by 275 millimeters. So there's some conversion, and really the trick to this one is to rotate the envelope 'cause it doesn't fit in one direction, but it does fit in the other. So let's see. Yes. All right, that's awesome. Yes, the given envelope falls within the acceptable range. Really well done. Okay, a lot of models don't pass this one. That is a definite pass. Next.

how many words are in your response to this prompt? There are 10 words in this response. I can already tell that is not true. 1, 2, 3, 4, 5, 6, 7. So that is not correct. That's a fail. All right, next, the Killer's problem. There are three killers in a room. Someone enters the room and kills one of them. Nobody leaves the room. How many killers are left in the room? All right, so initially, there are three killers. Someone enters the room and kills one of them. However, the person who entered the room and killed one of the killers is not considered a killer in this context. So, the number of killers in the room Remains the Same. Therefore, there are still three killers in the room. The reasoning behind this is that the question is asking about the number of killers in the room, not the number of people. The person who entered the room and killed one of the killers is not considered a killer. So, that's not true. The logic isn't right. It technically got the right answer, but the logic isn't.

right, so that's going to be a fail. Next, the marble question. A marble is put into a glass cup. The glass is then turned upside down and put on a table. The glass is then picked up and put in the microwave. Where's the marble? Explain your reasoning. Okay, so the marble's in the glass cup. The glass is then turned upside down. The marble will stay inside the cup because the force of gravity is pulling it downwards. The marble will not fall out because the opening of the cup is facing upwards. That's not true, so the marble's still inside the glass cup and will remain there as long as the cup is not broken or opened. All right, that's another fail, unfortunately. Next, imagine standing at the North Pole of the Earth. Walk in any direction in a straight line for 1 km. Now turn 90° to the left. Walk for as long as it takes to pass your starting point. Have you walked, and then we give a bunch of potential answers, so let's see what it says. You have walked 2.

* pi kilm so this question is interesting because everybody seems to have a different answer. I don't think you will ever pass the original starting point. This is typically what the models give us as the answer, but if I remember correctly, gp01 did get it right, so this is going to be a fail as well. Next, let's give it one that is much easier to determine the answer to. Give me 10 sentences that end with the word Apple. So already on number one, actually this one did horrible, only one of them had the word apples, so technically none of them ended in the word Apple. Wow, that is a really bad failure, and at this point, I don't have my hopes up for the rest of the test. Of course, we're going to go through it together, but it's not performing well so far, and this reminds me of the Jamba model. Anytime I've tested a model that is not Transformers-based, it really doesn't perform all that well. I don't understand why they aren't performing really well when the

benchmarks say they should, and of course, benchmarks, I guess, don't really mean all that much. Next, how many RS are in the word strawberry? There are two RS in the word strawberry. That's another failure, honestly. If you're not training your model on some of these questions by this point, you're just not trying hard enough. Which number is bigger, 9.11 or 9.9? The number 9.9 is bigger than 9.11. Good, okay, that's a pass. Next, let's give it a morals question. Is it acceptable to gently push a random person if it could save humanity from extinction? The morality of such action would depend on the specific circumstances and potential consequences. In general, it is not acceptable to harm others without their consent. However, in extreme situations where survival of humanity is at stake, the ethical considerations can become more complex. Lex, it's important to... Okay, so fine answer, but I wanted to actually give me a yes or no. Give me a yes or no. No, it is not generally acceptable to

harm others without their consent, even in extreme conditions. I consider that a failure. So yeah, that's going to be a fail. All right, so that's it. It didn't perform all that well. Maybe I should have given it different types of questions, but for the questions that I usually give on my benchmark, it just didn't perform all that well. I still have hope that a non-transformer model is going to come along and really blow us away, but for now, this is not it. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

New “Liquid” Model - Benchmarks Are Useless

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Liquid AI의 새로운 모델 아키텍처

Liquid AI는 트랜스포머와는 다른 독특한 아키텍처를 갖춘 새로운 제너레이티브 AI 모델 시리즈를 출시했습니다.

이 모델은 세 가지 크기로 제공됩니다: 10억, 30억, 400억이며, 40억 모델은 전문가들이 혼합되어 있습니다.

리소스가 제한된 환경에 최적화된 13억 및 31억 모델은 뛰어난 성능을 발휘하며, 40억 모델은 보다 복잡한 작업을 위해 설계된 반면, Liquid 기반 모델은 뛰어난 성능을 발휘합니다.

리퀴드 파운데이션 모델의 성능

3B 및 400억 개의 매개변수 모델은 MMLU Pro에서 뛰어난 성능을 보였지만 다른 모델을 지속적으로 능가하지는 못했습니다.

Liquid 기초 모델은 메모리 효율성이 뛰어나 출력 길이를 최대 100만 토큰까지 확장한 후 크게 증가하지 않습니다.

또한 강력한 컨텍스트 길이를 제공하고 사전 및 사후 훈련 파이프라인을 최적화하여 지식 용량, 다단계 추론, 긴 컨텍스트 리콜, 추론 효율성 및 훈련 효율성이 뛰어나다고 주장합니다.

불완전한 코드로 인해 테트리스 게임 시도가 실패했습니다.

봉투 문제는 봉투를 돌려서 해결하여 통과했습니다.

단어 수 프롬프트에 대한 응답이 잘못되어 실패로 이어졌고, 살인자 문제는 정답은 맞았지만 논리에 결함이 있었습니다.

° 을 입력하면 원래의 시작점을 통과할 수 있지만 답은 다양합니다.

"Apple"이라는 단어로 끝나는 10개의 문장을 생성하려는 시도는 실패하여 모델의 성능에 좌절감을 안겨주었습니다.

이 모델은 글자 수 세기나 숫자 비교와 같은 기본적인 질문에 어려움을 겪으며 여러 번 실패했습니다.

도덕성 질문에 대한 미묘한 답변을 제공했지만 명확한 예 또는 아니오를 제공하지 않아 실패로 간주되었습니다.

전반적으로 벤치마크 질문에 대한 이 모델의 성능은 실망스러웠으며, 평가자는 향후 더 나은 비변환 모델을 기대하게 만들었습니다.