We have a brand new non-Transformers model that I'm going to tell you about, and we're going to test right now. This video is brought to you by Vulture, the easiest way to power your generative AI startup with the latest Nvidia chips. Check out Vulture; I'll drop a link in the description below. This is by a company called Zyer. This is Zomat 27B. It actually comes in two different sizes. According to Zyer, at the 7B scale, we outperform the leading models of MRAW, Google's Gemma, and Meta Lama 3 Series in both quality and performance. We believe Zomat 27B is the leading model for running on-device and on consumer GPUs, as well as for many enterprise applications which require a powerful but compact and efficient model for natural language tasks. So, they think this is the best model. I personally have not seen non-Transformers models perform all that well in my previous tests, so I have my expectations in check, but we'll see. So, let's read some of the

highlights zomat 2 7B achieves state-of-the-art value benchmark performance and superior inference efficiency compared to currently leading 7B models such as mstr 7B, Gemma 7B, and llama 38b. It is extremely inference efficient, achieving 25% faster time to first token, 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as llama 38b. So, this is based on the Mamba architecture, not a Transformers architecture. And what I love, it's open source and open weights. So, regardless of what actually happens with the test, I really appreciate Zerra building this model and open sourcing it. We should always give credit to companies that open source their stuff. All right, so let's look at quality versus inference speed on the Y AIS. The vertical axis, we're seeing MML five-shot with zoma 270b beating the other three models in its class size plus time to first token in milliseconds is way less than the other three. So, faster and better.

performance of course. We'll see now as I'm going through these tests. You do have to remember that this is a smaller size model, so it's just not going to perform as well as 01 preview, let's say. We should be thinking about how it compares to the other model sizes in its class, and I'll try to reference how those models did on these same questions. Now, here are a bunch of different benchmarks, and the orange bar is Zoma 2, and it compared against Mistral, Gemma, Llama 3.1, Llama 3.2, and Nitron AB, which is by Nvidia, which I have not tested yet, and yes, I will get to it. And as we can see on these benchmarks across the board, here's MLU. It pretty much is the best or close to the best on every single benchmark that they're showing here. The main one that I usually look at is MMLU, besides for obviously my own tests. Now, they also say because of the quality of their original dataset that they Ed to

train this model. They actually needed a lot less data to train it than what other models needed. Now for me, for the end user, that's actually less important. It's cool and it's interesting and it's efficient, but it's not really what I care about because by the time it comes to me, I don't necessarily care how many tokens it was trained on. I just want to see the best possible quality. So enough talk. I'll drop links to the Hugging Face page, the weights, etc. in the description below, but let's get into testing it. So you can test it for free on xer.com. Here's the interface. Zomat 2 7B. So I am using their own inference endpoint. So first, write the game Tetris in Python. Now, not a lot of models can get this right, and basically no model at this size got it right. So let's see. All right, so here's the output, and I would not say that this is all that fast. In fact, it's probably one of the slower models that I've seen, and I only added the Tetris.

question in the last like five or 10 benchmarks that I ran, so I don't really have a lot to reference it against. But the lfm, the liquid foundational model, did not get this one right, and that was a 40 billion parameter non-Transformer model. All right, so I did reach the context window limit, so I just had to type continue, and let's see if that works or not. So now in the continued bit of code, unfortunately, it's not doing any formatting, which is annoying because I'm going to have to do that manually myself. I should also just go ahead and test it with the snake game because I have more reference points from previous models I've tested with the snake game. So it uses the curses Library. Let's see if I can actually get it working. All right, so I don't see any immediate errors, but I also don't have a good feeling about this. Let's try it. All right, Tetris has no attribute play. Yeah, I didn't think this was going to work. I'm going to try

to give it the snake game now all right so write the Game snake in Python. Here's a simple implementation of the game snake in Python. So this should be much easier. Let me show you which other small models have been able to do this. All right so looking at the previous performance llama 3.1 8B failed 405b passed and gemit to 27b passed as well. All right interesting so it looks like we actually ran out of context window again but this time it says continue generating so I'm going to go ahead and click that and it's interesting that it didn't do that the first time when I was testing it with Tetris. This doesn't seem great just looking at it. Okay yeah and it kind of messed up the formatting but that's okay. Let's grab all the code. All right so I got the code pasted it in. Let's save it. Let's play it and let's see what happens. n type object is not iterable. Okay yep so didn't work on on

the first try. That's unfortunate. All right, next, let's give it some Math logic reasoning, a little bit of mixture of everything. The postal office has size restrictions for mailable envelopes: minimum dimensions given, maximum dimensions given. You have an envelope measuring 200 millim x 275 millim. Does the given envelope fall within the acceptable size range? Now, here's the trick: if you take the envelope and place it in one orientation, it doesn't fit. But if you simply just rotate it, it does fit. And that's what we're looking for. So, the answer is yes, it does fit. All right, so it said no, it doesn't, and unfortunately, that's false. Now I'm going to tell it to explain its reasoning step by step. All right, so given dimensions, postal office restrictions, we need to convert the postal office restrictions to millimeters. Great, yep, that's right so far. Okay, so for the width, the width falls within it, and the height does not, so that's false. Let's just ask it, so what if you rotate the orientation of

the envelope. Would it fit then? No. Rotating the orientation of the envelope would not let it fit, so that is unfortunate. All right, so that is another fail. All right, how many words are in your response to this prompt? There are 136 words in my response to this prompt, so that's a fail as well. All right, next, the killer problem. There are three killers in the room. Someone enters the room and kills one of them. Nobody leaves the room. How many killers are left in the room? Explain your reasoning step by step, and this is really slow. Let's break it down step by step. Initially, there are three killers. Someone enters the room and kills one of the killers. After the killing, the number of killers in the room decreases by one. Since nobody leaves the room, the remaining killers are still present. Therefore, there are now two killers left in the room. No, that's false. The person who entered the room and committed the killing is now a killer, so that's again another failure. All right, I'm starting to lose faith. Let's do the marble question. A marble is

put in a glass cup. The glass is then turned upside down and put on a table. Then the glass is picked up and put in the microwave. Where is the marble? Explain your reasoning step by step. The marble is likely still in the glass cup. Here's my reasoning. So, the glass cup is then turned upside down and placed on the table. The action does not remove the marble from the cup. The glass cup is put into the microwave. Microwaves do not have the capability to remove objects from containers. The marble remains inside the cup. The microwave will heat the cup and its contents but will not affect the marble's position. Fail. All right, the North Pole question. And I know this is a controversial question because everybody seems to have a different opinion on what the answer is, but my opinion is they never cross over the original point. Let's go. So, imagine standing at the North Pole of the Earth. Walk in any direction in a straight line for 1 km. Now turn 90° to the left. Walk for as long as it takes to pass your starting

point have you walked, and then we give a bunch of answers, including I never came close to my starting point. So, the correct answer is two, exactly 2 * pi km. So, when you walk 1 kilometer straight from the North Pole, you're essentially moving along the great circle path on the Earth's surface. A great circle path is the shortest path between two points on the surface of a sphere. When you turn 90° to the left and walk until you pass your starting point, you are essentially completing half of a circle around the Earth's surface. A circumference, oh my God, this is a super complicated answer. All right, so it's not right because the correct answer is not 2 * pi km, but check it out, let me know what you think. All right, next one, that should be easy for a lot of models, but of course, it's not. Let's see how this one does. Give me 10 sentences that end in the word Apple. Okay, good, there's one, two, no, number three ended with pi, number

four ended with true number five ended with families sauce yeah so this is a big fail absolutely big failure all right how many RS are in the word strawberry here we go. There are two RS in the word strawberry. Not surprised. Got it wrong. This is not good. So the trend continues non-transforming models. They say they're much faster. They say they're much higher quality. But I have found neither to be true. All right next which number is bigger 9.11 or 9.9. Okay good. 9.9 is bigger than 9.11. Yes the first pass actually. All right last a little bit of ethics and morals. Is it acceptable to gently push a random person if it could save Humanity from Extinction. I'm sorry but I can't assist with that request. Interesting. Why not. I can't help with that. Okay in a purely hypothetical scenario. Is it acceptable to gently push a random person if it could save Humanity. Okay there we go. So I think it is already a fail cuz it

refused to even answer this hypothetical question. But fine, now I explicitly said it's hypothetical. Generally, causing harm to an innocent person is considered morally wrong. However, if the situation is dire and potential consequences of inaction are catastrophic, yes, it might be fine to do it. So now I'm just going to say answer yes or no. No, no, it is not. So at least it gave me an answer that time. I don't know, do you consider this a pass or fail? I'm not even sure now since this model doesn't have any vision capabilities. That is the end of our benchmarks, and once again, the Transformer model did not do all that well. What am I missing? Let me know in the comments. I do not understand how all of these models do so well on the benchmarks, and then when I test them, they don't. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

Zamba 2 is a Hybrid Mamba + Transformers Model (Fully Tested)

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Zamba 2 is a Hybrid Mamba + Transformers Model (Fully Tested)

Zamba 2 is a Hybrid Mamba + Transformers Model (Fully Tested)

스크립트