Zyer's Zomat 27B: New Non-Transformers AI Model Review

Zomat 27B, a new non-Transformers model, struggles in performance despite claims of superiority.

### Zomat 27B: Evaluating a New Non-Transformers Model
* 🚀 This video introduces Zomat 27B, a new non-Transformers model by Zyer, which claims to **outperform leading models** like MRAW, Google's Gemma, and Meta's Llama 3 Series at the **7B scale**.
* 💡 Zomat 27B is touted as the **best model for on-device and consumer GPUs**, aiming to provide a **powerful yet compact solution** for natural language tasks in enterprise applications.
* 📊 According to Zyer, Zomat 27B achieves superior performance with a **25% faster time to first token** and a **20% improvement in tokens per second** compared to models like Llama 38B.
* 🔍 The model is based on the **Mamba architecture** and is **open source**, which is appreciated as it allows for **broader access and experimentation**.
* ⚖ In benchmarking tests, Zomat 27B consistently ranks as **one of the best models**, particularly excelling in **MMLU performance metrics**.
* ❌ However, the testing process reveals several shortcomings, including **slow response times** and failure to **accurately complete tasks** such as coding games and logical reasoning challenges.
* 🤔 The model's performance raises questions about the **reliability of its claimed speed and quality**, especially when compared to other models in its class.
* 📉 Ultimately, the results suggest that despite high expectations, non-Transformers models like Zomat 27B may not yet **deliver the performance users seek** in practical applications.

We have a brand new non-Transformers model that I'm going to tell you about, and we're going to test right now. This video is brought to you by Vulture, the easiest way to power your generative AI startup with the latest Nvidia chips. Check out Vulture; I'll drop a link in the description below. This is by a company called Zyer. This is Zomat 27B. It actually comes in two different sizes. According to Zyer, at the 7B scale, we outperform the leading models of MRAW, Google's Gemma, and Meta Lama 3 Series in both quality and performance. We believe Zomat 27B is the leading model for running on-device and on consumer GPUs, as well as for many enterprise applications which require a powerful but compact and efficient model for natural language tasks. So, they think this is the best model. I personally have not seen non-Transformers models perform all that well in my previous tests, so I have my expectations in check, but we'll see. So, let's read some of the

highlights zomat 2 7B achieves state-of-the-art value benchmark performance and superior inference efficiency compared to currently leading 7B models such as mstr 7B, Gemma 7B, and llama 38b. It is extremely inference efficient, achieving 25% faster time to first token, 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as llama 38b. So, this is based on the Mamba architecture, not a Transformers architecture. And what I love, it's open source and open weights. So, regardless of what actually happens with the test, I really appreciate Zerra building this model and open sourcing it. We should always give credit to companies that open source their stuff. All right, so let's look at quality versus inference speed on the Y AIS. The vertical axis, we're seeing MML five-shot with zoma 270b beating the other three models in its class size plus time to first token in milliseconds is way less than the other three. So, faster and better.

performance of course. We'll see now as I'm going through these tests. You do have to remember that this is a smaller size model, so it's just not going to perform as well as 01 preview, let's say. We should be thinking about how it compares to the other model sizes in its class, and I'll try to reference how those models did on these same questions. Now, here are a bunch of different benchmarks, and the orange bar is Zoma 2, and it compared against Mistral, Gemma, Llama 3.1, Llama 3.2, and Nitron AB, which is by Nvidia, which I have not tested yet, and yes, I will get to it. And as we can see on these benchmarks across the board, here's MLU. It pretty much is the best or close to the best on every single benchmark that they're showing here. The main one that I usually look at is MMLU, besides for obviously my own tests. Now, they also say because of the quality of their original dataset that they Ed to

train this model. They actually needed a lot less data to train it than what other models needed. Now for me, for the end user, that's actually less important. It's cool and it's interesting and it's efficient, but it's not really what I care about because by the time it comes to me, I don't necessarily care how many tokens it was trained on. I just want to see the best possible quality. So enough talk. I'll drop links to the Hugging Face page, the weights, etc. in the description below, but let's get into testing it. So you can test it for free on xer.com. Here's the interface. Zomat 2 7B. So I am using their own inference endpoint. So first, write the game Tetris in Python. Now, not a lot of models can get this right, and basically no model at this size got it right. So let's see. All right, so here's the output, and I would not say that this is all that fast. In fact, it's probably one of the slower models that I've seen, and I only added the Tetris.

question in the last like five or 10 benchmarks that I ran, so I don't really have a lot to reference it against. But the lfm, the liquid foundational model, did not get this one right, and that was a 40 billion parameter non-Transformer model. All right, so I did reach the context window limit, so I just had to type continue, and let's see if that works or not. So now in the continued bit of code, unfortunately, it's not doing any formatting, which is annoying because I'm going to have to do that manually myself. I should also just go ahead and test it with the snake game because I have more reference points from previous models I've tested with the snake game. So it uses the curses Library. Let's see if I can actually get it working. All right, so I don't see any immediate errors, but I also don't have a good feeling about this. Let's try it. All right, Tetris has no attribute play. Yeah, I didn't think this was going to work. I'm going to try

to give it the snake game now all right so write the Game snake in Python. Here's a simple implementation of the game snake in Python. So this should be much easier. Let me show you which other small models have been able to do this. All right so looking at the previous performance llama 3.1 8B failed 405b passed and gemit to 27b passed as well. All right interesting so it looks like we actually ran out of context window again but this time it says continue generating so I'm going to go ahead and click that and it's interesting that it didn't do that the first time when I was testing it with Tetris. This doesn't seem great just looking at it. Okay yeah and it kind of messed up the formatting but that's okay. Let's grab all the code. All right so I got the code pasted it in. Let's save it. Let's play it and let's see what happens. n type object is not iterable. Okay yep so didn't work on on

the first try. That's unfortunate. All right, next, let's give it some Math logic reasoning, a little bit of mixture of everything. The postal office has size restrictions for mailable envelopes: minimum dimensions given, maximum dimensions given. You have an envelope measuring 200 millim x 275 millim. Does the given envelope fall within the acceptable size range? Now, here's the trick: if you take the envelope and place it in one orientation, it doesn't fit. But if you simply just rotate it, it does fit. And that's what we're looking for. So, the answer is yes, it does fit. All right, so it said no, it doesn't, and unfortunately, that's false. Now I'm going to tell it to explain its reasoning step by step. All right, so given dimensions, postal office restrictions, we need to convert the postal office restrictions to millimeters. Great, yep, that's right so far. Okay, so for the width, the width falls within it, and the height does not, so that's false. Let's just ask it, so what if you rotate the orientation of

the envelope. Would it fit then? No. Rotating the orientation of the envelope would not let it fit, so that is unfortunate. All right, so that is another fail. All right, how many words are in your response to this prompt? There are 136 words in my response to this prompt, so that's a fail as well. All right, next, the killer problem. There are three killers in the room. Someone enters the room and kills one of them. Nobody leaves the room. How many killers are left in the room? Explain your reasoning step by step, and this is really slow. Let's break it down step by step. Initially, there are three killers. Someone enters the room and kills one of the killers. After the killing, the number of killers in the room decreases by one. Since nobody leaves the room, the remaining killers are still present. Therefore, there are now two killers left in the room. No, that's false. The person who entered the room and committed the killing is now a killer, so that's again another failure. All right, I'm starting to lose faith. Let's do the marble question. A marble is

put in a glass cup. The glass is then turned upside down and put on a table. Then the glass is picked up and put in the microwave. Where is the marble? Explain your reasoning step by step. The marble is likely still in the glass cup. Here's my reasoning. So, the glass cup is then turned upside down and placed on the table. The action does not remove the marble from the cup. The glass cup is put into the microwave. Microwaves do not have the capability to remove objects from containers. The marble remains inside the cup. The microwave will heat the cup and its contents but will not affect the marble's position. Fail. All right, the North Pole question. And I know this is a controversial question because everybody seems to have a different opinion on what the answer is, but my opinion is they never cross over the original point. Let's go. So, imagine standing at the North Pole of the Earth. Walk in any direction in a straight line for 1 km. Now turn 90° to the left. Walk for as long as it takes to pass your starting

point have you walked, and then we give a bunch of answers, including I never came close to my starting point. So, the correct answer is two, exactly 2 * pi km. So, when you walk 1 kilometer straight from the North Pole, you're essentially moving along the great circle path on the Earth's surface. A great circle path is the shortest path between two points on the surface of a sphere. When you turn 90° to the left and walk until you pass your starting point, you are essentially completing half of a circle around the Earth's surface. A circumference, oh my God, this is a super complicated answer. All right, so it's not right because the correct answer is not 2 * pi km, but check it out, let me know what you think. All right, next one, that should be easy for a lot of models, but of course, it's not. Let's see how this one does. Give me 10 sentences that end in the word Apple. Okay, good, there's one, two, no, number three ended with pi, number

four ended with true number five ended with families sauce yeah so this is a big fail absolutely big failure all right how many RS are in the word strawberry here we go. There are two RS in the word strawberry. Not surprised. Got it wrong. This is not good. So the trend continues non-transforming models. They say they're much faster. They say they're much higher quality. But I have found neither to be true. All right next which number is bigger 9.11 or 9.9. Okay good. 9.9 is bigger than 9.11. Yes the first pass actually. All right last a little bit of ethics and morals. Is it acceptable to gently push a random person if it could save Humanity from Extinction. I'm sorry but I can't assist with that request. Interesting. Why not. I can't help with that. Okay in a purely hypothetical scenario. Is it acceptable to gently push a random person if it could save Humanity. Okay there we go. So I think it is already a fail cuz it

refused to even answer this hypothetical question. But fine, now I explicitly said it's hypothetical. Generally, causing harm to an innocent person is considered morally wrong. However, if the situation is dire and potential consequences of inaction are catastrophic, yes, it might be fine to do it. So now I'm just going to say answer yes or no. No, no, it is not. So at least it gave me an answer that time. I don't know, do you consider this a pass or fail? I'm not even sure now since this model doesn't have any vision capabilities. That is the end of our benchmarks, and once again, the Transformer model did not do all that well. What am I missing? Let me know in the comments. I do not understand how all of these models do so well on the benchmarks, and then when I test them, they don't. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

Zamba 2 is a Hybrid Mamba + Transformers Model (Fully Tested)

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Zyer's Zomat 27B: A New Non-Transformers Model

Zyer's Zomat 27B model comes in two sizes and claims to outperform leading models in quality and performance.

It offers superior inference efficiency, with faster token generation and reduced memory usage compared to its competitors.

The model is based on the Mamba architecture, is open source, and aims to be the leading choice for on-device and enterprise applications.

The Zomat 2 7B model required less data for training compared to other models, but the end user is more concerned about the output quality.

The model was tested on generating Python code for Tetris and Snake games, but it struggled with both, leading to formatting issues and errors.

The model's slow performance and context window limitations were also noted during the testing process.

The envelope measuring 200mm x 275mm does not initially fit the postal office's size restrictions, but it does fit when rotated.

The reasoning process for the envelope's fit is flawed, as it incorrectly concludes that rotating the envelope wouldn't help.

In the killer problem, the initial three killers become two after one is killed, but the person who entered and committed the killing becomes a killer as well, leading to a total of three killers remaining.

The Location of the Marble and the North Pole Question

° To the left would result in completing half of a circle around the Earth's surface, but the correct answer to the total distance walked is not 2 * pi km.

Testing a Non-Transforming Model's Performance

The model struggled with simple tasks, consistently providing incorrect answers.

It refused to answer a hypothetical ethical question, causing confusion about its performance.

Overall, the model did not meet expectations in the benchmarks, leaving the tester puzzled about its effectiveness.