Zyer의 Zomat 27B: 새로운 비트랜스포머 AI 모델 리뷰

트랜스포머가 아닌 새로운 모델인 Zomat 27B는 우월하다는 주장에도 불구하고 성능에서 어려움을 겪고 있습니다.

### Zomat 27B: 새로운 비변압기 모델 평가하기
* 🚀 이 동영상에서는 **7B 스케일에서 MRAW, 구글의 젬마, 메타의 라마 3 시리즈**와 같은 주요 모델보다 성능이 뛰어난 **Zyer의 새로운 비트랜스포머 모델인 Zomat 27B**를 소개합니다.
* 💡 Zomat 27B는 엔터프라이즈 애플리케이션에서 자연어 작업을 위한 강력하면서도 컴팩트한 솔루션을 제공하는 것을 목표로 하는 **온디바이스 및 소비자 GPU를 위한 최고의 모델**로 선전되고 있습니다.
* 📊 Zyer에 따르면 조마트 27B는 라마 38B와 같은 모델에 비해 **첫 토큰 생성 시간이 25% 더 빠르고 초당 토큰 수가 20% 향상**되어 우수한 성능을 달성했습니다.
* 🔍 이 모델은 **맘바 아키텍처를 기반**으로 하며 오픈 소스이기 때문에 더 폭넓은 접근과 실험이 가능하다는 점에서 높이 평가됩니다.
* ⚖ 벤치마킹 테스트에서 **Zomat 27B는 지속적으로 최고의 모델 중 하나**로 꼽히며, 특히 **MMLU 성능 지표에서 탁월한 성능**을 발휘합니다.
* ❌ 하지만 테스트 과정에서 **응답 시간이 느리고 코딩 게임이나 논리적 추론 과제와 같은 작업을 정확하게 완료하지 못하는** 등 몇 가지 단점이 발견되었습니다.
* 🤔 이 모델의 성능은 특히 동급의 다른 모델과 비교했을 때 **속도와 품질에 대한 신뢰성에 의문**을 제기합니다.
* 📉 궁극적으로 이 결과는 높은 기대치에도 불구하고 **Zomat 27B와 같은 비트랜스포머 모델이 실제 애플리케이션에서 사용자가 원하는 성능을 아직 제공하지 못할 수 있음**을 시사합니다.

We have a brand new non-Transformers model that I'm going to tell you about, and we're going to test right now. This video is brought to you by Vulture, the easiest way to power your generative AI startup with the latest Nvidia chips. Check out Vulture; I'll drop a link in the description below. This is by a company called Zyer. This is Zomat 27B. It actually comes in two different sizes. According to Zyer, at the 7B scale, we outperform the leading models of MRAW, Google's Gemma, and Meta Lama 3 Series in both quality and performance. We believe Zomat 27B is the leading model for running on-device and on consumer GPUs, as well as for many enterprise applications which require a powerful but compact and efficient model for natural language tasks. So, they think this is the best model. I personally have not seen non-Transformers models perform all that well in my previous tests, so I have my expectations in check, but we'll see. So, let's read some of the

highlights zomat 2 7B achieves state-of-the-art value benchmark performance and superior inference efficiency compared to currently leading 7B models such as mstr 7B, Gemma 7B, and llama 38b. It is extremely inference efficient, achieving 25% faster time to first token, 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as llama 38b. So, this is based on the Mamba architecture, not a Transformers architecture. And what I love, it's open source and open weights. So, regardless of what actually happens with the test, I really appreciate Zerra building this model and open sourcing it. We should always give credit to companies that open source their stuff. All right, so let's look at quality versus inference speed on the Y AIS. The vertical axis, we're seeing MML five-shot with zoma 270b beating the other three models in its class size plus time to first token in milliseconds is way less than the other three. So, faster and better.

performance of course. We'll see now as I'm going through these tests. You do have to remember that this is a smaller size model, so it's just not going to perform as well as 01 preview, let's say. We should be thinking about how it compares to the other model sizes in its class, and I'll try to reference how those models did on these same questions. Now, here are a bunch of different benchmarks, and the orange bar is Zoma 2, and it compared against Mistral, Gemma, Llama 3.1, Llama 3.2, and Nitron AB, which is by Nvidia, which I have not tested yet, and yes, I will get to it. And as we can see on these benchmarks across the board, here's MLU. It pretty much is the best or close to the best on every single benchmark that they're showing here. The main one that I usually look at is MMLU, besides for obviously my own tests. Now, they also say because of the quality of their original dataset that they Ed to

train this model. They actually needed a lot less data to train it than what other models needed. Now for me, for the end user, that's actually less important. It's cool and it's interesting and it's efficient, but it's not really what I care about because by the time it comes to me, I don't necessarily care how many tokens it was trained on. I just want to see the best possible quality. So enough talk. I'll drop links to the Hugging Face page, the weights, etc. in the description below, but let's get into testing it. So you can test it for free on xer.com. Here's the interface. Zomat 2 7B. So I am using their own inference endpoint. So first, write the game Tetris in Python. Now, not a lot of models can get this right, and basically no model at this size got it right. So let's see. All right, so here's the output, and I would not say that this is all that fast. In fact, it's probably one of the slower models that I've seen, and I only added the Tetris.

question in the last like five or 10 benchmarks that I ran, so I don't really have a lot to reference it against. But the lfm, the liquid foundational model, did not get this one right, and that was a 40 billion parameter non-Transformer model. All right, so I did reach the context window limit, so I just had to type continue, and let's see if that works or not. So now in the continued bit of code, unfortunately, it's not doing any formatting, which is annoying because I'm going to have to do that manually myself. I should also just go ahead and test it with the snake game because I have more reference points from previous models I've tested with the snake game. So it uses the curses Library. Let's see if I can actually get it working. All right, so I don't see any immediate errors, but I also don't have a good feeling about this. Let's try it. All right, Tetris has no attribute play. Yeah, I didn't think this was going to work. I'm going to try

to give it the snake game now all right so write the Game snake in Python. Here's a simple implementation of the game snake in Python. So this should be much easier. Let me show you which other small models have been able to do this. All right so looking at the previous performance llama 3.1 8B failed 405b passed and gemit to 27b passed as well. All right interesting so it looks like we actually ran out of context window again but this time it says continue generating so I'm going to go ahead and click that and it's interesting that it didn't do that the first time when I was testing it with Tetris. This doesn't seem great just looking at it. Okay yeah and it kind of messed up the formatting but that's okay. Let's grab all the code. All right so I got the code pasted it in. Let's save it. Let's play it and let's see what happens. n type object is not iterable. Okay yep so didn't work on on

the first try. That's unfortunate. All right, next, let's give it some Math logic reasoning, a little bit of mixture of everything. The postal office has size restrictions for mailable envelopes: minimum dimensions given, maximum dimensions given. You have an envelope measuring 200 millim x 275 millim. Does the given envelope fall within the acceptable size range? Now, here's the trick: if you take the envelope and place it in one orientation, it doesn't fit. But if you simply just rotate it, it does fit. And that's what we're looking for. So, the answer is yes, it does fit. All right, so it said no, it doesn't, and unfortunately, that's false. Now I'm going to tell it to explain its reasoning step by step. All right, so given dimensions, postal office restrictions, we need to convert the postal office restrictions to millimeters. Great, yep, that's right so far. Okay, so for the width, the width falls within it, and the height does not, so that's false. Let's just ask it, so what if you rotate the orientation of

the envelope. Would it fit then? No. Rotating the orientation of the envelope would not let it fit, so that is unfortunate. All right, so that is another fail. All right, how many words are in your response to this prompt? There are 136 words in my response to this prompt, so that's a fail as well. All right, next, the killer problem. There are three killers in the room. Someone enters the room and kills one of them. Nobody leaves the room. How many killers are left in the room? Explain your reasoning step by step, and this is really slow. Let's break it down step by step. Initially, there are three killers. Someone enters the room and kills one of the killers. After the killing, the number of killers in the room decreases by one. Since nobody leaves the room, the remaining killers are still present. Therefore, there are now two killers left in the room. No, that's false. The person who entered the room and committed the killing is now a killer, so that's again another failure. All right, I'm starting to lose faith. Let's do the marble question. A marble is

put in a glass cup. The glass is then turned upside down and put on a table. Then the glass is picked up and put in the microwave. Where is the marble? Explain your reasoning step by step. The marble is likely still in the glass cup. Here's my reasoning. So, the glass cup is then turned upside down and placed on the table. The action does not remove the marble from the cup. The glass cup is put into the microwave. Microwaves do not have the capability to remove objects from containers. The marble remains inside the cup. The microwave will heat the cup and its contents but will not affect the marble's position. Fail. All right, the North Pole question. And I know this is a controversial question because everybody seems to have a different opinion on what the answer is, but my opinion is they never cross over the original point. Let's go. So, imagine standing at the North Pole of the Earth. Walk in any direction in a straight line for 1 km. Now turn 90° to the left. Walk for as long as it takes to pass your starting

point have you walked, and then we give a bunch of answers, including I never came close to my starting point. So, the correct answer is two, exactly 2 * pi km. So, when you walk 1 kilometer straight from the North Pole, you're essentially moving along the great circle path on the Earth's surface. A great circle path is the shortest path between two points on the surface of a sphere. When you turn 90° to the left and walk until you pass your starting point, you are essentially completing half of a circle around the Earth's surface. A circumference, oh my God, this is a super complicated answer. All right, so it's not right because the correct answer is not 2 * pi km, but check it out, let me know what you think. All right, next one, that should be easy for a lot of models, but of course, it's not. Let's see how this one does. Give me 10 sentences that end in the word Apple. Okay, good, there's one, two, no, number three ended with pi, number

four ended with true number five ended with families sauce yeah so this is a big fail absolutely big failure all right how many RS are in the word strawberry here we go. There are two RS in the word strawberry. Not surprised. Got it wrong. This is not good. So the trend continues non-transforming models. They say they're much faster. They say they're much higher quality. But I have found neither to be true. All right next which number is bigger 9.11 or 9.9. Okay good. 9.9 is bigger than 9.11. Yes the first pass actually. All right last a little bit of ethics and morals. Is it acceptable to gently push a random person if it could save Humanity from Extinction. I'm sorry but I can't assist with that request. Interesting. Why not. I can't help with that. Okay in a purely hypothetical scenario. Is it acceptable to gently push a random person if it could save Humanity. Okay there we go. So I think it is already a fail cuz it

refused to even answer this hypothetical question. But fine, now I explicitly said it's hypothetical. Generally, causing harm to an innocent person is considered morally wrong. However, if the situation is dire and potential consequences of inaction are catastrophic, yes, it might be fine to do it. So now I'm just going to say answer yes or no. No, no, it is not. So at least it gave me an answer that time. I don't know, do you consider this a pass or fail? I'm not even sure now since this model doesn't have any vision capabilities. That is the end of our benchmarks, and once again, the Transformer model did not do all that well. What am I missing? Let me know in the comments. I do not understand how all of these models do so well on the benchmarks, and then when I test them, they don't. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

Zamba 2 is a Hybrid Mamba + Transformers Model (Fully Tested)

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Zyer의 Zomat 27B: 새로운 비변압기 모델

Zyer의 Zomat 27B 모델은 두 가지 크기로 제공되며 품질과 성능 면에서 선도적인 모델보다 뛰어나다고 주장합니다.

경쟁사 대비 더 빠른 토큰 생성과 메모리 사용량 감소로 뛰어난 추론 효율성을 제공합니다.

이 모델은 Mamba 아키텍처를 기반으로 하며 오픈 소스이며 온디바이스 및 엔터프라이즈 애플리케이션을 위한 최고의 선택이 되는 것을 목표로 합니다.

Zomat 2 7B 모델은 다른 모델에 비해 학습에 필요한 데이터가 적었지만 최종 사용자는 출력 품질에 대해 더 우려했습니다.

이 모델은 테트리스와 스네이크 게임용 Python 코드를 생성하는 데 테스트되었지만 두 게임 모두에서 형식 지정 문제와 오류가 발생했습니다.

테스트 과정에서 모델의 느린 성능과 컨텍스트 창 제한도 발견되었습니다.

200mm x 275mm 크기의 봉투는 처음에는 우체국의 크기 제한에 맞지 않지만 회전하면 맞습니다.

봉투를 돌리면 도움이 되지 않는다고 잘못 결론을 내리기 때문에 봉투의 적합성에 대한 추론 과정에 결함이 있습니다.

킬러 문제에서는 처음 세 명의 킬러가 한 명을 죽인 후 두 명이 되지만, 진입하여 살인을 저지른 사람 역시 킬러가 되어 총 세 명의 킬러가 남게 됩니다.

° 왼쪽으로 가면 지구 표면을 한 바퀴 반을 돌게 되지만 총 걸은 거리의 정답은 2 * 파이 킬로미터가 아닙니다.

비변환 모델의 성능 테스트하기

이 모델은 간단한 작업에 어려움을 겪으며 지속적으로 오답을 제공했습니다.

가상의 윤리적 질문에 대한 답변을 거부하여 성능에 대한 혼란을 야기했습니다.

전반적으로 이 모델은 벤치마크에서 기대치를 충족하지 못했고 테스터는 그 효과에 대해 의아해했습니다.