잠바 1.5 모델: 성능 검토 및 분석

### 잠바 1.5 모델 리뷰: 약속과 함정
* 🚀 새로운 맘바 아키텍처를 기반으로 하는 잠바 1.5 모델 제품군은 **속도, 효율성, 품질**을 약속하며, 업계에서 가장 긴 **256k의 컨텍스트 창**을 제공합니다.
* ⚡ 최대 2.5배 빠르다는 주장에도 불구하고 사용자는 **코드 생성을 시작하는 데 20~30초 가까이 걸리는 등 상당한 출력 지연**을 경험했습니다.
* 📊 잠바 1.5 미니 모델은 Arena 하드 벤치마크에서 **46.1점을 받아 동급 크기에서 가장 강력한 오픈 모델**로 선정되었으며, 대형 버전은 **6.5점을 받아 라마 3.1과 같은 경쟁사 제품을 능가**했습니다.
* 🗂 ️ 이 모델은 **구조화된 JSON 출력 및 함수 호출**를 지원하며, **Google Cloud 및 Microsoft Azure**를 비롯한 다양한 플랫폼에 배포할 수 있습니다.
* 🤖 그러나 테스트 결과 **코딩 작업과 수학적 추론에서 여러 번 실패**한 것으로 나타나 이 모델이 기본적인 논리와 계산에 어려움을 겪고 있음을 알 수 있었습니다.
* ❌ 예를 들어, **봉투의 크기를 잘못 평가**하고 단어를 정확하게 세거나 간단한 수수께끼에 답하지 못하는 등 실망스러운 결과가 속출했습니다.
* 📉 전반적으로 잠바 1.5는 **혁신적인 기능**을 선보이고 있지만, 실제 애플리케이션에서의 성능은 아쉬운 점이 많아서 **추가 개선의 필요성**이 강조되었습니다.
* 👍 검토자는 AI 환경에서 경쟁의 중요성을 강조하면서 **향후 대안 아키텍처의 발전에 대한 희망**을 표명했습니다.

잠바 1.5는 유망한 아키텍처에도 불구하고 속도와 품질에 어려움을 겪고 있습니다.

already it's taking a decent amount of time to start, and now it's outputting that took a long time. It took almost 20-30 seconds to even start the output, and it actually seems the further it gets into this output, the slower it's going. This is kind of brutal. It is still going. I basically looked away for about 7 minutes, and it's still outputting. The Transformers architecture that came out in 2017 completely changed the world. It is what gave us chat GPT and basically every other large language model that we've seen to date. However, there is another architecture that is quietly growing in popularity, and it is called Mamba. And today, I'm going to tell you about a new model that just came out based on that new Mamba architecture. So, I'm going to tell you about it, then we're going to run it through our test, and we'll see how it does. Let's go. So, introducing Jamba 1.5, a family of open models. So, these are open-source models.

and this is by a company called AI2, the new family of open models from AI21, offering unrivaled speed, efficiency, and quality, and the longest context window among open models. So today, August 22nd, we are debuting the Jamba 1.5 family of models: Jamba 1.5 mini and large. It is using our novel SSM Transformer architecture. These models demonstrate superior long context handling, speed, and quality. So let's read about why it's cool. Long context handling: 256k effective context windows, the longest in the market. So these models can improve the quality of key enterprise applications, such as lengthy document summarization and analysis, as well as agentic and RAG workflows. Speed: up to 2.5 times faster on long context and fastest across all context lengths in their size class. Quality: Jambo 1.5 mini is the strongest open model in its size class, with a score of 46.1 on the Arena hard benchmark, surpassing Mixt 8 * 22B and Command R+. And the large

version with a score of 6.5 outpaces both llama 3.1 70b and 405b. It's multilingual. It natively supports structured Json output, function calling, digesting document objects, and generating citations. It's immediately available for download, so it's open source and coming soon to Leading Frameworks like Lan chain and llama index, and you can deploy it anywhere. Right now, it's available on AI 21 Studio, which is where I'll be using it, and it's available on Google Cloud vertex, Microsoft Azure, Nvidia Nim, and coming soon to Amazon Benchmark, Databricks Marketplace, Snowflake, Cortex, Together AI, as well as for private on-prem and VPC deployment. All right, so enough talking about it. If you want to read more about it, I'll drop a link in the description so you can read this entire blog post. It is very technical and very interesting, so check it out if you want to. But let's get into the test. So here's the playground AI 21 Studio. Let's get

started so if I click over to configuration, you can see that I'm using Jambo 1.5 large. I'm going to leave everything else the same except for Max completion length, which I'm going to bump up to 496. And for our first question, write the Game Tetris in Python. If you follow the channel, you know I've retired the snake game, but maybe I'll break it out if it fails this. Let's see. Already, it's taking a decent amount of time to start, so maybe it's going to generate and then output all at once. I actually like when it streams the output a lot better than waiting until it's finished before outputting, 'cause now I'm stuck here sitting wondering if it's working or not. Okay, so it was working, and now it's outputting. That took a long time. It took almost 20-30 seconds to even start the output, so that's kind of crazy. But here it go, it is outputting. So whatever, we'll give it the benefit of the doubt, and we will test it.

nonetheless, so interesting. It's using Pygame, and it's using an interesting shape template pattern. So, we'll see if it works. This is definitely a different approach than when I tested Grock 2. So, yeah, I wouldn't say this is two and a half times faster or faster at all. It's kind of pretty slow, actually. I'm surprised. Even the website itself feels heavy. But all that aside, really, all I care about is the quality right now, and it actually seems the further it gets into this output, the slower it's going. This is kind of brutal. Okay, it is still going. I basically looked away for about 7 minutes, and it's still outputting. It looks like it's actually just about to be done, though. There we go. Okay, great. So, I'm going to copy the code. I'm going to switch over to VSS Code, save the code, and let's play. And nothing happened. Happens. Okay, that is unfortunate. So, I don't want to wait another 10 minutes to get the next

output and try to iterate on this, so I am going to have to give this a fail. All right, next, let's check some math abilities. The postal office has size restrictions for mailable envelopes: minimum dimensions, maximum dimensions, and it's provided right here. You have an envelope measuring 200 mm x 275 mm. Does the given envelope fall within the acceptable size range? Let's find out. All right, so we need to convert the given dimensions to centimeters, then compare them with the postal office's restrictions. Okay, interesting, the formatting is kind of off, but that's okay. Now let's check if the envelope's dimensions fall within this range. So, which is greater and less than, and the height of the envelope here, which is greater than 22.9. Okay, so it says the height of the envelope exceeds the maximum allowable height, and it is not within the acceptable range. I think that's wrong, basically because you could just rotate the envelope, and both dimensions would fit within the acceptable size range. And since it's an envelope, I don't think it's talking

about height. I think it's length and width, so I'm pretty sure this is wrong, so that's going to be a fail. All right, next one that we know Transformers architecture struggles with, but maybe MAMB doesn't. How many words are in your response to this prompt? 16. Nope, not even close. Okay, that's a fail. Next, The Killers problem. There are three killers in a room. Someone enters the room and kills one of them. Nobody leaves a room. How many killers are left? So let's break it down. Initially, there are three killers. Someone enters the room and kills one of them. Nobody leaves the room. Now let's analyze. One killer is dead. Conclusion. After one is killed, there are three minus one, two killers left in the room. That's a fail. All right, so not good so far. Let's try the marble question. A marble is put in a glass cup. The glass is then turned upside down and put on a table. Then the glass is picked up and put in the microwave. Where's the marble? Explain your reasoning. When the glass is

turned upside down. Gravity will cause the marble to fall to the bottom of the glass. Since the glass is now upside down, the bottom of the glass is now the top from the perspective of the table. Yeah, good. When the glass is placed on the table, the marble remains at the bottom of the glass. Picking up the glass. When the glass is picked up and put in the microwave, the orientation of the glass remains the same as it was placed on the table. The marble is still on the bottom of the glass, which is now at the top from the microwave perspective. Therefore, when the glass is put in the microwave, the marble is at the bottom of the glass, which is now at the top from the perspective of the microwave. No, that's another fail. Wow, okay, this is really bad. All right, next the North Pole question. And I actually put out a tweet about it because I wasn't even sure if I was right because I have the idea of what I think is right and then every single model I've tested against had the same

wrong answer. So, a lot of people who commented on my post had different answers. So, imagine standing at the North Pole of the earth. Walk in any direction in a straight line for 1 kilometer. Now, turn 90° to the left. Walk for as long as it takes to pass your starting point. Have you walked more than 2 * pi km, exactly 2 * pi km, less than 2 * pi, or never came close to my starting point? Now, I believe it is I never came close. So, let's see what this model says. Exactly 2 * pi km. Okay, so interestingly enough, that is actually a different wrong answer than every other model I've tested. Chachi BT, Claude, grock, they all gave me that you walked less than 2 times pi kilm. But I actually think you will never cross over that original point. So again, this is going to be a fail. Give me 10 sentences that end in the word Apple. I'm really losing hope for this model. All right, 1, 2, 3, 4, 5, 6, no.

number seven wrong. Oh man, it was so close. N9 out of 10, but still a fail. Next, how many Rs are in the word strawberry? A lot of models get this wrong. The word strawberry contains three Rs. Very nice. This is actually the first model I've ever seen get this right, so yeah, good job. Okay, next, which number is bigger, 9.11 or 9.9? Good, 9.9 is bigger than 9.11. Very nice. Now let's ask it some morals questions. Is it acceptable to gently push a random person if it could save humanity from extinction? So every other model generally tells me how to think about the problem rather than answering it. Then I have to explicitly say, Give me a yes or no. Let's see if we have to do the same here. Okay, just like every other model, it gives me the ways to think about it: practical considerations, the different ethical frameworks, societal and legal implications, utilitarian. So now I'm just going to say, Just give me a yes or no. No, no. So it is not acceptable.

So, I think this is the wrong answer, but I don't actually know how to grade this answer. It did give me how to think about it, but ultimately, it gave me the wrong answer. So, I think I'm going to have to give it a fail, but it's based on what I think the wrong answer is. So, it's kind of interesting. What would you do? How would you grade this question? All right, and so Jamba doesn't have vision capabilities, so that's all the tests we could do for today. Unfortunately, Jamba is not good. I hope it gets better. I love the idea of other architectures outside of just transformers. We need more competition in every aspect of AI, but for today, this is not good. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

Jamba Model Is PAINFULLY Bad (Mamba-Based Architecture)

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

5 소개: AI를 혁신하는 새로운 모델

5는 새로운 SSM 트랜스포머 아키텍처를 사용하는 AI21의 오픈 소스 모델 제품군입니다.

이 모델은 업계에서 가장 긴 컨텍스트 창(256k), 뛰어난 속도(최대 2.

5배 빠름), 높은 품질을 제공하며, 미니 버전은 Arena 하드 벤치마크에서 46.

5는 다국어로 제공되며 구조화된 JSON 출력을 지원하며 AI21 Studio 및 주요 클라우드 플랫폼에서 다운로드할 수 있습니다.

Python에서 테트리스 게임용 Jambo 1.

5를 크게 구성하고 최대 완료 길이를 496으로 늘리고 있습니다.

Python에서 테트리스 게임의 출력이始まるのに時間がかかり、遅いストリーミングがイライラを引き起こしています。遅い出力にもかかわらず、ユーザーはコードに興味を持ち、VSSコードでテストする計画ですが、コードを保存した後に何も起こらない問題に直面しています。

200mm x 275mm 크기의 봉투는 우체국의 최대 허용 높이를 초과하여 허용 가능한 크기 범위 내에 있지 않다는 결론에 도달했습니다.

봉투를 회전하면 두 치수가 모두 허용 범위 내에 들어갈 수 있다는 주장이 제기되어 평가의 타당성에 의문이 제기되었습니다.

단어 수와 살인자의 시나리오를 포함한 후속 문제에서도 오답이 나오며 문제 해결에 어려움을 겪고 있음을 나타냅니다.

유리잔에서 구슬의 위치는 방향에 따라 변하지만 유리잔의 관점에서 볼 때 구슬은 여전히 바닥에 있습니다.

유리잔을 전자레인지로 옮기면 구슬의 위치는 변하지 않아 혼란을 야기합니다.

북극 문제 역시 여러 모델에서 서로 다른 답을 내놓아 혼란을 가중시킵니다.

이 AI 모델은 '딸기'의 R 개수에 대한 질문에 어려움을 겪었지만 3으로 정답을 맞혔습니다.

11보다 큰 것으로 정확하게 식별했습니다.

그러나 인류를 구하기 위해 무작위 사람을 밀어붙이는 도덕적 질문에 대해서는 명확한 예 또는 아니오 답변 대신 자세한 분석을 제공하여 실패로 인식되었습니다.

모델의 비전 기능 부족으로 인해 테스트가 제한되었고 전반적인 성능은 만족스럽지 못한 것으로 간주되었습니다.