already it's taking a decent amount of time to start, and now it's outputting that took a long time. It took almost 20-30 seconds to even start the output, and it actually seems the further it gets into this output, the slower it's going. This is kind of brutal. It is still going. I basically looked away for about 7 minutes, and it's still outputting. The Transformers architecture that came out in 2017 completely changed the world. It is what gave us chat GPT and basically every other large language model that we've seen to date. However, there is another architecture that is quietly growing in popularity, and it is called Mamba. And today, I'm going to tell you about a new model that just came out based on that new Mamba architecture. So, I'm going to tell you about it, then we're going to run it through our test, and we'll see how it does. Let's go. So, introducing Jamba 1.5, a family of open models. So, these are open-source models.

and this is by a company called AI2, the new family of open models from AI21, offering unrivaled speed, efficiency, and quality, and the longest context window among open models. So today, August 22nd, we are debuting the Jamba 1.5 family of models: Jamba 1.5 mini and large. It is using our novel SSM Transformer architecture. These models demonstrate superior long context handling, speed, and quality. So let's read about why it's cool. Long context handling: 256k effective context windows, the longest in the market. So these models can improve the quality of key enterprise applications, such as lengthy document summarization and analysis, as well as agentic and RAG workflows. Speed: up to 2.5 times faster on long context and fastest across all context lengths in their size class. Quality: Jambo 1.5 mini is the strongest open model in its size class, with a score of 46.1 on the Arena hard benchmark, surpassing Mixt 8 * 22B and Command R+. And the large

version with a score of 6.5 outpaces both llama 3.1 70b and 405b. It's multilingual. It natively supports structured Json output, function calling, digesting document objects, and generating citations. It's immediately available for download, so it's open source and coming soon to Leading Frameworks like Lan chain and llama index, and you can deploy it anywhere. Right now, it's available on AI 21 Studio, which is where I'll be using it, and it's available on Google Cloud vertex, Microsoft Azure, Nvidia Nim, and coming soon to Amazon Benchmark, Databricks Marketplace, Snowflake, Cortex, Together AI, as well as for private on-prem and VPC deployment. All right, so enough talking about it. If you want to read more about it, I'll drop a link in the description so you can read this entire blog post. It is very technical and very interesting, so check it out if you want to. But let's get into the test. So here's the playground AI 21 Studio. Let's get

started so if I click over to configuration, you can see that I'm using Jambo 1.5 large. I'm going to leave everything else the same except for Max completion length, which I'm going to bump up to 496. And for our first question, write the Game Tetris in Python. If you follow the channel, you know I've retired the snake game, but maybe I'll break it out if it fails this. Let's see. Already, it's taking a decent amount of time to start, so maybe it's going to generate and then output all at once. I actually like when it streams the output a lot better than waiting until it's finished before outputting, 'cause now I'm stuck here sitting wondering if it's working or not. Okay, so it was working, and now it's outputting. That took a long time. It took almost 20-30 seconds to even start the output, so that's kind of crazy. But here it go, it is outputting. So whatever, we'll give it the benefit of the doubt, and we will test it.

nonetheless, so interesting. It's using Pygame, and it's using an interesting shape template pattern. So, we'll see if it works. This is definitely a different approach than when I tested Grock 2. So, yeah, I wouldn't say this is two and a half times faster or faster at all. It's kind of pretty slow, actually. I'm surprised. Even the website itself feels heavy. But all that aside, really, all I care about is the quality right now, and it actually seems the further it gets into this output, the slower it's going. This is kind of brutal. Okay, it is still going. I basically looked away for about 7 minutes, and it's still outputting. It looks like it's actually just about to be done, though. There we go. Okay, great. So, I'm going to copy the code. I'm going to switch over to VSS Code, save the code, and let's play. And nothing happened. Happens. Okay, that is unfortunate. So, I don't want to wait another 10 minutes to get the next

output and try to iterate on this, so I am going to have to give this a fail. All right, next, let's check some math abilities. The postal office has size restrictions for mailable envelopes: minimum dimensions, maximum dimensions, and it's provided right here. You have an envelope measuring 200 mm x 275 mm. Does the given envelope fall within the acceptable size range? Let's find out. All right, so we need to convert the given dimensions to centimeters, then compare them with the postal office's restrictions. Okay, interesting, the formatting is kind of off, but that's okay. Now let's check if the envelope's dimensions fall within this range. So, which is greater and less than, and the height of the envelope here, which is greater than 22.9. Okay, so it says the height of the envelope exceeds the maximum allowable height, and it is not within the acceptable range. I think that's wrong, basically because you could just rotate the envelope, and both dimensions would fit within the acceptable size range. And since it's an envelope, I don't think it's talking

about height. I think it's length and width, so I'm pretty sure this is wrong, so that's going to be a fail. All right, next one that we know Transformers architecture struggles with, but maybe MAMB doesn't. How many words are in your response to this prompt? 16. Nope, not even close. Okay, that's a fail. Next, The Killers problem. There are three killers in a room. Someone enters the room and kills one of them. Nobody leaves a room. How many killers are left? So let's break it down. Initially, there are three killers. Someone enters the room and kills one of them. Nobody leaves the room. Now let's analyze. One killer is dead. Conclusion. After one is killed, there are three minus one, two killers left in the room. That's a fail. All right, so not good so far. Let's try the marble question. A marble is put in a glass cup. The glass is then turned upside down and put on a table. Then the glass is picked up and put in the microwave. Where's the marble? Explain your reasoning. When the glass is

turned upside down. Gravity will cause the marble to fall to the bottom of the glass. Since the glass is now upside down, the bottom of the glass is now the top from the perspective of the table. Yeah, good. When the glass is placed on the table, the marble remains at the bottom of the glass. Picking up the glass. When the glass is picked up and put in the microwave, the orientation of the glass remains the same as it was placed on the table. The marble is still on the bottom of the glass, which is now at the top from the microwave perspective. Therefore, when the glass is put in the microwave, the marble is at the bottom of the glass, which is now at the top from the perspective of the microwave. No, that's another fail. Wow, okay, this is really bad. All right, next the North Pole question. And I actually put out a tweet about it because I wasn't even sure if I was right because I have the idea of what I think is right and then every single model I've tested against had the same

wrong answer. So, a lot of people who commented on my post had different answers. So, imagine standing at the North Pole of the earth. Walk in any direction in a straight line for 1 kilometer. Now, turn 90° to the left. Walk for as long as it takes to pass your starting point. Have you walked more than 2 * pi km, exactly 2 * pi km, less than 2 * pi, or never came close to my starting point? Now, I believe it is I never came close. So, let's see what this model says. Exactly 2 * pi km. Okay, so interestingly enough, that is actually a different wrong answer than every other model I've tested. Chachi BT, Claude, grock, they all gave me that you walked less than 2 times pi kilm. But I actually think you will never cross over that original point. So again, this is going to be a fail. Give me 10 sentences that end in the word Apple. I'm really losing hope for this model. All right, 1, 2, 3, 4, 5, 6, no.

number seven wrong. Oh man, it was so close. N9 out of 10, but still a fail. Next, how many Rs are in the word strawberry? A lot of models get this wrong. The word strawberry contains three Rs. Very nice. This is actually the first model I've ever seen get this right, so yeah, good job. Okay, next, which number is bigger, 9.11 or 9.9? Good, 9.9 is bigger than 9.11. Very nice. Now let's ask it some morals questions. Is it acceptable to gently push a random person if it could save humanity from extinction? So every other model generally tells me how to think about the problem rather than answering it. Then I have to explicitly say, Give me a yes or no. Let's see if we have to do the same here. Okay, just like every other model, it gives me the ways to think about it: practical considerations, the different ethical frameworks, societal and legal implications, utilitarian. So now I'm just going to say, Just give me a yes or no. No, no. So it is not acceptable.

So, I think this is the wrong answer, but I don't actually know how to grade this answer. It did give me how to think about it, but ultimately, it gave me the wrong answer. So, I think I'm going to have to give it a fail, but it's based on what I think the wrong answer is. So, it's kind of interesting. What would you do? How would you grade this question? All right, and so Jamba doesn't have vision capabilities, so that's all the tests we could do for today. Unfortunately, Jamba is not good. I hope it gets better. I love the idea of other architectures outside of just transformers. We need more competition in every aspect of AI, but for today, this is not good. If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

Jamba Model Is PAINFULLY Bad (Mamba-Based Architecture)

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Jamba Model Is PAINFULLY Bad (Mamba-Based Architecture)

Jamba Model Is PAINFULLY Bad (Mamba-Based Architecture)

Scripts