[Music] Hi, welcome to another video. Just yesterday, Google launched multiple minor features like code execution and some other updates. However, they also released their new open-source model, Gemma 2. If you don't know, let me tell you that Gemma is a family of lightweight, state-of-the-art open models built by the same research and technology used to create the Gemini models. So, you can think of it as a simplified version of Gemini that is open source. The first generation of Gemma models was pretty cool, but now they have released a new generation. It comes in two variants: a 9 billion parameter model and a 27 billion parameter model. They claim that Gemma 2 is higher performing and more efficient at inference than the first generation, with significant safety advancements built in. In fact, the 27b variant offers competitive alternatives to models more than twice its size. So, they say that it can match the performance of a larger model while being smaller, which is pretty cool. They also say that the 27b model can almost match the performance

of llama 370b B. While the 9B model can outperform the Llama 38b model, which is impressive. Additionally, they say that the inference speed for this model is very fast across multiple types of hardware, which is great. Now let's look at the benchmarks. In benchmarks, the Gemma 29b model beats Llama 3 across all benchmarks except Helis Swag, where it falls short by 0.1 points. The 27b model is the really interesting one here because it almost matches the performance of the 70b model while being three times smaller, which is amazing. In the MMU, it falls short by four points, and in the BBH, it falls short by six points, but considering the size, it's pretty good. At least in the benchmarks. However, I don't know why they didn't include the Quen 2 model, which can outperform it significantly. So I don't know why people would use this model, but we'll see if it's any good. Anyway, the license for these models

is also pretty good. You can use it commercially and for personal use, which is cool. The model is available on their Vertex AI platform, AI Studio, for free usage. Hugging Face and Alama for local usage. I'll be using it through Alama. Let's get it installed first, then we'll check it out. Make sure you have Alama installed. If you don't, go to Alama's site, click on the download button, choose your operating system, and install it. Once done, go to the models page and click on the Gemma 2 link. We'll need to get both the 9B and 27B models installed because I'll be checking out both of them. First, select the 9B model, copy the command, and paste it into your terminal. This will install the model. Once done, you'll see a chat interface. Send a message and check if it works. After that, go back to the model page and do the same thing for the 27B model. Once done, the models are now installed. Let's check them out. We'll be using these

prompts to test them. We'll first send the prompt to the 9B model, then the 27B model. I'll be using open web UI as the front end. Now, let's start with the first question. What is the number that rhymes with the word we use to describe a tall plant? The answer should be three because it rhymes with tree, or it could be nine because it rhymes with vine. Let's send it to the 9B model first. Okay, it answers correctly. Pretty cool. Now, let's try the 27B model. It does not answer correctly. This is interesting because the smaller 9B model did better on this question. So, the 9B model passes while the 27B model fails. Next, let's ask the second question. I have two apples. Then I buy two more. I bake a pie with two of the apples. After eating half of the pie, how many apples do I have left? The answer should be two. Let's send it to the smaller model first. Okay, it answers correctly. So, this is a pass.

now let's send the same prompt to the bigger model. Okay, it also answers correctly so this one's also a pass. Next, let's move on to the third question. Sally is a girl. She has three brothers. Each of her brothers has the same two sisters. How many sisters does Sally have? The answer should be one. Let's see if it can answer. Let's send the prompt to the smaller model. Okay, it answers incorrectly so this is not a pass. Now let's try the bigger model. It also answers incorrectly, which is really bad, so I don't think it's good at complex prompts. Next, I have a mathematical IC question. If a regular hexagon has a short diagonal of 64, what is its long diagonal? The answer should be 73.9. Let's send it to the smaller model first. Okay, it answers incorrectly so this one's a fail. Now let's try the bigger model. It also fails so it's not good at math either. Now let's try the coding questions. The first one is to create an HTML page with a

button that explodes confetti when you click it. Let's see if it can do it. Okay, I sent the prompt, the code it generates kind of works, but the confetti keeps showing up and doesn't stop. And when I click it again, it keeps growing, which is not good. So this is a fail. Let's try the bigger model. I tried this multiple times, and it just keeps going and never finishes. So this one's also a fail. Next, let's ask it to write a Python function that prints the next 20 leap years. Reply with only the function. Let's send and check. Okay, the smaller model produces a working function, so this is a pass. Now let's try the bigger model. It produces a function that only prints five leap years, which was not what was asked. So this one's a fail. Next, let's give it this prompt: generate the SVG code for a butterfly. Let's see if it can do it. The smaller model's code is extremely bad, just two circles. This one's a fail. Let's try the 27b model. Its code also fails.

next, let's ask it to write a landing page. Given its record, it will probably fail, but let's try. Okay, the code it generates is not usable and very bad. The 27b model also produces unusable code. Next was The Game of Life program. It failed very badly with syntax errors and stuff, so that was also a fail for both models. Now, here's the final chart from my testing. The 9B model is actually better than the 27b model, which is insane. Overall, Gemma can only provide good results in three tests, which is very disappointing. The quen 2 model is much better. I don't know why Google made this model, but let me know your thoughts in the comments. If you liked this video, consider donating to my channel through the super thanks option below. Give this video a thumbs up and subscribe to my channel. I'll see you in the next video. Till then, bye. [Music]

Gemma-2 : Google's NEW Opensource Model can beat Llama-3 & Others in Benchmarks (Fully Tested)

Facing the future with AI! 

Here you'll find content about multiple AI Tools that are actually useful and sometimes free.