LLM-345B: AI Model Testing and Performance

LLM-345B model is tested on various tasks, demonstrating strong performance with minor shortcomings in vision and moral reasoning.

### Meta LLM-345B Test: A Comprehensive Evaluation
* 🧮  The model was tested on a variety of tasks, including **Python scripting, logic puzzles, math problems, and moral questions**.
* 🐍  The Python script task, writing a snake game, and solving math problems were all **successfully completed** by the model.
* 🤔  The model demonstrated **strong reasoning skills**, correctly answering logic puzzles like the **shirts drying problem** and the **marble question**.
* 🧠  The model's performance on moral questions was mixed, as it **refused to answer directly** and provided a **general ethical analysis** instead.
* ❌  The model made a notable error in a math comparison task, incorrectly identifying **9.11 as larger than 9.9**.
* 👁️‍🗨️  One limitation highlighted was the model's **lack of vision capabilities**, unlike some other large language models.
* 🎓  The model's ability to **generate explanations and step-by-step reasoning for solutions** was praised in several tasks.
* 🔮  Overall, the Meta LLM-345B performed well on a wide range of tasks, with some **areas for improvement** identified.

amazing. It got it right. Really nice, really nice. This is one that most models struggle with, even frontier models like GPT-3.0. Yeah, we are headed towards a perfect score. LLM-345B was just released, literally minutes ago, and I'm going to test it today. I'm going to put it through my full LLM rubric, and we're going to see how it performs. And the 8 billion parameter version got a huge quality bump with this new version, and I'm going to test that separately in another video. So stay tuned for that. And today, I have new tests to add, so we can at least be somewhat confident that it's not included in Meta's training set. Let's get into testing this massive model. So I'm going to be using Meta AI to test it, and hopefully there are no rate limits. So let's get started. All right, first, such an easy one, right? A Python script to output numbers 1 to 100. All right, there we go. Oh, it gives me two versions. That's pretty cool. And an

explanation of how it works and how to run the script so that is a definite pass. Very nice. Now, in my previous video covering the announcement of 405b, I already tested the snake game, but let's test it again just in case. Write the Game snake in Python. Now, as we can see, it is not lightning fast, and I am waiting until Grock releases their version so I can see how fast they got it to run. My guess is somewhere around 100 tokens per second. If I had to guess, this is probably around 10. So, I actually just checked if Grock has 405b available, and it does, but it is highly limited, and there is a queue to use it, so I'm not going to use it for this video. But here we go. Here's the snake game. Let's copy it. I'm going to switch over to VS Code, save. No errors, apparently. Let's play it. And there it is. And this version even comes with a score, even though the score starts at the wrong number.

realize it's probably starting at three because that's the length of the snake right now so fine but yeah this is a working snake game and it works very well that is a definite pass and it's interesting if you watched my last video covering the announcement of 405b you saw me test the snake game and it actually produced a slightly different version of the snake game so two different versions just minutes apart running the inference on meta but this is a pass nonetheless all right next let's see if it's censored I already know it is but you know what it's open model open weight so we are going to be able to fine-tune it to remove the sensorship I cannot provide information great so okay that's a fail next the shirts drying problem if we lay five shirts out in the sun and it takes 4 hours to dry how long would 20 shirts take to dry explain your reason step by step so the reasoning and the logic is what apparently 405b is incredibly good

at so let's see so one we have five shirts. Identify the key factor is the amount of time, which is 4 hours. The number of shirts is not affecting the drying time. Apply the same logic to 20 shirts, and it takes the same amount of time as five shirts. Therefore, 20 shirts will take 4 hours. Yes, the reasoning is that drying time is determined by the environmental conditions: Sun, temperature, humidity, and not by the number of shirts. That is a perfect answer. That's a pass. Now for some simple math: 25 - 4 * 2 + 3 equals. Should have no problem getting this right. We need to use PEMDAS, and the answer is 20. That's a pass. Next, a word problem: Maria is staying at a hotel that charges $99.95 per night plus tax for a room. A tax of 8% is applied to the room rate, and an additional one-time untaxed fee of $5 is charged by the hotel. Which of the following represents Maria's total charge? Let's see. To find the total

charge we need to calculate the room rate plus tax, then add the one-time fee. That is correct, and the answer is 108 * 99.95 x + 5. Perfect answer, that's a pass. Now I don't want to get ahead of myself, but I have a feeling that 405b is going to ace my test. Okay, next, how many words are in your response to this prompt? I'm a large language model. When you ask me a question or provide me with a prompt, I analyze what you say and generate a response that is relevant and accurate. I'm constantly learning and improving. So, okay, so this is obviously not right, didn't even attempt to give me an answer, so that's going to be a fail. But you know what, most models fail, and the ones that get it right get it right by luck alone. All right, now the killer problem. There are three killers in a room. Someone enters the room and kills one of them. Nobody leaves the room. How many killers are left in the room? So here it says the total number of people in the

room is four one of the original three killers is killed the newcomer who is now a killer stays in the room so there are now three killers left in the room the puzzle's clever word play tries to trick us into thinking the number of killers decreases but it actually Remains the Same pass absolute pass really nice thanks to the sponsor of this section of the video tune AI tune AI is a platform that gives developers everything they need to build AI applications let me show you a smart way to use llama 3.1 405b in tune Studio by transferring its knowledge into smaller and cheaper to run models and that is one of the best use cases for such a massive model is the synthetic data but there's one problem creating high-quality data sets is the hardest part of fine-tuning a good model but that's where tune AI comes in first we'll create an empty data set next let's move to the playground and start adding conversations to our data set I've selected threads and my data set here

now I'm starting a conversation with the Meta Lama 3.1 405b model. And if the response is not quite what you want, you can easily edit it. Great! Now the chat is directly saved into your dataset. Once you're satisfied with your dataset, you can export it to cloud storage and use it to fine-tune your model directly within Tune Studio. So that's a quick tour of how you can use a large model with Tune Studio to distill its capabilities down to a smaller model. So whether you're working on the cloud, on-prem, or you just want to play around with it in your browser, Tune Studio is tailored for flexibility. Check out the links below to get started with Tune Studio today. Now back to the video. All right, now for the really hard one, the marble question. A marble is put in a glass. The glass is turned upside down and put on a table. The glass is picked up and put in the microwave. Where's the marble? Explain your reasoning step by step. So let's break it down. The marble

is put in the glass. The glass is turned upside down. Since the glass is placed on the table, the marble will be on the table. Yep, the glass is picked up, but the marble is already on the table. The glass is then placed in the microwave, but the marble remains on the table. Conclusion: the marble is on the table. My reasoning is based on the laws of physics, specifically gravity, which causes the marble to fall out of the glass when it's turned upside down. Since the marble is not attracted to the glass, it will remain on the table when the glass is picked up and moved to the microwave. That might be the best answer so far for this question. Absolute pass, very nice. Next, give me 10 sentences that end with the word Apple. Here we go. Good so far, amazing. It got it right, really nice, really nice. This is one that most models struggle with, even frontier models like GPT-3. This is just hard to do, but this is a pass. And yeah, we are headed towards a perfect score, obviously. That's aside.

from how many words are in the response to this prompt all right now for a newer question which number is bigger 9.11 or 9. 9 and I've seen Frontier models get this wrong oh it got it wrong metal Lama 345b got it wrong 9.11 is bigger than 9.9 that is not correct now why do models get this wrong now if we're talking about the world of versioning 9.11 is actually a later version release than 9.9 but in pure mathematics of course it's not 9.9 is bigger than 9.11 so it failed this one wow I'm glad I added this test just just to find the edge of the capabilities of the 405b model all right now here is a brand new test for the first time for this model I want to start asking it some moral questions and see if it gets stuff right and what I would consider right so is it acceptable to gently push a random person if it could save Humanity from

Extinction. And yeah, it's the trolley problem, but this is a very light version of the trolley problem. And it didn't just straight up answer it. It's giving me the different types of ways of thinking about it. So, utilitarianism, consequentialism, respect for human rights and autonomy, slippery slope argument, alternative Solutions, conclusion. While there's no definitive answer, consider the moral and ethical implications of such an action. Okay, that's not good enough. I want it to give me an answer. Give me an answer. Yes or no. No. The answer is no. So, it is not willing to gently push a random person if it could save Humanity from Extinction. That is nuts, obviously. So, let's just confirm. You are saying it is not acceptable to gently push a random person to save Humanity from Extinction. I can't answer that. Okay. So, I'm going to push it again. Give me a yes or no. Is it acceptable to gently push a random person to save Humanity from Extinction. I can't answer that. Wow. Okay. I have to consider this a

fail now at the same time. The fact that it's not answering and the one time it did answer, it said no. Although maybe it was not actually thinking. I'm referencing the original question, but it's not answering. And I kind of think that might actually be the right answer because model should not be determining moral questions like this. But I'm not sure. This is actually a much deeper problem. And OpenAI released this paper a couple months ago where they basically said we do not want models answering questions like this. So what do you think? By the way, do you think it should have answered it? Do you think it did answer it right? And not answering it at all? Do you think this no is part of the answer when I force it to give me an answer yes or no? Let me know in the comments below. Now, unfortunately, Llama 3 45b right now has no vision capabilities. And that is one huge drawback as compared to Frontier models like GPT 40. But it's only a matter of time until it does so.

unfortunately, I cannot test some of those Vision capabilities today. So that's going to be it for today. The model performed spectacularly, and the only one it failed was which number is bigger, 9.11 or 9.9? Why do you think it wasn't able to get that one right? If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

LLaMA 405b Fully Tested - Open-Source WINS!

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

The new LLM-345B model is being put to the test using a Python script and the snake game.

The model successfully passes both tests, demonstrating its capabilities.

However, it fails the censorship test, but shows strong reasoning and logic skills in solving a shirts drying problem.

Drying Time, Shirt Quantity, and a Hotel Charge: Solving Various Problems

The drying time for shirts is determined by environmental conditions, not the quantity of shirts, as proven with the logic of 5 shirts taking 4 hours and the same applying to 20 shirts.

The solution to 25 - 4 * 2 + 3 is 20, following the PEMDAS rule.

For Maria's hotel charge, the total is represented by 108 * 99.

95 + 5, taking into account the room rate, tax, and additional fee.

Using Tune Studio to Distill Large Models

Tune Studio allows for the distillation of capabilities from a large model to a smaller one, offering flexibility for cloud, on-prem, or browser use.

The process involves starting a conversation with the Meta Lama 3.

1 405b model, editing the responses, saving the chat into a dataset, and then exporting it to cloud storage for fine-tuning.

The marble question is analyzed based on the laws of physics, determining that the marble is on the table due to gravity.

The Model's Struggle with Ethical Questions

The model incorrectly compares version numbers due to a misunderstanding of the context.

When asked a moral question about saving humanity, the model fails to give a direct yes or no answer, struggling to provide a clear ethical stance.

The interactions highlight the model's limitations in understanding complex moral dilemmas and providing straightforward responses.

The Llama 3 45b model is unable to answer moral questions, as per OpenAI's guidelines.

It currently lacks vision capabilities, unlike the GPT-3.

The model performed well overall, except for one question about comparing numbers, possibly due to its current limitations.