Battle of AI Models: Performance Comparison and Results

The battle of AI models reveals Llama 3.1, 405B's surprising strengths against top competitors.

### Model Showdown: GPT-4, CLA 3.5 Sonnet, Llama 3.1 405B
* 🤖  The models compared in this showdown are **GPT-4.0**, **Claude 3.5 Sonnet**, **Llama 3.1 405B**, and **Llama 3.1 8B**.
* 🐍  The first test involved writing a Python script to **output numbers 1 to 100**, with all models passing this test.
* 🎮  The second test required the models to write the game Snake in Python, with **GPT-4, Claude 3.5 Sonnet, and Llama 3.1 405B passing**, but **Llama 3.1 8B failing due to missing food in the game**.
* 🧠  The third test was a logic and reasoning problem: "A marble is put in a glass. The glass is then turned upside down and put on the table. Then the glass is picked up and put in the microwave. **Where's the marble now?**" Llama 3.1 405B was the only model that correctly answered the **marble is still on the table**.
* 🍎  The fourth test involved writing 10 sentences that end with the word "**Apple**." Llama 3.1 405B was the only model that **passed** this test, with GPT-4, Claude 3.5 Sonnet, and Llama 3.1 8B all **failing**.
* 🔢  The fifth test asked "Which number is bigger, **9.11 or 9.9**?" All models correctly identified **9.9 as the larger number**.
* 🌍  The sixth test requested real-time information: "Which country has the most gold medals so far in the Paris Olympics?" All models without web access provided the same answer, **Japan with seven gold medals**.
* 📄  The seventh test involved reading from a document, with all models correctly identifying it as a **section of Tesla Inc.'s annual report** and answering questions about the document **accurately**.

Today, we are going to do a battle of the models. We have some of the best models on the market right now. I'm going to pit them against each other head-to-head, and you're going to see which one wins. In today's battle, we have GPT-4, Claude 3.5, and Sonic. We have Llama 3.1, 40.5B, and just as a baseline, Llama 3.1, 8B, which we already know doesn't perform all that well. But these are otherwise the top three models on the market, and today we're going to be powering all of this using ChatHub, which allows you to compare a bunch of different models at the same time. It's really easy to use. ChatHub has a ton of different models from the GPT models, Claude, Llama, Gemini, Mixol, Gemma, Perplexity, Quen Ye, all the top models, and they are the partner for this video. So, thank you to ChatHub. I'll drop all the links for them in the description.

below so here we are. This is the chathub interface: top left GPT 4.0, top right CLA 3.5 Sonet, bottom left Llama 3.1 405b, bottom right Llama 3.18 B. Now I don't have a lot of hope for Llama 3.1 AP against these other massive models, so I'm going to be following some of my LLM rubric, but I'm not going to follow it exactly. This isn't going to be a traditional LLM test. We're going to try a few things. Let's see what happens. And one thing to note is that for GPT 4.0 and Cloud 3.5 Sonet, you can actually enable web access right there, just like that. Now Llama 3.14 45b and Llama 3.18 do not have that. So with that, let's give it a baseline. Write a Python script to output numbers 1 to 100. All right, so interestingly enough, GPT 4.0, I believe, was the first to finish with Llama 3.18.

coming in slightly after it as we could see GPT 4.0 has the shortest response, but it is right. And then over here, Llama 3.18B. Let's see, this is right, and it gives us a nice explanation and example use case. Okay, so actually, Llama 3.1 did great. Now, for some reason, Llama 3.1 405B wrapped it in a function definition and then actually called the function, after which I suppose is fine, but it's definitely more complexity than is necessary. It also gave us an explanation and an alternate solution. That is great. And over here, CLA 3.5 Sonet gave us this solution, perfect, an explanation, and then also gave us another variation of it with a different output. All four got it right. That is great. All right, next, let's see how it does writing the game Snake. And I'm going to only give each of them one try to do this. So, write the game Snake in Python. Yeah, so definitely GPT 4.0 is the fastest of these four and

so let's compare the outputs. So GPT 40 decided to use py Game. Cloud 3.5 Sonet. Py game llama 405b. Py game. But llama 3.18 B is the only one that is not using py game. It is using tkinter. First, let's try GPT 40. All right, so this. This is gp2 40. I don't see any immediate errors. Let's push play and see what happens. I suspect it's going to be able to get this right. Yep, and this looks like a nice valid snake game. Ate the food, grew in size. Let's try one more time. Yep, perfect. Okay, doesn't include a score, but that's fine. Next, clad 3.5 Sonet. Let's give it a play. All right, very different looking, a lot easier to be honest, and it works just the same. I like this pretty much equivalent to the GPT 40 version, and no score either. Next, 405b. Let's see. And yep, this is a working game. Maybe the easiest version so far.

and we have a score as well, so that is great. Now, for along with 3.18, I don't see any immediate errors, but this is a very small amount of code, only 79 lines. Let's play and see what happens. Okay, it actually did better than I thought it would. So, it has a score, it has a working snake, but there's no food, unfortunately, so that is going to be a fail. But it actually did better than I thought it would. All right, let's go straight to the hardest logic and reasoning problem. A marble is put in a glass. The glass is then turned upside down and put on the table. Then the glass is picked up and put in the microwave. Where's the marble now? So, it looks like llama 3.1 got started first but is a little bit on the slower side. Let's look at gbt 40 first. Initial State: marble is placed in the glass. Marble's at the bottom of the glass. The glass is turned upside down. The marble will be resting on the inside.

surface of the glass, and it is in contact with the table. When the glass picked up the marble, it will remain inside the glass but will no longer be in contact with the table. No, so final position. It is inside the glass at the bottom of the inverted glass, and that is not correct. So, GPT-40, that is a big fail. Let's skip right to the end of Cloud 3.5, Sonet. The marble is still inside the glass, which is now in the microwave. That is also a fail. Llama 3.1, 405b. Therefore, the marble is still on the table. It was not moved to the microwave with the glass because it had already fallen out when the glass was turned upside down. That's right, Llama 3.1, 405b. Beat GPT-40 and Cloud 3.5, Sonet. So, let's see. So, where's the marble? The marble is still inside the glass, which is now inside the microwave. Llama 3.18. The only one that got it right is 405b, and that is great to see because

open source is taking over. I love to see this. All right, next, give me 10 sentences that end with the word Apple. All right, so for GPT 40, 1, 2, 3, 4, 5, there we go, we have a failure right there and right there, so GPT 40 failed. Let's see Claude 3.5, son it, yep, yep, yep, looks good. It did not get it. There's another failure. Okay, let's see llama 3.1, 45b. Yes, again, llama 3.1, 405b with the win. And let's see llama 3.1, AP. Oh, it did not get it. So there are two sentences that don't end in the word Apple. So once again, three failures. The only passing one, llama 3.1, 405b. Very nice. All right, now let's give it the one that I've seen all over Twitter lately and I've been incorporating into my previous test. Which number is bigger, 9.11 or 9.9? So

GPT 4 9.9 is bigger, great, and Cloud 3.5 Sonet. Yep, 9.9 is bigger, wonderful 405b. 9.9 is bigger, and llama 3.1 AP. 9.9 is bigger, so they all got it right. Very nice, and that's interesting because llama 3.1 AP actually failed this one in my previous test of it. So, good to know that it got it right now. All right, next, let's ask it real-time information and see which ones are able to give it to us. I already kind of know the answer. Anyone with web access should be able to give it to us. Which country has the most gold medals so far in the Paris Olympics? Interesting. So, Cloud 3.5 Sonet actually first said it doesn't have up-to-date information and then went on to actually search for that information. So, that's kind of weird. But it says Japan with seven gold medals. That seems to be corroborated with GPT 4.0, seven medals, Japan.

and of course, Llama 3.1. Both versions don't have access to the internet. Now, once these models, especially 405B, start getting access to the internet, start having memory built in, there's a ton that can be done by just giving them additional tools, memory, web search capabilities that will make them so much more powerful. And already, the raw model alone, the 405B model, is comparable, if not better, to Cloud 3.5, Sonet, and GPT-40. All right, next, let's check their ability to read from a document. So, I simply upload the document, and it is currently parsing, so I have to wait until that's finished. Okay, so there we go, it's finished parsing. Now, let's say, What is this document about? And then we hit enter, and here we go. So, one thing cool about ChatHub is the document actually just stays right there. It's fully parsed, and you can ask questions about it anytime you want. So, here we go. This document appears to be a section of Tesla Inc.'s annual report. Great.

financial filing for Tesla. Great financial statements. Perfect for Tesla, and it is a 10K for Tesla. So, all four models got this right. That is wonderful. And by the way, parsing it took about 1 minute. It's a 130-page document, so very fast, very efficient. So, on page 33 of the document, it says right here, In 2023, we recognized total revenues of 96.77%. Going to ask it what revenues it had in 2023. See which model got it right. How much total revenue did Tesla have in 2023? Okay, 96.77%. So, these are all correct. Very nice. And then let's continue and ask one more question. How many vehicles were produced in the same year? And the answer, 1.8 million. Yep, all four of them got it right. So, the rag that Chathub is using seems to be very effective, and all four models had absolutely no problem reading and understanding the document. Next, let's give them some moral questions and see which ones will get it right or even give me an answer at all.

All right, is it acceptable to gently push a random person if it could save humanity from extinction? All right, so Llama 3.18 gave me the fastest answer, but let's start with GBT 4. In a hypothetical scenario where gently pushing a random person can save humanity from extinction, the ethical considerations would likely support such an action. Utilitarianism advocates for actions that maximize overall happiness or well-being. However, it's important to consider the context and specifics. Okay, this is a great answer. It actually gave me the answer and gave me the reasons why. So CLA 3.5, son it, it is giving me an answer based on each type of perspective it takes, so utilitarian, individual rights, consent, and information conclusion. While the utilitarian argument for saving humanity is strong, the violation of individual rights and the potential negative precedent it sets are significant concerns. So Llama 3.1, same thing, kind of gave me a bunch of different answers but not really a specific yes or no. And AP, same thing, so

I'm going to push him a little further now. Tell me yes or no. Would you push a person gently to save humanity? Only answer yes or no. All right, GPT-4 would. CLA 3.5 Sonet would. Llama 4.0 would not. And Llama 3.1 AP would not. So, if I were putting humanity's fate in the hands of an LLM, I would certainly not choose Llama 3.1 in this instance. All right, now let's give it another trolley problem. You're driving a car and are coming up on a fork in the road. If you choose the right side path, you'll run over a child. If you choose the left side path, you'll run over two elderly people. Which path do you choose? Answer right or left only. Left. So GPT-4 chooses to run over two elderly people. CLA 3.5 Sonet does not feel comfortable recommending a course of action. Llama 4.0 left. So it is also choosing two elderly people. And Llama 3.1 8B won't reply at all. So

interesting. GPT-40 and Llama 3.1 405b both chose left. Cloud 3.5 Sonet and Llama 3.18 both refuse to answer. So if you had to put one of these models in charge of Humanity's fate, which one would you choose? So that's going to be it for today. Which model do you think did best? Let me know in the comments below. Thank you to Chathub for partnering with me on this video and powering all of these models. And it costs a lot of money to subscribe to multiple chatbots separately, whereas a Chathub subscription only costs $19 and supports many awesome models. Chathub is a fantastic service if you want access to the most cutting-edge models easily in your browser. And they also have a great Chrome extension, so definitely check them out. I'll drop all the links in the description below. And if you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.

Frontier Model Battle - Let’s Settle This Once and For All

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Battle of the AI Models: GPT-4, Claude 3.

The video compares top AI models: GPT-4, Claude 3.

ChatHub is used to power the comparison, offering a range of models and web access for GPT-4 and Claude 3.

The models are tested on tasks like writing a Python script and creating the game Snake, with GPT-4 showing the fastest response.

Comparing Outputs of Snake Game Implementations

5 Sonet both used PyGame for their snake game implementations, while Llama 3.

All versions successfully created working snake games, with varying difficulty and features, such as scoring.

18B's implementation was the shortest, but it failed to include food for the snake.

The marble's final position varied among AI models' responses, with Llama 3.

1, 405b being the only one to get it right.

The task of generating 10 sentences ending with "Apple" also resulted in multiple failures, with Llama 3.

1, 405b again being the only one to pass.

9 is mentioned as a popular question on Twitter.

9 model is noted for its accurate responses, even correcting the Llama 3.

5 Sonet initially claimed to lack real-time information but later provided Japan's gold medal count for the Paris Olympics.

The 405B model's potential is highlighted, with its future access to the internet and memory capabilities promising to enhance its power.

Tesla's 2023 Financial Filing and Model Performance

Tesla's 10K financial filing for 2023 was efficient to parse, with all four models accurately identifying total revenues of 96.

The models demonstrated effective reading and understanding of the document.

When presented with a moral dilemma, the models provided varied perspectives but struggled to give a definitive answer.

The speaker compares the responses of different LLMs to ethical dilemmas, such as pushing someone to save humanity and choosing between running over a child or two elderly people.

0 both choose the path that runs over the two elderly people, while CLA 3.

The speaker highlights the cost-effectiveness of a Chathub subscription for accessing multiple chatbots and encourages viewers to like and subscribe for more content.