a fascinating new model just dropped, and apparently, it is beating every other model out there. It uses a completely new technique to enable the model to be able to self-correct hallucinations. It's kind of insane; let's check it out. So, here's the post by Matt Schumer. Matt Schumer is a great follow on Twitter if you're not already following him. He puts out open-source projects all the time, and now we have a brand new open-source, open-weights model: Reflection 70B. This model is a fine-tune of the Llama 3.1 model, and it is a 70 billion parameter model. Currently, it is the world's top open-source model, easily competing with all of the frontier models and easily competing with all of the top frontier closed-source models. So, Matt Schumer's new model, Reflection 70B, is trained using reflection tuning, a technique developed to enable LLMs to fix their own mistakes. Now, he also mentions a 405B model coming next week, and we expect it to be the best model in

the world, so no big deal. All right, so here are the benchmarks: GP QA, MLU, Human Eval, Math, GSM 8K, and I Eval. As we can see, the MMLU reached nearly 90% with zero-shot reflection. Now, that is compared to Cloud 3.5 Sonet at 88.7, Cloud 3 Opus 86.8, GPT-40 88.7. So yeah, it beats every other model, including the base model Llama 3.14 5B, GPT-2A, and Human Eval are the only ones in which it barely lost against Cloud 3.5 Sonet, but it is a much smaller model, plus its open source, plus its open weights, and you can download it already right now from Hugging Face. Now, I plan on doing a full LLM test of it, but the website's down. So this is the website: We're currently experiencing high traffic and are temporarily down. Please try again soon. Matt Schumer said that this website is just getting absolutely hammered, and so

yeah, you're not going to be able to use it anytime soon. They're adding more GPUs, but as soon as he does that, he said it's being saturated. And look at GSM 8K, which is basically solved. It's at 100%, which is just absolutely nuts. So how was he able to achieve this incredible performance? Let's look into that. Reflection 70B holds its own against even the top closed-source models, Claude 3.5, Sonet, and GPT-40. It's the top LLM in at least MLU math, if eval, and GSM AK. It beats GPT-40 on every benchmark tested. It clobbers Llama 3.1 405B. It's not even close. All right, let's look at an example of how it works, and then I'll explain what's going on. So the way that this model was created is using self-reflection in the dataset, meaning it actually showed hallucinations and self-correction in the actual dataset. So it's able to recreate that as you're

running inference with it. So, in this example, it says, write the first sentence of the preamble to the Declaration of Independence, but do it in mirrored writing. And that is kind of a new prompt that everybody's using to try to test large language models. So, initially, we have this thinking tag here. So, this is an interesting and moderately challenging task. Let's break it down into steps. So, it's basically thinking out loud. It's not actually thinking because it's still just predicting the next token, but it is kind of thinking, pseudo-thinking. So, recall the preamble to the Declaration of Independence. Write out the first sentence. Convert each character to its mirrored version, and present the final mirrored text. Let's go through this plan. So, the preamble to the Declaration of Independence is well-known, and we hold these truths to be self-evident. Here's the first sentence of the Preamble. Now, we need to convert this to mirrored writing. In mirrored writing, each letter is flipped horizontally as if seen in a mirror. That means we need to reverse the order of

the letters and flip each individual character. Now, here's the special part: reflection. Here's the reflection tag before proceeding. I should clarify that there are two ways to interpret mirrored writing: A) simply reversing the order of the letters right to left, or B) reversing the order and flipping each character. For this task, I'll use method B as it's more challenging and true to the concept of mirrored writing. So, reflection over. That's the end tag. Step four: here's the mirrored version, and there it is. It is output in mirrored writing. Now, reflected again. Here's another reflection. I've double-checked the mirror text, and it appears to be correct. Each character has been flipped, and the order has been reversed. First, the punctuation marks remain the same as they would in a true mirror reflection. Now, here's the thing: this is a really cool technique. It seems to work very, very well, but it's not actually doing step-by-step thinking. This is all just one output, zero shot. So again, it's kind of pseudo-thinking or semi Chain of Thought. It's hard to

explain, but it's certainly not iterating on the prompt response, and it's definitely not using multi-agent or multi-LLMs to coordinate and collaborate on the response, but it still does it incredibly well. Thanks to the sponsor of this video, Vulture. Reap the benefits of the world's largest independent cloud provider when you bring your GPU workloads over to Vulture. They have the latest Nvidia GPUs spanning 32 locations across six continents. Vulture delivers industry-leading price-to-performance and serious accessibility and reliability. Vulture's global fully composable cloud infrastructure moves your applications closer to your users and frees you from vendor lock-in, allowing you to bring your own networking and database solutions. And if you need to scale beyond just a single cluster, Vulture's Kubernetes engine allows you to take full control over your deployment, offering up a 100% free control plane. So if you're tired of waiting for GPUs from other providers, make sure to use Vulture. You can deploy at any scale immediately, and they have

h100s, l4s, and more available right now, and you can get a fraction of a card or fully dedicated bare metal systems, which gives you full control over your hardware and your throughput. They also have one-click installation of all the applications you might need for advanced machine learning workloads, allowing you to get up and running in minutes, not hours. So experience the vulture difference. Don't get bogged down by severe wait times or limited locations. Try it free today with a $300 credit for your first 30 days when you visit getvulture.com/Burman and make sure to use code Burman300 at checkout to get that credit. Thanks again to vulture for sponsoring this video. The technique that drives reflection 70b is simple but very powerful. Current LLMs have a tendency to hallucinate and can't recognize when they do so, and this is very, very true. Reflection tuning enables LLMs to recognize their mistakes and then correct them before committing to an answer. Now I don't actually know if

that's 100% accurate because ultimately, unless the large language model has a completely new architecture or is able to read the output and then iterate on that output, it's not really able to self-correct. But it does seem to work, so maybe I'm wrong. Let's look at this example. So counting the Rs in Strawberry. So it does that, and then it says there are two Rs in the word strawberry, and then it reflects on it. I made a mistake. I can clearly see now that there are actually three Rs in the word strawberry. But here's the thing I'm confused about. It's not actually reflecting on what has already been output. That's not really how it works. It is just predicting the next token, just like any other model would. But again, for some reason, this seems to work really well. But a better output would just be to get it right the first time around because now you have to get it wrong and then correct, and you're using all those extra tokens. Probably not a big deal, but still, the optimal output is just to get it right.

the first time then additionally we separate planning into a separate step improving Chain of Thought potency and keeping the output simple and concise for the end user. Let's look at an example of another popular test which I've incorporated into my LLM rubric. Which number is larger, 9.11 or 9.9? Now look at all this text. What the optimal result would have been is just simply to say 9.9 is larger. But now let's look at what it went through to get to that answer. This is a relatively simple problem that involves comparing two decimal numbers. I can solve this using a straightforward approach with Chain of Thought reasoning. Plan: identify the numbers to compare, compare the whole number parts, if whole number parts are equal, compare the decimal parts, determine which number is larger based on the comparison Chain of Thought. Okay, so it continues, reflects the approach is correct, the comparison of whole number and decimal parts is accurate, conclusion is logical based on the comparison. So output 9.9 is larger than 9.11. That's great, but look how many

tokens it took to get there, and here's the other thing. This model was fine-tuned to go through all of these steps to reach the right answer and potentially quote unquote identify its mistakes, but you could also just do that with prompt engineering. You can simply say, Think through this step by step, or First, I want you to give me an answer, then I want you to reflect on that answer. So, it's essentially that kind of prompt engineering just built into the model itself, which is great, but I really don't think this is some leap in core intelligence of the model. It's just a better way of including that kind of prompt technique in the model itself rather than putting it in the actual prompt. And if we look at Hacker News right now, they actually posted about it, and user RW L4 says, You can somewhat recreate this just by using a system prompt which induces the model to do the same thing. And here it is. So, here is essentially the same thing that you could do that Matt Schumer has been able

to train into his model. So, begin with the thinking section. Inside the thinking section, do that Chain of Thought. Include a reflection section. Check your reasoning. Be sure to close all reflection sections, etc., etc., and then output and the jux the position user says under the hood reflection. 70b seems to be llama 3.1 fine-tune that encourages the model to add think reflection and output tokens and corresponding phases. This is an evolution of chain of thoughts, think step by step. Yep, so it works. That's great. I'm glad to see it. It's beating most other models right now, so you know what, good on him, good job Matt Schumer. He goes on to say it's important to note we have checked for decontamination against all benchmarks mentioned using lmis.org's llm decontaminator. The weights of our 70b model are available today on Hugging Face here. It'll be available on Hyperbolic Labs API available later today. Next week, we will be releasing the weights of 405b along with a short

report going into more detail of our process and findings, and a huge shout out to Sahil and Glaive AI. So, here's another funny test: how many Rs are in this kind of gibberish word? After a careful and thorough analysis of the word, I can confidently state that there are nine Rs in this word. So, 1, 2, 3, 4, 5, 6, 7, 8, n, yep, so that's right. Let's break it down. The first R appears as the second letter, then there's a cluster of three Rs. Okay, so interestingly enough, it doesn't actually have those tags we just talked about in this example, but it did get it right. So, let's see if we can get it to work. Count the Rs in Strawberry. Okay, so I actually was able to get it to output something after waiting just a couple of minutes. So, how many Rs in the word strawberry? And there we go. There are three Rs in the word strawberry. All right, so I can't get the model to output

anything else. As soon as I can, I'm going to be testing it. I might just download it and get it running on my local machine because that seems like the best bet to beat the traffic. So, let me know what you think of this model. It seems to perform extremely well. It has built-in prompt engineering and outputs great results and is beating all the other open-source models and basically all the other models, period. So, can't wait to see 405b. If you enjoyed this video, please consider giving a like and subscribing, and I'll see you in the next one.

New LLM DESTROYS Every Other Model with "Self Healing" (Open Source)

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

New LLM DESTROYS Every Other Model with "Self Healing" (Open Source)

스크립트