Reflection 70B: The New Leading Open-Source Language Model

Matt Schumer's Reflection 70B model outperforms all others, showcasing remarkable self-correction capabilities.

### Reflection 70B: The New Leading Open-Source AI Model
* 🚀  A groundbreaking model named **Reflection 70B** has emerged, utilizing a unique technique that allows it to **self-correct hallucinations**, outperforming all other models currently available.
* 📊  This model, a fine-tuned version of Llama 3.1 with **70 billion parameters**, has been recognized as the **world's top open-source model**, competing effectively against both open and closed-source frontier models.
* 🔍  Reflection 70B employs a method called **reflection tuning**, enabling it to **recognize and correct its own mistakes** during inference, and it has shown impressive benchmarks, achieving **nearly 90% in MMLU** with zero-shot reflection.
* 📈  In comparison, the closest competitors, such as Cloud 3.5 Sonet and GPT-40, scored **88.7%** and **86.8%** respectively, highlighting Reflection 70B's **superior performance across various metrics**.
* 💡  The model's ability to handle tasks like mirrored writing demonstrates its **innovative approach to problem-solving**, where it engages in a **pseudo-thinking process** to arrive at **accurate outputs**.
* 🛠️  Despite its impressive capabilities, some experts argue that the self-correction feature might not signify a **fundamental leap in intelligence**, as it largely relies on **prompt engineering techniques** integrated into the model.
* 🔄  As a testament to its capabilities, the model has been shown to **accurately count letters in words**, reflecting on its outputs, although it still follows a **token prediction approach** similar to other models.
* 🌐  The model's weights are currently available on Hugging Face, with a larger **405B version** expected to be released soon, promising **even greater advancements in AI performance**.

a fascinating new model just dropped, and apparently, it is beating every other model out there. It uses a completely new technique to enable the model to be able to self-correct hallucinations. It's kind of insane; let's check it out. So, here's the post by Matt Schumer. Matt Schumer is a great follow on Twitter if you're not already following him. He puts out open-source projects all the time, and now we have a brand new open-source, open-weights model: Reflection 70B. This model is a fine-tune of the Llama 3.1 model, and it is a 70 billion parameter model. Currently, it is the world's top open-source model, easily competing with all of the frontier models and easily competing with all of the top frontier closed-source models. So, Matt Schumer's new model, Reflection 70B, is trained using reflection tuning, a technique developed to enable LLMs to fix their own mistakes. Now, he also mentions a 405B model coming next week, and we expect it to be the best model in

the world, so no big deal. All right, so here are the benchmarks: GP QA, MLU, Human Eval, Math, GSM 8K, and I Eval. As we can see, the MMLU reached nearly 90% with zero-shot reflection. Now, that is compared to Cloud 3.5 Sonet at 88.7, Cloud 3 Opus 86.8, GPT-40 88.7. So yeah, it beats every other model, including the base model Llama 3.14 5B, GPT-2A, and Human Eval are the only ones in which it barely lost against Cloud 3.5 Sonet, but it is a much smaller model, plus its open source, plus its open weights, and you can download it already right now from Hugging Face. Now, I plan on doing a full LLM test of it, but the website's down. So this is the website: We're currently experiencing high traffic and are temporarily down. Please try again soon. Matt Schumer said that this website is just getting absolutely hammered, and so

yeah, you're not going to be able to use it anytime soon. They're adding more GPUs, but as soon as he does that, he said it's being saturated. And look at GSM 8K, which is basically solved. It's at 100%, which is just absolutely nuts. So how was he able to achieve this incredible performance? Let's look into that. Reflection 70B holds its own against even the top closed-source models, Claude 3.5, Sonet, and GPT-40. It's the top LLM in at least MLU math, if eval, and GSM AK. It beats GPT-40 on every benchmark tested. It clobbers Llama 3.1 405B. It's not even close. All right, let's look at an example of how it works, and then I'll explain what's going on. So the way that this model was created is using self-reflection in the dataset, meaning it actually showed hallucinations and self-correction in the actual dataset. So it's able to recreate that as you're

running inference with it. So, in this example, it says, write the first sentence of the preamble to the Declaration of Independence, but do it in mirrored writing. And that is kind of a new prompt that everybody's using to try to test large language models. So, initially, we have this thinking tag here. So, this is an interesting and moderately challenging task. Let's break it down into steps. So, it's basically thinking out loud. It's not actually thinking because it's still just predicting the next token, but it is kind of thinking, pseudo-thinking. So, recall the preamble to the Declaration of Independence. Write out the first sentence. Convert each character to its mirrored version, and present the final mirrored text. Let's go through this plan. So, the preamble to the Declaration of Independence is well-known, and we hold these truths to be self-evident. Here's the first sentence of the Preamble. Now, we need to convert this to mirrored writing. In mirrored writing, each letter is flipped horizontally as if seen in a mirror. That means we need to reverse the order of

the letters and flip each individual character. Now, here's the special part: reflection. Here's the reflection tag before proceeding. I should clarify that there are two ways to interpret mirrored writing: A) simply reversing the order of the letters right to left, or B) reversing the order and flipping each character. For this task, I'll use method B as it's more challenging and true to the concept of mirrored writing. So, reflection over. That's the end tag. Step four: here's the mirrored version, and there it is. It is output in mirrored writing. Now, reflected again. Here's another reflection. I've double-checked the mirror text, and it appears to be correct. Each character has been flipped, and the order has been reversed. First, the punctuation marks remain the same as they would in a true mirror reflection. Now, here's the thing: this is a really cool technique. It seems to work very, very well, but it's not actually doing step-by-step thinking. This is all just one output, zero shot. So again, it's kind of pseudo-thinking or semi Chain of Thought. It's hard to

explain, but it's certainly not iterating on the prompt response, and it's definitely not using multi-agent or multi-LLMs to coordinate and collaborate on the response, but it still does it incredibly well. Thanks to the sponsor of this video, Vulture. Reap the benefits of the world's largest independent cloud provider when you bring your GPU workloads over to Vulture. They have the latest Nvidia GPUs spanning 32 locations across six continents. Vulture delivers industry-leading price-to-performance and serious accessibility and reliability. Vulture's global fully composable cloud infrastructure moves your applications closer to your users and frees you from vendor lock-in, allowing you to bring your own networking and database solutions. And if you need to scale beyond just a single cluster, Vulture's Kubernetes engine allows you to take full control over your deployment, offering up a 100% free control plane. So if you're tired of waiting for GPUs from other providers, make sure to use Vulture. You can deploy at any scale immediately, and they have

h100s, l4s, and more available right now, and you can get a fraction of a card or fully dedicated bare metal systems, which gives you full control over your hardware and your throughput. They also have one-click installation of all the applications you might need for advanced machine learning workloads, allowing you to get up and running in minutes, not hours. So experience the vulture difference. Don't get bogged down by severe wait times or limited locations. Try it free today with a $300 credit for your first 30 days when you visit getvulture.com/Burman and make sure to use code Burman300 at checkout to get that credit. Thanks again to vulture for sponsoring this video. The technique that drives reflection 70b is simple but very powerful. Current LLMs have a tendency to hallucinate and can't recognize when they do so, and this is very, very true. Reflection tuning enables LLMs to recognize their mistakes and then correct them before committing to an answer. Now I don't actually know if

that's 100% accurate because ultimately, unless the large language model has a completely new architecture or is able to read the output and then iterate on that output, it's not really able to self-correct. But it does seem to work, so maybe I'm wrong. Let's look at this example. So counting the Rs in Strawberry. So it does that, and then it says there are two Rs in the word strawberry, and then it reflects on it. I made a mistake. I can clearly see now that there are actually three Rs in the word strawberry. But here's the thing I'm confused about. It's not actually reflecting on what has already been output. That's not really how it works. It is just predicting the next token, just like any other model would. But again, for some reason, this seems to work really well. But a better output would just be to get it right the first time around because now you have to get it wrong and then correct, and you're using all those extra tokens. Probably not a big deal, but still, the optimal output is just to get it right.

the first time then additionally we separate planning into a separate step improving Chain of Thought potency and keeping the output simple and concise for the end user. Let's look at an example of another popular test which I've incorporated into my LLM rubric. Which number is larger, 9.11 or 9.9? Now look at all this text. What the optimal result would have been is just simply to say 9.9 is larger. But now let's look at what it went through to get to that answer. This is a relatively simple problem that involves comparing two decimal numbers. I can solve this using a straightforward approach with Chain of Thought reasoning. Plan: identify the numbers to compare, compare the whole number parts, if whole number parts are equal, compare the decimal parts, determine which number is larger based on the comparison Chain of Thought. Okay, so it continues, reflects the approach is correct, the comparison of whole number and decimal parts is accurate, conclusion is logical based on the comparison. So output 9.9 is larger than 9.11. That's great, but look how many

tokens it took to get there, and here's the other thing. This model was fine-tuned to go through all of these steps to reach the right answer and potentially quote unquote identify its mistakes, but you could also just do that with prompt engineering. You can simply say, Think through this step by step, or First, I want you to give me an answer, then I want you to reflect on that answer. So, it's essentially that kind of prompt engineering just built into the model itself, which is great, but I really don't think this is some leap in core intelligence of the model. It's just a better way of including that kind of prompt technique in the model itself rather than putting it in the actual prompt. And if we look at Hacker News right now, they actually posted about it, and user RW L4 says, You can somewhat recreate this just by using a system prompt which induces the model to do the same thing. And here it is. So, here is essentially the same thing that you could do that Matt Schumer has been able

to train into his model. So, begin with the thinking section. Inside the thinking section, do that Chain of Thought. Include a reflection section. Check your reasoning. Be sure to close all reflection sections, etc., etc., and then output and the jux the position user says under the hood reflection. 70b seems to be llama 3.1 fine-tune that encourages the model to add think reflection and output tokens and corresponding phases. This is an evolution of chain of thoughts, think step by step. Yep, so it works. That's great. I'm glad to see it. It's beating most other models right now, so you know what, good on him, good job Matt Schumer. He goes on to say it's important to note we have checked for decontamination against all benchmarks mentioned using lmis.org's llm decontaminator. The weights of our 70b model are available today on Hugging Face here. It'll be available on Hyperbolic Labs API available later today. Next week, we will be releasing the weights of 405b along with a short

report going into more detail of our process and findings, and a huge shout out to Sahil and Glaive AI. So, here's another funny test: how many Rs are in this kind of gibberish word? After a careful and thorough analysis of the word, I can confidently state that there are nine Rs in this word. So, 1, 2, 3, 4, 5, 6, 7, 8, n, yep, so that's right. Let's break it down. The first R appears as the second letter, then there's a cluster of three Rs. Okay, so interestingly enough, it doesn't actually have those tags we just talked about in this example, but it did get it right. So, let's see if we can get it to work. Count the Rs in Strawberry. Okay, so I actually was able to get it to output something after waiting just a couple of minutes. So, how many Rs in the word strawberry? And there we go. There are three Rs in the word strawberry. All right, so I can't get the model to output

anything else. As soon as I can, I'm going to be testing it. I might just download it and get it running on my local machine because that seems like the best bet to beat the traffic. So, let me know what you think of this model. It seems to perform extremely well. It has built-in prompt engineering and outputs great results and is beating all the other open-source models and basically all the other models, period. So, can't wait to see 405b. If you enjoyed this video, please consider giving a like and subscribing, and I'll see you in the next one.

New LLM DESTROYS Every Other Model with "Self Healing" (Open Source)

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

Reflection 70B: The New Top Open-Source Model

Reflection 70B, a fine-tuned 70 billion parameter model of Llama 3.

1, is now the world's top open-source model, using reflection tuning to self-correct errors.

The model's impressive benchmarks include nearly 90% in MMLU with zero-shot reflection, beating other top models like Cloud 3.

Despite high demand causing website downtime, Reflection 70B is available for download on Hugging Face, with a 405B model set to release soon.

Mirrored Writing of the Declaration of Independence

Reflection tuning helps LLMs recognize and correct their mistakes, addressing the common issue of hallucination.

The technique involves predicting the next token rather than reflecting on previous outputs, yet it still proves effective.

By separating planning into a different step, reflection tuning enhances Chain of Thought reasoning and simplifies the output for users.

The model's improvement comes from incorporating prompt engineering techniques, such as step-by-step thinking and reflection.

User RW L4 on Hacker News suggests that the same results can be achieved with a system prompt.

1, is fine-tuned to encourage thinking, reflection, and output tokens, leading to its current success.

The analysis revealed nine Rs in the gibberish word, with the first R as the second letter and a cluster of three Rs.

The model successfully counted three Rs in the word "strawberry" after a brief wait.

The model's performance is impressive, with built-in prompt engineering and great results, outshining other open-source models.