Meta's Llama 3.1: Open-Source 405 Billion-Parameter Model

Llama 3.1 405b, an open-source model, rivals top closed-source models, marking a pivotal day in AI history.

### Meta Releases Llama 3.1: 405 Billion-Parameter Open-Source Model
* 🚀  Meta introduces Llama 3.1, a **405 billion-parameter model**, described as "**the most sophisticated open-source model**" and capable of competing with and often **outperforming GPT-40**.
* 💡  The release is a significant milestone, as it marks open source catching up with frontier closed-source models, with Meta investing **hundreds of millions of dollars** to train these models and then **releasing them for free**.
* 🌐  Llama 3.1 expands the context length to **128k tokens**, up from 8k, and supports **eight languages**, with the smaller models also receiving a **substantial quality increase**.
* 🔗  The release is part of Meta's broader strategy to build the Llama ecosystem, empowering developers to create **custom agents and behaviors**, and providing security and safety tools like **Llama Guard 3** and **Prompt Guard**.
* 🏆  Llama 3.1 405b outperforms GPT-40 in most benchmarks, with the smaller 8 billion-parameter model seeing **significant improvements, including doubling its human eval score in some cases**.
* 💰  The models are available for **immediate download and development**, with Meta changing the license to allow developers to use the outputs from Llama models to **improve other models**.
* 🛠️  Meta quantizes the models from **16-bit to 8-bit** for large-scale production inference, making it possible to **run the model within a single server node**.
* 🌍  The release is seen as a **pivotal moment in AI history**, democratizing access to cutting-edge models and ensuring that **AI benefits and opportunities are more widely distributed across society**.

the first time we're releasing a 405 billion-parameter model. By far the most sophisticated open-source model that I think anyone has put out. Today is the day that open source catches up to Frontier models once and for all. Llama 3.1 is dropping today. 405 billion parameters of a model that can compete directly and often beat GPT-40. Why is this so important? Well, Mark Zuckerberg and team have been hard at work spending hundreds of millions of dollars to train these models only to give them away for free. The scorched Earth approach. And that's why close-source model companies are shaking in their boots right now. We now have a completely open-source model that is considered state-of-the-art. The code, the weights, and the model will be fully released. And the 405 billion-parameter model isn't the only great news today. All of the models are getting a version bump. That is Llama 3.1. And what I'm going to show you in a bit is

actually, the 8 billion parameter version, the small version, got a huge quality increase, and it's almost flown under the radar because everybody wants to know about the large model, but I think the smaller models might actually be just as impactful. And if you've been watching my recent videos, you know that I'm incredibly confident in models and AI compute in general being pushed to Edge devices. So, the better these small models get, the quicker that will happen. So, in this video, I'm going to tell you all about it. So, here it is, headline llama 3.1, our most capable models to date. Our latest models expand context length through 128k, and that is up from, I believe, 8K, so a huge increase natively. Now, I already talked to the gradient AI team who was able to expand that context window to a million tokens and still keep the quality really high. But now natively, we have a 128k context window, plus support across eight languages, and llama 3.1 405b, the industry-leading open

source Foundation model. I don't even think they need to say open source. I think this is an industry-leading model. Llama 3.1 45b is in a class of its own, unmatched flexibility, control, state-of-the-art capabilities that rival the best closed-source models. Our new model will enable the community to unlock new workflows, such as synthetic data generation and model distillation. This is incredibly important. Nvidia just a few weeks ago released Neaton, which was a massive model that was created for the same thing. These massive models can then train smaller models to be much better, and it does so by creating synthetic data, fake data. And so, think about it like this: a company that doesn't necessarily have billions of dollars of the resources of an OpenAI can actually generate its own data rather than trying to compete in the deals to acquire data like OpenAI is doing. And that's important because everybody has access to the same open web data, and so for a model to really stand out, it needs its own unique data.

and now we have two strong models: Nitron and now 405b that can create synthetic data to train smaller models. But that's not all it does, of course, but that is an interesting facet. And apparently, a big part of this announcement is the fact that they are turning Llama into an ecosystem, not just one-off models that are being released. And I think this is part of the broader scorched Earth strategy that Mark Zuckerberg and The Meta team are employing. We're continuing to build out Llama to be a system by providing more components that work with the model, including a reference system. We want to empower developers with the tools to create their own custom agents and new types of agentic behaviors. So, I cannot wait to plug this into agentic Frameworks like crew Ai. And I wonder if this means that Meta is building their own agent system. We're bolstering this with new security and safety tools, and they're releasing Llama Guard 3 and Prompt Guard to help build responsibly. We're also releasing a request for comment on the Llama stack API.

standard interface. We hope to make it easier for third-party projects to leverage llama models. Now I wonder if the Llama stack API is going to mimic the OpenAI API standard, which has become the standard. So if they're making their own standard, not a fan of that. I really hope they just stick to what's out there so developers can just drop in a replacement to an API anytime they want. The ecosystem is primed and ready to go with over 25 partners: AWS, Nvidia, Databricks, Grock, Dell, Azure, Google Cloud on day one. Gro, yes, I cannot wait to try this massive model on the fastest inference speeds I've seen. So you can try Llama 3.1 405b in the US on WhatsApp and Meta. This is where I will be testing it by asking a challenging math or coding question. Now in this video, I'm not going to run it through my LLM rubric, but I am going to make another video just for that. So stay tuned for that. All right, so now let's read about

3.1 the first openly available model that rivals the top models in AI when it comes to state-of-the-art capabilities in general knowledge steerability math tool use and multilingual translation. This is such an important day. This is the day in which open source might catch up for good. I actually thought open source would be about 3 to 6 months behind state-of-the-art frontier closed-source models for a while, but it turns out I was wrong, and open source is here. And in fact, this is the leaked benchmark of the new llama models. So what we see here is llama 3.3.1 405b, and here are all the scores against all these different benchmarks. Then we have GPT-40, the frontier model from OpenAI. Now, obviously, they're probably working on 4.5 and 5 right now, but still as of this time, look at these scores. Nearly across the board, llama 3.3.1 405b, an open-source, open weights model, is beating GPT-40. There are only a couple of benchmarks.

in which llama 3 actually doesn't do as well, but it's close. Now they're also comparing the previous llama 3 versions of 70b and 8B to 3.1 version, their new version. And for the 70 billion parameter version, across the board, they scored better. So, an improvement, not a massive improvement, but still an improvement nonetheless. Now here's where it gets exciting. The small model here is llama 38b on this column, and here is 3.1 now. Now if you're looking at this, you should be blown away. It is a substantial improvement with 3.1 over three. Look at these scores. Some of them are massive. So here's hella swag. It goes from 46 to 67 on human eval, one of the most important benchmarks. 34 to 68. That is double. They literally doubled their human eval score and so on. And you can see just across the board, a massive improvement. Now meta llama 3.18 is the one that you will be

running locally on your computer. So the fact that this had such a massive improvement, I'm super excited about. Now think about this: you can run this on your computer essentially free. And then just yesterday, I released a tutorial about RoBERTa. So if you're a developer or if you're just wanting to set up a really powerful local AI system, you can plug this into RoBERTa, have it answer essentially 95 to 98% of all of your prompts, and then only when necessary, send it off to GPT-3 for the massive model. So really exciting. And again, if you have this running on your computer locally, you could do it completely offline. So if you lose internet or anything else happens, as long as you have electricity or battery power, you can run these models. Now before I move on, I want to show something that Bill Gurley, the famous venture capitalist who has invested in numerous multi-billion dollar tech companies, has said. Now, if you've watched my channel, I've mentioned Bill Gurley multiple times in the past.

he did an incredible talk about regulatory capture, which is when companies who are ahead but aren't looking to innovate go to Washington DC and essentially convince lawmakers to implement regulation that makes it easier for them to keep control and harder for startups and new companies to compete. Now look what he says here for those that wonder why the closed AI model exist, open AI, anthropic, and their investors spend so much time in Washington DC. It is because they feared this day would come when open source becomes the market leader. Glad the FTC and Lena Khan FTC are paying attention. And then he goes on to dig at Scott Weiner, who is a California state Senator. Too bad that Scott Weiner doesn't realize what is happening. He is a shill for big business and doesn't know it, working to make California more bureaucratic, less innovative, and less competitive. Okay, enough of the politics. I've had enough of that this week. Let's get back to llama 3. Okay, now as part of the latest release, we're introducing

upgraded versions of the 8 billion parameter and 70 billion parameter models. These are multilingual and have significantly longer context length of 128k. State-of-the-art tool use, which by the way, Grock just released. Really powerful tool use models, better than actually anything on the market, including closed source on Grock. But I think now they're actually going to have to do that again with 3.1 and overall stronger reasoning capabilities. This enables our latest models to support advanced use cases such as long-form text summarization, multilingual conversational agents, and coding assistance, one that I'm especially excited for. We've also made changes to our license, allowing developers to use the outputs from Llama models, including 405b, to improve other models. I mean, I am so happy with what Meta is doing, and if you were to ask me even two years ago if I would have this attitude towards Meta, I would have never guessed it. True to our commitment to open source, starting today, we're making these models available for the community to download.

on l.a. comom and available for immediate development on our broad ecosystem of partner platforms. They evaluated performance on 150 Benchmark data sets that span a wide range of languages, and we found that our Flagship model is competitive with leading Foundation models across a range of tasks, including GPT 4 40 and Cloud 3.5 Sonet. The current best model that I've ever used Cloud 3.5 Sonic. Additionally, our smaller models are competitive with closed and open-source models that have a similar number of parameters. So llama 3.1 405b was trained on over 15 trillion tokens, and they trained it on, listen to this, 16,000 h100 gpus, making the 405b the first llama model trained at this scale. So there are only a handful of companies that could have pulled this off, and apparently there's only one who is going to release it for free, and that's meta. So I am just so so pleased with what meta is doing, and I think we

just have to give a little bit of credit to Mark Zuckerberg and his newfound effit attitude. So to support large-scale production inference for a model at the scale of 405b. By the way, we're not going to be able to run this locally, most likely, even if they completely quantize it. I just don't think it's going to be able to run locally. And even though I have 96 GB of vram, I don't think I'm going to be able to run it. But you know, I'm going to try. We quantize our models from 16-bit to 8-bit, effectively lowering the compute requirements needed and allowing the model to run within a single server node. In post-training, we produce final chat models by doing several rounds of alignment on top of the pre-trained model. Each round involves supervised fine-tuning, rejection sampling, direct preference optimization. If you don't know what these are, these are all techniques to get more out of the model. We use synthetic data generation to produce the

vast majority of our sftt examples iterating multiple times to produce higher and higher quality synthetic data across all capabilities. I wonder if they use the output from 405b, the synthetic data created by it, to help improve the 8 billion parameter model. So if you go to meta AI, you can use the brand new llama 3 405b Model right now. So again, like I mentioned, this isn't going to be a full test. I'll do that in a separate video, but I just want to show you it working really quickly. So write the Game snake in Python so it's pretty fast. This looks to be maybe about 15 tokens per second. It's usable, but obviously what I'm waiting for is grock, and there it is. So I'm just going to try it out. Let's see if it works. Copied the code, let's save the code. I don't see any immediate errors, which is good. Let's play, and there's the Snake Game. There it is, first go llama 3.1 405b, and it seems to work.

perfectly, yep, awesome. Let's see if we can go through the wall and it restarts. Okay, so this is a great implementation of Snake. So, they're also releasing something called The Llama stack. The implementation of components in this Llama system is still fragmented. That's why we've started working with industry startups and the broader community to help better define the interfaces of these components. Now, why is this important? Mark Zuckerberg is not a dummy. He is not just going to release all of this stuff for free without the obvious business value. And the business value is: release it all for free, and then everybody's going to want to build on top of your ecosystem. And when it's your ecosystem, you get to define the standards. And when you bring developers into your ecosystem, well, you've seen what happens over and over again, such as when the smartphone revolution happened and everyone started to build on top of iOS. Apple became the most valuable company in the world. So, to support this, we're releasing a request for comment on GitHub for what we're calling the Llama.

stack llama stack is a set of standardized and opinionated interfaces for how to build canonical tool chain components. Fine-tuning synthetic data generation and agentic applications. Our hope is to have these become adopted across the ecosystem, which should help with easier interoperability. Unlike Clos models, llama model weights are available to download. Developers can fully customize the models for their needs and applications, train on new data sets, conduct additional fine-tuning. While many argue that closed models are more cost effective, llama models offer some of the lowest cost per token in the industry, according to testing by artificial analysis. As Mark Zuckerberg noted, open source will ensure that more people around the world have access to the benefits and opportunities of AI, that power isn't concentrated in the hands of a small few, and that the technology can be deployed more evenly and safely across Society. So no matter what their true motives are, I'm very happy that they're releasing all of this open source. So it seems like the Llama stack or the Llama ecosystem is now going to include real-time and batch

inference, supervised fine-tuning, evaluations, continual pre-training, rag function calling, and synthetic data generation. They've worked with VM tensor RT and Pie torch to build support from day one to ensure the community is ready for production deployment. So that's it. I cannot wait to test this out, and in fact, I'm going to right after this. Let me know what you think about this in the comments. I think today is going to be known as a pivotal day in the history of artificial intelligence, the day that everybody has access to the most cutting-edge models any way that they want to. Open source, open weights. Great job to Meta. I'm going to be testing this out. I'm also going to be testing the 8 billion parameter version, which got a huge bump, and I will directly compare that to the results of the previous test that I did of it. And a quick reminder, there are only a few days left to enter to win this Asus via co-pilot plus PC. I'm giving this one away, and all you have to do is subscribe to my newsletter.

Check out the link in the description to find out. How? See you in the next one.

BREAKING: LLaMA 405b is here! Open-source is now FRONTIER!

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

1: A Groundbreaking 405 Billion-Parameter Open-Source Model

1, a 405 billion-parameter model, is being released as open source, shaking up the close-source model companies.

The new model offers unmatched flexibility, control, and state-of-the-art capabilities, allowing the community to explore new workflows such as synthetic data generation and model distillation.

The release also includes a significant quality increase in the 8 billion parameter version, which is expected to expedite the use of AI models in Edge devices.

Llama is expanding into an ecosystem, offering synthetic data generation and a full stack API.

1 405b model is openly available and rivals top AI models in general knowledge, steerability, math use, and multilingual translation.

It has also surpassed the GPT-40 frontier model in various benchmarks, marking a significant achievement for open source AI.

The Llama ecosystem has already gained over 25 partners and is set to be tested on WhatsApp and Meta for its fast inference speeds.

1 shows a significant improvement in performance compared to the previous versions, especially in the 38 billion parameter model.

18 brings the massive improvement to a model that can be run locally, offering an exciting opportunity for developers to use a powerful AI system offline.

The discussion also includes the impact of open source models on the market and the involvement of AI companies in regulatory activities.

Meta has released upgraded versions of the 8 billion and 70 billion parameter models, with longer context length and stronger reasoning capabilities.

The latest models support long-form text summarization, multilingual conversational agents, and coding assistance, and are now available for the community to download.

The 405b model, trained on over 15 trillion tokens, is the first Llama model of this scale and is being made available for free by Meta, with the aim of supporting large-scale production inference.

The Llama Stack: Revolutionizing AI Development

Meta AI has released the Llama 3 405b model for synthetic data generation and agentic applications, aiming to define standardized interfaces for AI development.

The Llama stack's open-source approach allows for customizable model weights and offers cost-effective solutions, promoting access and deployment of AI technology across society.

The release of the Llama stack is strategically aimed at building a developer ecosystem, similar to the impact of iOS on Apple's success, while also seeking industry-wide adoption for easier interoperability.

Meta has made cutting-edge AI models accessible to everyone through open source and open weights.

They have worked on supervised fine-tuning, continual pre-training, and synthetic data generation, using VM TensorRT and PyTorch for production deployment support.

The 8 billion parameter version is set for testing, marking a pivotal day in AI history, with Meta's contribution to the field.