Meta's Llama 3.2: Vision-Enabled AI Model Innovations

Meta's Llama 3.2 introduces vision capabilities and efficient models for on-device AI applications.

### Meta Unveils Llama 3.2: Enhanced Vision and Edge Device Capabilities
* 🚀 Meta has just launched Llama 3.2, introducing significant advancements including **vision capabilities**, **new model sizes**, and **enhanced performance**, following the success of Llama 3.1.
* 👁 The standout feature of Llama 3.2 is its ability to **"see,"** marking a major upgrade in the Llama model family with both **11 billion and 90 billion parameter vision-capable versions** available as drop-in replacements for Llama 3.1.
* 📱 Additionally, two smaller text-only models, 1 billion and 3 billion parameters, are designed for edge devices, which include smartphones and IoT devices, emphasizing the **shift towards on-device AI compute**.
* 📊 These new models boast impressive capabilities for tasks like **summarization and instruction following**, with **128k context windows** and **state-of-the-art performance** compared to peers in the same class.
* 🔧 Meta is also rolling out the **Llama Stack**, a suite of tools for developers to simplify working with Llama models across various environments, enhancing the **deployment of AI applications**.
* 📈 Benchmark tests show that Llama 3.2 models outperform competitors, with the **3 billion parameter model achieving notable scores** in various challenges, confirming its **efficiency and effectiveness**.
* 🖼 The **11 billion and 90 billion models** are specifically designed for image reasoning tasks, integrating a **new architecture** that maintains the text capabilities while adding **visual input support**.
* 🎥 Future videos will test both text and vision intelligence of these models, showcasing Meta's commitment to **pushing the boundaries of AI technology and its applications**.

Meta Connect just happened, and Meta just dropped Llama 3.2. We have a new model, new sizes, vision capabilities, and so much more. So, that's what we're going to go through today, and thank you to Meta for partnering with me on this video. So, I'm going to talk about the highlights right away, get you that information immediately, and then I'm going to go more in-depth on these topics in a moment. So, first, Llama 3.2. That's the big news. Llama 3.1 was a huge improvement over Llama 3.0, and now we have 3.2. What's different about Llama 3.2? Well, now Llama has vision. Llama can actually see things, and that is an incredible update to the Llama family of models. We have an 11 billion parameter version and a 90 billion parameter version of their new vision-capable models, and these are drop-in replacements to Llama 3.1, which means you don't have to change any of your code if you're already using it. You

don't have to really change anything. You simply drop in these new models. They're different sizes, but they have all the capabilities of the text-based intelligence, and now they also have vision-based intelligence. They also dropped two text-only models that are tiny: 1 billion and 3 billion. These are specifically made to be run on edge devices. Now, if you've been watching my videos at all, you know I really believe in AI compute getting pushed to edge devices. And what are edge devices? Cell phones, computers, Internet of Things devices, basically anything that's not in the cloud. And I truly believe more and more AI compute is going to be pushed to edge devices, and this is a huge step in that direction. Models are becoming much more capable at a much smaller size, and that's what we're seeing here: Llama 3.2, 1 billion and 3 billion parameter Texton versions. These are pre-trained and instruction-tuned, ready to go. So, I can imagine these fitting easily into the meta AI Rayband.

glasses. The 1 billion and 3 billion parameter versions are 128k context windows out of the box, and they are state-of-the-art compared to their peers on use cases like summarization, instruction following, rewriting tasks, all again running locally. And this again confirms what I really believe the future of AI looks like, which is a bunch of really small, capable, specialized models that can run on-device. So specifically for these models, they're really good at these types of tasks. And if you remember when I worked with Qualcomm on that video, Qualcomm was very much about pushing AI compute to Edge devices. And of course, Meta is partnered with Qualcomm on this, and these models are ready to go out of the box, optimized for Qualcomm and MediaTek processors, as I said, supported by a broad ecosystem. Llama 3.2, 11b, and 90b vision models are drop-in replacements for their corresponding text model equivalence while exceeding on image understanding tasks compared to closed models such as CLIP-3, Haiku. Now you know I'm going to

be testing all of these models in subsequent videos, so make sure you're subscribed to see those tests. Additionally, unlike other open multimodal models, both pre-trained and aligned models are available to be fine-tuned for custom applications using Torch Tune and deployed locally using Torch Chat. And they're also available to try using our smart assistant, Meta AI. Now, it's clear that Meta is investing a ton into their ecosystem, building out the tooling to fine-tune and services to host, and basically everything that you need to have an open-source model in your personal life or your business. They're also releasing their first Llama Stack distributions, and that is a set of tools that developers can use to work with the Llama models and build everything around the core LLM that is necessary to build production-level applications. Here, it describes Llama Stack as a way to greatly simplify the way developers work with Llama models in different environments, including single node, on-prem, cloud, and on-device, enabling turnkey deployment of retrieval-augmented generation and tooling-enabled applications with integrated safety and

looking at the open source of course llama stack GitHub repo. Here are the things that it supports: inference, safety, memory, agentic system evaluation, post-training synthetic data generation, and reward scoring. Each of those has a REST endpoint that you can use easily. So, you can download llama 3.2 from llama.com or Hugging Face, and it's going to be available on some of Meta's cloud partners, including AMD, AWS, Databricks, Dell, Google Cloud, Grok, IBM, Intel, Azure, Nvidia, Oracle Cloud, Snowflake, and more. All right, now let's look at some of the benchmarks. So, here are some benchmarks in this column, and then in this row, the different models that it's comparing against. So, here's llama 3.2 1B and llama 3.2 3B versus Gemma 2B and 53.5 mini. So, these are comparing these small on-device models, and as we can see, this llama 3.2 3B model actually performs incredibly well versus their peers in the same class of models. Here's MMLU at

63 gsmk at 77. Here's the ark challenge at 78, and here's one for Tool use. So, here's Nexus and bf clv2, really, really good for being such a small model. Now, let's look at the larger variants that have Vision enabled. So, here we have llama 3.29 B and 11b, and comparing against Claud 3 Hau and GPT 40 mini, and the Llama 3.2 90b seems to be the Best in Class almost across the board. So, let's test the tiny model first. I'm on gro.com llama 3.2 1B preview right there, and let's see how fast this thing is going to go. Write me a story. Oh my God, 2,000 plus tokens per second. Look at that. Let's give it something a little bit more specific now. Let's just see if we can do it. Write the Game snake in Python. Okay, there it is, 2,000 tokens per second, and we'll see if it actually

works. Oh, look at that. It worked. Unbelievable. With 2,000 tokens per second, a total output time of less than 1 second, a 1 billion parameter model got the snake game on the first try. Very, very impressive. So, I'm going to save the vision test for another video, but for now, let me tell you a little bit more about it. The two largest models of the Llama 3.2 collection, 11b and 90b, support image reasoning use cases such as document-level understanding, including charts and graphs, captioning of images, and visual grounding tasks such as directionally pinpointing objects in images based on natural language descriptions. For example, a person could ask a question about which month in a previous year their small business had the best sales, and Llama 3.2 can then reason based on an available graph and quickly provide the answer. Now, I definitely want to try the world's Waldo with this Vision model. As the first Llama models to support Vision tasks, the 11b and 90b models required an entirely

new model architecture that supports image reasoning to add image input support. We trained a set of adapter weights that integrate the pre-trained image encoder into the pre-trained language model, so built right into that core model. But they used a new technique to do so. The adapter consists of a series of cross-attention layers that feed image encoder representations into the language model. We trained the adapter on text-image pairs to align the image representations with the language representations. During adapter training, we also updated the parameters of the image encoder but intentionally did not update the language model parameters. By doing that, we keep all the Texton capabilities intact, providing developers a drop-in replacement for Llama 3.1 models. So again, it's going to be just as good as Llama 3.1 text models, but now it also has vision. And if you want to read more about the details of how they actually achieved this, I will drop links to everything in the description below. In post-training, they did several rounds of alignment on supervised fine-tuning, rejection sampling, and direct preference optimization (DPO). They leveraged synthetic

data Generation by using the Llama 3.1 model to filter and augment questions and answers on top of in-domain images. So, synthetic data is here. It is here, and it is ready, and Llama 3.1 is capable of it, let alone Llama 3.2. Now, they also use Llama 3.1, the larger one, as a teacher model to teach a much smaller version, and that's how we got the 1 and 3 billion parameter Llama 3.2 versions. They use two methods: pruning and distillation on the 1B and 3B models, making them the first highly capable lightweight Llama models that can fit on devices efficiently. I am 100% behind on-device AI compute. So, that's it. Congrats to Meta on another fantastic open-source release. I am going to be testing all of these different models. I'm going to create two different test videos: one for testing the text intelligence and then one for testing the vision intelligence. Thanks again to Meta for partnering with me on this video. If you

enjoyed this video. Please consider giving a like and subscribe, and I'll see you in the next one.

Llama 3.2 is HERE and has VISION 👀

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

2 introduces vision capabilities, with new models available in 11 billion and 90 billion parameters, serving as drop-in replacements for Llama 3.

Additionally, two tiny text-only models (1 billion and 3 billion parameters) have been launched for edge devices, pushing AI compute closer to users.

This update marks a significant step towards more capable AI models at smaller sizes, aligning with the trend of increasing AI compute on edge devices.

Meta's Llama Models: A Game Changer for AI

Meta's Llama models, with 1 billion and 3 billion parameters, offer 128k context windows and excel in tasks like summarization and instruction following, all while running locally.

These models are optimized for Qualcomm and MediaTek processors, providing a broad ecosystem for deployment.

Meta is also offering pre-trained and aligned models for fine-tuning, along with the Llama Stack distribution to simplify development and deployment of production-level applications.

2 offers support for inference, safety, memory, agentic system evaluation, post-training synthetic data generation, and reward scoring, all accessible via REST endpoints.

The model is available for download from llama.

com and Hugging Face, with additional access through Meta's cloud partners like AWS, Google Cloud, and Azure.

2 performing exceptionally well against its peers, with the 3B and 90B variants leading in performance across various tasks.

2 collection's largest models, 11b and 90b, support image reasoning tasks such as document understanding and visual grounding.

These models use a new architecture with adapter weights to integrate image input into the language model, trained on text-image pairs for alignment.

The post-training process included supervised fine-tuning and preference optimization to enhance the model's performance.

1 is generating synthetic data by filtering and augmenting questions and answers on in-domain images.

1 model is used to teach the smaller 1B and 3B parameter versions of Llama 3.

These lightweight Llama models are designed to fit efficiently on devices, promoting on-device AI compute.