IBM's Granite 3.0 Models and Instruct Lab Innovations

IBM's Granite 3.0 models revolutionize enterprise AI with open-source, versatile, and powerful language models for diverse applications.

### IBM's Innovative AI Developments: Granite Models and Instruct Lab
* 🗽 Last week, I visited IBM in New York, where they showcased their latest AI projects, particularly the **open-sourced Granite models** and **Instruct Lab**.
* 💻 "Granite 3.0, released today, is an **open-source family of models** that can be run locally and is designed for **enterprise use cases**, featuring a variety of sizes including a **mixture of experts**."
* 📊 The Granite models are built to utilize enterprise-specific data, allowing businesses to **enhance their AI capabilities** by integrating private data into the model for **improved performance** in various tasks such as classification and summarization.
* 🔍 IBM's approach addresses the challenge of **utilizing data behind paywalls or authentication**, making it possible for enterprises to **leverage their valuable data for model training**.
* ⚙ The Instruct Lab, described as a **revolutionary alignment technique**, allows users to add external knowledge to the Granite model without overwriting its original training, creating a balance between **fine-tuning and retrieval-augmented generation**.
* 📈 With the introduction of mixture of experts models, IBM provides options for **high performance with fewer active parameters**, making them suitable for deployment in **resource-constrained environments**.
* 🔗 Additionally, IBM's commitment to **open-source solutions** is evident through their collaboration with Red Hat and the integration of their new **coding assistant powered by Granite models**.
* 🔮 Lastly, IBM is actively exploring quantum computing, hinting at **potential future intersections between AI and quantum technologies**, which could **revolutionize the field even further**.

So last week, I was in New York to visit IBM. They invited me there to show off all of their new AI projects, and there are two that I want to tell you about today because they were especially interesting. Before we dive in, I want to thank IBM for inviting me to New York, showing me around their new office, which was awesome, and thank you to them for partnering with me on this video. And why I'm especially excited is because both of the things I'm going to show you right now are open-sourced by IBM. So first, let's talk about the granite models. Granite 3.0 was released today. It is an open-source family of models produced by IBM, and they are permissively licensed with Apache 2.0 licenses. It comes in a number of different sizes, including a mixture of experts. And after I tell you about these, I'm going to show you what instruct lab is because that's super interesting as well. It allows you to give the raw models a lot of additional knowledge, but it's not Rag, and it's not

quite fine-tuning either. It's a new technique that they have at least. All right, so let's start from Granite. It is Granite 3.0, the third version of their flagship open-source large language model. These are relatively smaller models that you can run locally, and they even have a very small model that can run on a device with really good performance. These are general-purpose models really built for enterprise use cases, but you can use them for anything. And what makes them special is they have this raw intelligence that you can then take and then supplement with external knowledge. Now, here's how they're thinking about it, and I agree completely. Most of the publicly available data on the internet has already been baked into all of the frontier models. There's essentially no more data on the internet that hasn't already been used. So, what do you do after that? Well, you have a few options. One, you can create synthetic data, which we haven't seen a really strong example of a frontier model having used synthetic data. You could also do a lot more with the existing

data you have, and that's kind of the approach that the 01 models took with test time compute, basically allowing the model to think. And then there's the third bucket, which is data that is either behind authentication, behind paywalls, or siloed in different enterprises. And that third bucket is what IBM is going for with the granite model plus instruct lab. And so, what does that actually mean? Well, first of all, you have data behind authentication and paywalls. So, Reddit has data, X has data, Meta has their Facebook data. All of this data requires you to be logged in, and it is owned by these companies. Then you have data that is owned by enterprise companies. So, let's say you're an insurance company. You have an immense amount of incredibly valuable data that is not public, that you want to use to train your own models. That is what Granite plus instruct lab is really able to unlock for you. So, you take this raw Granite 3.0 model, you take all of your enterprise private data, and then you put it into the Granite model. And then all

of a sudden, you have an extremely capable model that is very intelligent at your specific vertical. All right, so let's read a little bit about the granite model now. So, the granite models come in two different flavors: one is the general-purpose model, and then the other is the mixture of experts. So, as it says here, there's Granite 3.0 8B instruct and base, as well as Granite 3.0 2B instruct and base. Now, especially for the 2 billion parameter model, this can easily fit on any laptop, any phone, and will still perform really well. I'm very excited to hear more about what IBM has in store for their Edge compute strategy. Then, they have the small mixture of experts models. So, first, they have a 3 billion parameter version with 800 million active parameters, and then they have a 1 billion parameter version with 400 million active parameters. So, very, very performant for their size. These models have been trained by IBM to be good at a few things and specific, especially around Enterprise use cases. So,

the new granite 3.0 AP and 2B language models are designed as Workhorse models for Enterprise AI, delivering strong performance for tasks such as retrieval augmented generation (rag), classification, summarization, entity extraction, and Tool use. These compact, versatile models are designed to be fine-tuned with Enterprise data and seamlessly integrated across diverse business environments or workflows. But again, they're just raw models; you can use them for anything you want. Now, obviously, IBM, they cater to businesses, so that is what they built the models for and all the tooling. But you can easily use these models for your own personal use cases as well. They go on to talk about instruct lab, which I'll touch on after this, but instruct lab they describe as a revolutionary alignment technique. So, not quite fine-tuning, but it allows you to give this Granite Model a bunch of additional knowledge without overwriting any of the core knowledge it was first trained with. The other they released is the guardian family of models, which are basically alignment models, very similar to how meta has llama and Lam guard.

Granite has granite and granite Guardian, so although you might not want to use it for your personal use cases, if you're in a business environment, you definitely want to make sure that these models are aligned to your best practices and your guidelines. Let's take a look at some of the benchmarks. So, this is hugging fa's open llm leaderboard V2, and as we can see, compared to similarly sized models, there's gran 3.0 AP instruct, llama 3.1 AP instruct, and mraw 7B instruct. Granite 3.0 leads the pack. Now, here's a benchmark that shows how good it is at rag. So, as we can see here against those two other models that I already mentioned, with faithfulness, it is coming in at the top, and at correctness, it is also leading the pack. Here we have cybersecurity benchmarks, and of course, these are the benchmarks that matter if you're an enterprise and you want to use the granite models for your business. For the public benchmarks, again, Granite wins, and here are some examples of the benchmarks.

that are being tested SEC MML U cyber metric 80 cyber metric 500 LinkedIn sect trust assessment and so on. And of course, if you're in the Enterprise environment, what's also important tool calling. So let's look how it performs in tool calling against other models. Again, these same three models, Granite, Llama, and Mistral, all about the same size. Granite, you can see almost across the board, beats the other two models in tool calling. But they also released, as I mentioned, mixture of experts models. So I want to tell you a little bit about those, and I really like mixture of experts models because it allows you to have a larger model, but the actual active parameters that are being used when you're running inference are far fewer, making it much easier to run larger models on lower spec computers. So Granite 3.03 B, A 800m. And let me actually take a second to explain how to read this. So it's Granite 3.0, that's the model name. 3B is the total number of

parameters, and then the 800m means that there's 800 million active parameters being used in the mixture of experts. Here, they also have the granite 3.0 1B a400m delivering high inference efficiency with minimal trade-off and performance, trained on over 10 trillion tokens of data. The new granite Moe models are ideal for deployment in on-device applications, CPU servers, and situations requiring extremely low latency. Both of these mixture of experts models come open source, open weight, and you can download the Bas version or the instruction-tuned versions. I'll drop a link to the Hugging Face pages and these articles in the description below. Before I get to instruct lab, I just want to also mention IBM is releasing their own coding assistant powered by the granite model. They are really continuing to get deeper into coding assistance, and I absolutely love their embracing of open source. It could not be more clear with their acquisition of Red Hat just a few years ago. If you're not familiar with Red Hat, they

are open-source enterprise software solutions. They have a Linux distribution for enterprise environments. They have OpenShift, which is a Kubernetes-based container platform for building and deploying applications at scale. They have the Ansible automation platform. They have the OpenStack platform, which is for building and managing cloud computing environments. And then they have the middleware, again, all of which is open source. And here's another announcement that I didn't actually know was coming until I saw it today. So my favorite agentic framework, Crew AAI, is now integrated into the Watson X platform. So now you can build agentic workflows and deploy them in an enterprise environment on IBM's services. All right, now let me talk about Instruct Lab because I didn't actually understand it until they explained it to me at the event. So, as I mentioned, imagine you're an enterprise company. You take this Granite model, and it has the raw intelligence, but you want to fill it with your own enterprise data, private data, stuff that you don't want to put out in the public. So how do you do that?

well, typically, there were two ways: one, you use Rag, and Rag is great, but it's not always the most effective. Then, there's fine-tuning, but what I always understood fine-tuning for is more for guiding the tone and basically changing how the model responds, not really for giving it additional information. But now we have instruct lab by IBM, which is kind of this middle ground between those two. You're actually baking external knowledge into the model itself, but it's not quite fine-tuning; it's more like alignment, but giving it a bunch of additional data without replacing any of the original data. So, IBM and Red Hat's new open-source project is designed to lower the cost of fine-tuning large language models by allowing people to collaboratively add new knowledge and skills to any model, so not just a granite; this is any model. And instruct lab itself is open source, so I'll drop that in the description below. So, here's how it's described: it gives communities the tools to create and merge changes to LLMs without having to retrain the model.

from scratch instruct LAB Works by augmenting human-curated data with high-quality examples generated by an llm, lowering the cost of data creation. That generated data can then be used to customize or improve the base model without having to retrain it. So, it's a command-line interface that allows you to add and merge new alignment data to your Target Model through a GitHub workflow on your laptop. Lab's unique training regimen allows new information to be assimilated into the model during an alignment without causing the model to overwrite what it previously learned. So, basically, if you wanted to give a raw Model A bunch of new data, you would basically just use rag, but now we have this new technique where you can add a bunch of data without retraining the model from scratch, and that is what is unique about instruct lab. Again, it's open-source; you can contribute to it yourself. And there's one last thing I want to talk about, which is quantum computing. I don't know much about quantum computing, but IBM is deep in it.

they have their own quantum computer. They have already commercialized Quantum Computing, although it's very new still, but they have it, and they've been working on it for years and years. In fact, one of the lead researchers on the quantum team did a talk during the AI event and started talking about the combination of AI and Quantum Computing, which is interesting to say the least. I'm trying to get an interview with him, so hopefully I do, but let me know in the comments, do you want me to talk about quantum computer? I find it fascinating; I just don't know much about it yet. So, that's it for today. Be sure to check out the new granite models. Let me know if you want me to run them through my Benchmark in the comments below. Check out instruct lab; all of its open source. I'll drop all the links in the description below. And I want to thank IBM again for inviting me to their AI event, showing me all the cool stuff that they're building, and for partnering with me on this video. If you enjoyed this

video. Please consider giving a like and subscribe, and I'll see you in the next one.

IBM Unveils Granite 3.0 - Open Source Family of Small Models!

Artificial Intelligence (AI), Open Source, Generative Art, AI Art, Futurism, ChatGPT, Large Language Models (LLM), Machine Learning, Technology, Coding, Tutorials, AI News, and more

** Exclusive Pinecone Partner **

Matthew Berman

0, released as an open-source family of models, offers various sizes and is designed for enterprise use, with the ability to run locally.

The model aims to supplement its raw intelligence with external knowledge, targeting data behind authentication and paywalls, as well as enterprise-owned data.

Instruct Lab, paired with Granite, provides a new technique to unlock valuable private data for training models, offering a unique solution for enterprises.

The Granite models come in two types: general-purpose and mixture of experts, with the 2B model being compact enough to fit on any laptop or phone.

These models are designed for Enterprise AI, offering strong performance in tasks like retrieval augmented generation, classification, and summarization.

IBM's Instruct Lab provides a unique alignment technique for the Granite models, while the Guardian family ensures alignment with business best practices.

Granite Model's Performance and Mixture of Experts

The Granite model excels in tool calling compared to its counterparts, Llama and Mistral.

It offers a mixture of experts models, allowing for a larger model size with fewer active parameters, ideal for low-spec computers.

The Granite models are open source, with IBM also releasing a coding assistant powered by Granite and integrating Crew AAI into the Watson X platform for agentic workflows.

IBM's Instruct Lab: A New Approach to Fine-Tuning Language Models

Instruct Lab by IBM offers a middle ground between using Rag and fine-tuning for language models, by baking external knowledge into the model without replacing original data.

The open-source project aims to lower the cost of fine-tuning by allowing collaborative addition of new knowledge and skills to any model.

Instruct Lab's unique training regimen enables the assimilation of new information during alignment, without overwriting the model's previous learning.

IBM has successfully commercialized Quantum Computing and has been working on it for years.

A lead researcher from their quantum team discussed the intersection of AI and Quantum Computing at an AI event.

The speaker is seeking an interview with the researcher and is open to discussing Quantum Computing further.