Running Llama 3.2 Locally with Gradio Interface

Running the Llama 3.2 model locally with Gradio enables interactive code generation.

### Running Llama 3.2 Locally with Gradio Interface
* 🚀 To create a Gradio app, we begin by utilizing the **Gradio chart interface** and passing the defined **predict function**.
* 🐍 Executing the script with Python 3 along with the file name launches the Gradio app, which can be accessed via a **local host link**.
* 💻 After prompting the Llama 3.2 model for a code example on creating a Gradio app, we receive a **detailed response showcasing the code**.
* 🛠 A crucial step involves creating a **predict function** that handles user messages and maintains a **conversation history for context**.
* 📜 The predict function processes the latest user message and appends it to the history, while also **storing the model's response**.
* ⚡ By setting the streaming option to true, the model generates responses in a **typewriting effect**, enhancing user interaction.
* 📥 Users can download the model's response, and once downloaded, it can be found in the **designated downloads folder**.
* 📦 Finally, to run the Llama 3.2 model locally, one must ensure the model is in **GG UF format** and install the Llama CPP app using **pip**.

hi Lama 3.2 consists of a collection of models. In this video, I show you how to run any of these models locally using your own Gradio interface. To get started, you should come to llama.com and download any of the models by clicking Download Models. You can fill out the details specified here and download any of the models. However, we need the model in a particular format, so I'm not going to fail this information. We want to use the GG UF format, which is also known as GPT Generated Unified Format. So, in order to get the G UF format, we go to Hugging Face and search for Lama 3.2 GUF or G or whatever. So, you can see here we have all these models. I already downloaded this one, but you can try any of these. In order to run this model locally, you need to install the Llama CPP app. You can install it with pip install llama CPP. So, once you come here, you can see we are under the model card category. So, you click Files and Versions. All you need to do here is

to click the download button to download it and save it on your machine. I have already done that, so if I go to my downloads folder, you can see I have Lama 3.2 instruct to run the model locally. We also need an interface to interact with it, so you can use a python framework like radio or streamlet. In this example, we're going to use gradio, so let's launch Ker. I have already created and run the project, so over here I have the file local llm U.P. First, I set up a virtual environment in this project, so you can see it in this folder, and install gradio 5 with P install gradio. So over here, we import gradio as gr, and from the Llama CPP app, we import llama. Next, we have to load the local model into the project using a specified location, so you can see here I have it in my download folder. Let's go to the folder again. That is what I showed previously. That is this file. So here, all you need to do is to specify

the name of the model and the location where it is found. The next step is to create a predict function. The predict function will be called when a user sends a message, so it has the following parameters: message and history. Message is the latest message from the user; history consists of a list of two PS containing previous messages from the user as well as previous responses from the model. Next, over here, we loop through the history of conversations and append both messages from the user and messages from the model. Then, over here, we add the latest message to the list as well. To store the model's response, we should create an empty string as you see here. Then, we use the Lama 3.2 model to generate the response in chunks. You can see over here we have specified stream as true; that will help the model to generate a response in a streaming manner, which is similar to a typewriting effect. Then, over here, we provide a list of messages history to the model, and with this part, we extract the content from the model's response. Next, we yield

the response to return each part of it as it is generated, and finally, we need to create the UI to generate the UI. We use Gradio chart interface and pass the predict function we defined over here. So when we execute this script with Python and the file name, we run the Gradio app. I have already run the script with Python 3, followed by the file name, which is here. So you can see over here, we have a link to the Local Host, and once I command-click that, we can view the app in the browser. Let's prompt the Llama 3.2 model about how to create a Gradio app by giving a code example. After entering the prompt, we can send it. So you can see over here, this is the prompt we just entered, and here we have the response from the model with a code example. So this is how you can run the Llama 3.2 model locally using Gradio with your own Gradio interface.

Run Llama 3.2 Locally in Cursor With Gradio Interface

Core SwiftUI: Animation & Design

2 offers a collection of models that can be downloaded from llama.

The models should be in the GG UF format, which can be found on Hugging Face.

To run the model locally, you need to install the Llama CPP app using pip.

Setting Up a Local Llama Model with Gradio

2 model is downloaded and saved locally for use.

Gradio is chosen as the framework to create an interface for interacting with the model.

A virtual environment is set up for the project, and Gradio is installed to launch the application and load the local model.

The Gradio app is set up by defining a predict function that handles user messages and conversation history.

2 model is used to generate responses in a streaming manner, creating a typewriting effect.

The app can be run locally, and it provides a code example for creating the Gradio interface.