Hello everyone, welcome to the AI Anytime channel. In this video, we are going to explore SAM 2, which is an acronym for Segment Anything Model. It's a foundation model towards solving prompt visual segmentation, as you can see it here on the screen on the GitHub repository. Meta has, you know, unveiled this model recently and it's a revolutionary advancement when it comes to object segmentation. So earlier, Meta released SAM, and now they have SAM 2, which integrates real-time prompt-based object segmentation. Now you can prompt, you know, to segment some objects, you know, in images and videos, enhancing the accuracy and speed.

Its ability to operate across previously unseen visual domains holds significant promise for various fields, from creating, uh, you know, video editing to scientific research and whatnot. Right now, we're going to not only look at, uh, SAM 2 theoretically, but we're going to, uh, you know, use this on a GPU machine. The key features, you know, that they also explain if you come down and they have a research paper on this, so you can read that, you know, they also have some how you can use it. Now my plan is to show you how you can use it on a GPU, okay, and I'm going to use RunPod to basically spin up a GPU and then show you how you can do it.

Some of the key features are a unified segmentation, so for image and video, it's unified, uh, you know, segments both static images and dynamic videos. Zero-shot generalization means even if it hasn't seen anything, you know, it can, uh, you know, perform really well on unseen visual content.

Now my plan is to show you how you can use it on a GPU, okay, and I'm going to use RunPod to basically spin up a GPU and then show you how you can do it. Some of the key features are a unified segmentation, so for image and video, it's unified, uh, you know, segments both static images and dynamic videos. Zero-shot generalization means even if it hasn't seen anything, you know, it can, uh, you know, perform really well on unseen visual content.

without custom adaptations that's fantastic real-time processing 44 frames per second guys that's what it says you know on the on the little on the benchmarks and all now I'm going to show you how you can do it on uh runpod but

you can use any other cloud you can see I'm here on runpod I have a bit of credits left so I'm going to leverage that credit I'm going to use h00 nvl over here you can see this is the GPU that I'm going to use it gives me 94 GB

of VRAM which is video Random Access Memory has 180 GB of RAM and 16 CPUs and it's going to charge you know it's going to cost me around $3.14 per hour you know so what I'm going to do now is

select my GPU configuration I'm I'm selecting h00 nvl but you can select any other GPU depending on you know like whatever suits your VRAM requirement now I'm going to keep everything as default setting and I'm just going to uh click on Deploy on

demand it's going to take a few seconds to deploy this on this cloud you can use Lambda Labs Paperspace Weights & Biases SayMaker Vertex AI ML whatever depending on what machine or hyper-scalers you are using feel free to use any GPUs that you have now once

this button is enabled that's called connect you have to click on that connect and then we're going to open Jupyter lab that runs on port 8888 and now you can see we have uh something called connect which is enabled and it so it's running now so

let's click on connect and it shows connect to Jupyter Lab which is Port 8888 if you want to do it locally I have shown this in a previous video that how and I have shown Llama 3.1 45B model how you can now use it locally within

a Docker container and route it through this runpod uh you know uh the machine that you are running if you want to do it that you have to then do basic SSH terminal and so on and so forth I'm going to click on connect to Jupyter Lab

so let's click on that and once I clicked on that it's going to open a lab you know within the environment and this is a CUDA enabled notebook that you can access over here now so let's let's have a look so for that what I'm going to do

is I'm going to first do a few things here in the setting you can first make this a dark theme so you can see I'm I feel more comfortable in dark you can also do uh increase code font size so let's increase it up a bit and this is fine and now what I'm

going to do is open a notebook rename this as Sam 2 and then let's click on rename and now here I'm going to first do a git clone thingy so let's do a git clone you know for this repository here so that's what I'm going to do so let's

just copy come here and then just do git clone so I'm just doing git clone and this will clone this repository right now uh when I'm doing git clone https github.com facebook research segment you also probably don't need dogit also it's

Fine, you can see now it says cloning into segment anything. It will create a folder over here. You can see it has created this folder and it's taking a bit of time. Meanwhile, I'll make this font size a bit bigger. Let me just do that.

It's going to take a bit of time, guys, because you know it's kind of a lot of files in the repository, so you know it takes time. As you can see, it says receiving objects and now it is done. Now, when that is done, what you're going to

do is let me first show you this. You can see all of these files over here, right? And folders, files, and you have to go notebooks, checkpoints, blah, blah, blah. Let me go back to workspace and then come here and we have to now CD into it. Let's

change the directory. So what I'm going to do is CD segment and the name of that folder anything to. And even you can do a working directory. You can see we are in segment anything to now. The next thing is we're going to do a pip install thing, right?

Because we have to install everything which is there in the requirements.txt. And when you do dot, it means that use the requirements file within the folder I am in currently. So let me just run this now. All right, guys. As you can see, everything has been installed here now.

After installation, I already ran a couple of commands because it, anyway, it's going to take some time for the model to download. So what I did, I went inside checkpoints. So there is a folder called checkpoints now. If you go inside checkpoints, you will find out a file

which is a script that you see within the checkpoint. Something is wrong, it's it's hanging. But yeah, you see what I'm doing now inside checkpoints. I'm going to use download checkpoint. Dosh file. I'm running this file and it downloads the model so you can see tiny. PT, small. PT, 

all the files you know base PLU, large PT, right? So you can see the large model as well, you know everything has been downloaded. All checkpoints are downloaded successfully. So I have downloaded everything that I need. Now what I'm going to do is I'm going to use 

some image processing library because we're going to use, we're going to basically do image processing here. So I'm going to do pip install open CV, and then I'm going to use headless open CV here, so open CV hyphen python hyphen headless. 

Let's run this now. I'm going to run this after you install open CV python. What we're going to do is we're going to uh look at the present working directory and you can see we are currently within checkpoint. So we have to come out of 

checkpoint into the segment anything folder. So we just did a change directory, came back to segment anything and then we have to import everything uh that we need. So we're going to use torch, numpy, M plot live, pillow, Sam. The files that we need and what I'm doing, I'm defining the 

file path. So as you know that we have stored our files, model weights in the checkpoints, and then I'm using the checkpoint, which is within the checkpoint. I don't know why it's not opening, but there's something wrong. And then in model config there's a YAML file. 

that we have in the image I have a brain.jpg. So if you look at this is the image I am using, you know, just for your, you know, reference. You can use any other image, you know. This is a brain tumor uh image that you see, um CT scan of brain.

tumor in the Radiology segmentation has good use cases when it comes to brain tumor segmentations, lung nodule segmentations, you know, sclerosis, you know, liver segmentations, cardiac structures and so on and so forth. So healthcare is a fantastic uh, you know,

industry for using Sam tools to solve complex Radiology problems or assist radiologists. Now here I'm just giving the image path. I'm opening the image and converting that into array over here so you can see it's a numerical representation of your image and that's

what we are doing now. I'm initializing the predictor using build as Sam2 and then I am processing the image to produce the embeddings of that image and defining the input input point prompt X and Y and label one for foreground. That's what I'm doing here. Then it

performs the prediction using inference mode uh function of torch and then it sorts mass by scores and displays the results as you can see it over here. You can see it over here. And the next thing is guys you can also do if you come here let me just uh let me just go back, you

know on this repository now if you if you look at this repository they have given like all the explanations, you know, that you need to uh run this. I can find it out over here. I took most of the code from here and and one thing that I

wanted to cover as well uh let me meanwhile let me change this first let's try with the other image the other image is lung nodules you can also look at this image predictor example Sam 2 also supports automatic mask generation on images just like Sam so if you open this

notebooks right let me open that notebook here they so about and they have three different things over here you can see let me just expand that they have given you some sample notebooks that you can run you can see videos automatic mask generator video predictor example and image predictor example

you can see you can see it over here for this is a lung nodules example that we've been using it but you can try it out with different images as well there's something wrong with the notebooks I don't know why it's not loading but yeah

you can also follow up these notebooks automatic mask generator image predictor example and video predictor example I will also give my notebook so you can also have a look you know how to run this how to set this up and you know how to set this up and how you can run this

as well now we used an h00 nvl to basically download all the model weights and run it if you I'll give this notebook on my GitHub repository so if you have any questions thoughts feedback do let me know in the comment box you can

also reach out to me through my social media channels find those information in the channel banner and channel about us if you like the content please hit the like icon if you haven't subscribed to the channel yet please do subscribe to the channel guys that motivates me to create

more such videos that's all for this video guys thank you so much for watching see you in the next one

How to run SAM 2 (Segment Anything AI Model)?

Welcome to AI Anytime, your source for the latest and greatest in Generative AI and ML. From tutorials to expert interviews, I cover it all. Subscribe for cutting-edge content on LLMs, Deep Learning, NLP, etc. Join us on this journey of AI and ML.




How to run SAM 2 (Segment Anything AI Model)?

How to run SAM 2 (Segment Anything AI Model)?

Scripts