Today I'll be showing you how to build an AI web scraper using Python. That's right. And we'll be building an application that can scrape literally any website using AI. This is super cool. So let's dive in and let me give you a quick demo of how this application works. So the functionality here is pretty basic. All we need to do is give it a website URL. So maybe something like the medal table from the recent Olympics. So let's paste that right here. We will scrape the site, grab the DOM content, and then we can pass a prompt to our AI that will be able to grab any information from this website that we want. All right. So we have our DOM content here. This is all the content from the site. Now I can paste in a prompt like give me a table that contains all of the countries and their medal count. We can wait a second and we'll get that information. So there we go.

We just got a table now that contains a bunch of medal counts from the countries that were on that website. Let's look at a few other examples, then dive into the code, and I'll show you exactly how we build something like this. So here's a more e-commerce-related example. It's a website called True Classic, which actually has my favorite t-shirts. And I'll show you how we can scrape this now and grab all of the different t-shirts from this site. So I'm just looking at the polo shirts here. I'll paste in this URL, scrape the site, and then we'll give it a prompt and grab that information. So let's parse the content now using this prompt. And here you go. We get a result here with a few different tables containing the different products' names, their price, currency, ratings, etc. Lastly, let's look at a real estate example. You can see here I have a property finder website for properties

in Dubai Marina, which is a place I'm going to be moving to shortly. And if we scrape the site and give it a prompt again, we can get that data. And here you go. We get a bunch of different properties here with all of their relevant information. All right. So now that we've looked at that demo, we're going to start building out the project. Now, in order to do this, we're going to need to install a few different dependencies because we're going to be using things like Streamlit for the front end. We're going to be using Selenium for actually doing the web scraping. And then we're going to be using Long Chain for actually calling the AI and having the AI pass through our data. So what we're going to do is set up a virtual environment here. And notice I've just opened up a new window in Visual Studio Code. So let's do that. Let's install our dependencies, and then we'll start building this project.

So in order to do that, I'm going to say Python 3.-M, then v, and then env. Or I can name this anything I want. In this case, I'm going to name it AI. Now, if you're on Windows, you can simply replace Python three with Python. And this should spin up a new virtual environment for you using Venv. Okay, so let's hit enter here. And now you can see that we have a new one called AI. Now we need to activate that virtual environment. Now, if you're on Mac or Linux, the command to do that will be source, the name of your virtual environment, then slash bin, and then slash activate. When you do this, you should see that you get a prefix before your command line, saying the name of the virtual environment. Now, if you're on Windows, I'm going to show two commands on screen right now that will show you how to activate your virtual environment. They'll differ depending if you're using CMD or PowerShell.

So just type that command in. Activate your virtual environment. And now we're going to install a few different dependencies. Now all the dependencies that we need I've actually put up on GitHub. So I'm going to link this in the description. But go there and click on requirements.txt. Now requirements.txt will contain all of the requirements that we need to install. So I'm just going to copy the contents of this file. And I'm going to make a new file in my local directory here that contains the same. So I'm going to say requirements.txt. Here I'm going to paste all of these requirements inside. So again go to the GitHub. It'll be linked in the description. Copy the contents of the requirements.txt file and paste them inside of a new file in your local directory. And then we can install them using this file. So now what we're going to do is simply type pip install -r and then requirements.txt.

Obviously, make sure that when you're running this command, you're in the same directory as this file. And then go ahead and hit enter. Now this will install all the dependencies that we need inside of this virtual environment so that they're contained without installing them system-wide. Now you'll notice here that we're using Streamlit, Link Chain, Selenium, BeautifulSoup, and Python Dot Env, a few other modules as well. And these will allow us to kind of build the entire project. And I'll explain each library and why we're using it when we get to that stage. It's just easier to install all of them at once. All right. So all of those dependencies have been installed. So I'm going to close my terminal now. And we're going to move on to the next step. So the first thing I'm going to do is create a Python file called main dot here. Now just to quickly walk you through the steps that we're going to follow through here.

The first thing that we're going to do is create a very simple Streamlit user interface. If you're unfamiliar with Streamlit, this is a way to create really simple Python web applications that just takes a few lines of code. So, it's probably the easiest way to interact with things like Lmms, which is what we're going to be using here. Once we build the Streamlit UI, the next step is to actually grab data from the website that we want to scrape. So, in order to do that, we're going to use a Python module known as selenium. Selenium allows us to automate a web browser. So, we can actually navigate to a web page. We can grab all of the content that's on that page. And then, once we have that content, we can do some filtering on it, and we can pass it into an let in line being something like ChatGPT, right? Or GPT four or llama three or whatever you want to use.

And then we can use that to actually parse through the data and give us a meaningful response. So let's start here by building out the Streamlit UI. And we'll kind of build this as we go through the video. So we're going to say import Streamlit as st. Now beneath here, what we're going to do is add a title for our website. So we're going to say st.title. And this is going to be AI web scraper. And then we're just going to put a kind of URL input box here. So we're going to say URL is equal to st.text_input like that. And then we're going to say enter a website URL. Okay. And this is just going to create a very simple kind of text input box for us. Perfect. Now beneath that, we're going to say if st.button. And then this button is going to be scrape site.

Then we're going to do something inside of here. So this is as simple as it is to create a Streamlit UI. This makes a title. This adds a text input. And then the button we can put here and we say if this button is clicked then we're just going to execute whatever code is inside of this if statement. So for now actually if we want to test that we can just say t dot write. And then we can write something like scraping the website. And then we can proceed to scrape the website down below. So in order to run our Streamlit application, the way that we do that is we type Streamlit and then run, and then the name of the Python file that contains our Streamlit app. In this case it's Main.py. Again, notice that I'm running this from the same directory as my Python file. So I do Streamlit run Main.py and now it's going to spin open a web server for us.

And you see that we have a web scraper. We can enter a website URL and then press scrape site. What do we do that it says scraping the website. If we refresh it'll just go back to that main state. Perfect. Now we can still write code while this is running. It will automatically refresh. But for now I'm just going to close this with control C, and we're going to go on to actually do the web scraping component of this app. So in order to do the web scraping here we're going to use selenium. Now what I'm going to do is make a new file called scrape.py. Now this is where we'll write our scraping code just to separate it from the main files. That's a little bit easier for us to navigate. So what we're going to do is import a few selenium modules or selenium I guess classes that we need to use. And then we're going to write a function that takes a website URL

and just returns all of the content from that website. Now, as I said, we're going to use Selenium to do this. And then I'm going to show you how we can connect this with a service known as Bright Data, which is actually the sponsor of this video. But don't worry, we'll talk about them later on, which will allow us to actually do this at scale and to get past things like captchas, IP bans, and other blocks that you'll commonly encounter. So what I'm going to do here is import Selenium.webdriver as WebDriver. And then I'm going to say from Selenium.WebDriver. And this is Dot chrome.dot.service. We are going to import service okay. Now we're going to write a simple function that will allow us to actually grab the content from a website using Selenium. Now again with Selenium allows us to do is control our web browser. We can actually do things like click buttons and interact with text widgets or,

you know, kind of interact with the page as if a human was doing it. But in this case, we're just going to use it to grab the content from a site. So I'm going to make a function called scrape_website. And then inside of here, I am going to put website. Now what I'm going to do is just do a print statement. And I'm going to say launching Chrome browser. Just so we know what's going on here. And I'm going to write a few different variables. So the first thing I'm going to do is say Chrome driver path is equal to an empty string. I am then going to specify some options. Now options is going to be WebDriver.Chrome options. We need this when we set up the WebDriver. Even though it's empty, we still need to pass it. And we could specify options here later on. And then I'm going to say driver is equal to WebDriver.Chrome.

And we're going to say service is equal to service. And we're going to pass inside of the service our chrome driver path. We're then going to say options is equal to options. So let's save that. And I'll just make this a bit bigger so we can see. And let's write the rest of the code. And that'll quickly explain what all of this is doing. So we're going to write a try block. And we're going to say driver dot get. And we're going to try to get the website. What this does here is use our WebDriver to actually go to this specific website. And then once we're on the site we can grab the content from it. So we're just gonna do a print statement. We're going to say print page loaded data dot. So we know that it's loaded. And then we can grab the HTML by simply saying HTML is equal to driver dot page underscore source. Then we can return the HTML.

And we can have a finally block here where we say driver dot quit. And that's it for this function. Okay, I know I went fast. Don't worry. I'm going to explain what's happening here and what the next step is. So the first thing we need to do is we need to specify where our Chrome driver is. Now this is actually an application that we'll need to download in just one second that allows us to control Chrome. So we need to have a Chrome driver in this application will be different depending on what operating system you're using. We then have options. Now the options are. So we can specify how the Chrome web driver should operate. We don't need to pass any options right now. That's why it's blank. But we could specify that we want to run it in headless mode. Or we want to ignore, you know, images or something along those lines. Next what we have is we set up our actual driver.

So we say WebDriver Scrub. In this case, we're going to use Google Chrome. But we could be using Firefox Safari. It doesn't really matter. We can automate pretty much any browser that we want. We then specify the service that we're using. And that's going to be wherever the Chrome driver application lives. And then we give it the options, and we're good to go. Then we can use the driver, which is effectively automating our browser by using commands like dot get okay. Then we grab the page source, which is our HTML, and we return that. Now lastly, I'm just going to import time, and I'm just going to do a sleep so we can see what's going on here. So I'm going to say time dot sleep test. Great. So now that we've done this, the next thing we need to do is grab our Chrome Web driver. So to do that, we're going to go to a link that I'll leave in the description.

It's this one right here. And you're going to click on stable. When you do that, it's going to give you all of the download links for various operating systems for appropriate Chrome drivers. And the Chrome driver version needs to match your Google Drive or, sorry, your Google Chrome version. So I recommend just updating your Google Chrome to the most recent version, and then you can just download the most recent version of Chrome Driver. Anyways, all the versions are here, and you want to just go with the latest stable version or whatever matches your Google Chrome version. Okay, so in my case, I'm going to download the Chrome driver for Mac Arm64 because I'm using the new M2 or M3 or whatever chips that they have. But if you're running an Intel Mac, it's going to be this one. And then you have Windows, obviously, and Linux, okay. So go to this download link, and it's going to download a zip folder for you.

Now I already have that downloaded. And you'll see that when you download it, it will give you a folder or a zip folder that looks like this. Now what you want to do is extract that zip folder so you get the actual folder, and then you want to copy the application that's inside of here and put it in the same directory where your code is. So I'm just going to copy the Chrome driver inside of here by simply pasting it in VS code. And now I have access to it in the same path where my code is. Okay. So again, go to this website, download the appropriate Chrome driver, extract the zip folder, grab the Chrome driver application, and maybe name something slightly different depending on your operating system, and then paste it in the directory where your code is. Great. Now that you've done that, you're simply going to write for the chrome driver path dot slash and then chrome driver. Now this will be the name of your application.

So if you're on Windows, I believe it's a dot x. But just whatever the name of this app is, you want this variable name to match. Okay, now that we've got that, what we can do is go back to Main.py. We can import this function, and then we can call it, and we can test this out. All right. So from Main.py here, in order to use this, we're going to say from scrape import scrape_website. Then inside of this function, we're going to call scrape_website with the URL. We're going to store that in a result. And we are simply going to go here and print out the result, just so that we can see in our terminal if we're getting any response. Okay. And let me just get out of whatever I have here and clear. Okay. So now we're going to rerun our application. So to do that, we're going to say Streamlit run and then Main.py.

So let's go ahead and do that. And now we can enter a URL to test. So I'm going to test my website. And feel free to do this by the way, which is just tech with Tim Dot net. I'm going to press scrape site, and we should see that the Chrome driver launches. It shows our website. We're just waiting for 10s here where you can check out my software development program. Great plug there anyways, and then it will quit. It will return the HTML, and we should be able to see that if we go to our terminal. And you can see if we go back to our terminal, we get all of the content from that page. Perfect. So that is exactly how you do the web scraping using selenium. However, I want to show you that this doesn't always work as expected. So if I actually try to scrape something like Amazon. So I'm going to do Amazon. Notice what will happen here.

So let's go to Amazon, okay, and see that we actually get a caption. Now the reason for this is that when we do web scraping locally on our own computer using something like Chrome Driver, it's very easy for the website to detect that we're a bot. There's all kinds of things that it can look for, and you'll notice that when you start doing this more and more, a ton of websites are actually going to block you. Now they could be blocking you with that captcha or they could be blocking you. Things like IP bans, not showing you the correct content, and it can be a huge nightmare, especially if you want to put this app out into production. Now that's where the sponsor of this video, Bright Data, comes in. Now, Bright Data provides a ton of different products that really just make web scraping and data collection on the internet significantly easier. Now we're actually going to use them in this video entirely for free.

and you're going to see how we're able to actually scrape sites that would typically block us using this service and this technology. Now, what I'm going to focus on here is something known as their scraping browser. However, they have a ton of other products as well, like residential proxies. They have web unblocker. They have this new search engine API. They've got a ton of stuff coming out. So definitely check them out, and I'll leave a link in the description where you can actually sign up for a new account here and get some free credits so that you can follow along with the rest of this tutorial. Now, keep in mind that you can just keep doing web scraping how we're doing it right now. You don't need to change this, but if you want to get unblocked or there's websites that are giving you CAPTCHAs or causing you issues, or eventually you want to push this into production and do web scraping at scale, you will need to use a service like Bright Data.

Anyways, I want to show you exactly what we're going to use here and how it works. So I'm going to go to my user dashboard, which is only available once you make an account. Now I have some credits on here. And what we're going to do is make a new instance of something called their scraping browser. So to do that, I'm going to click on this little icon right here for proxies and scraping infrastructure. But notice they have a ton of other stuff like a web scraper API. They have a web data, like they have data sets that are already made for you. And if you're just doing web scraping at all, they have a bunch of services. Now what we can do is create a new scraping browser by simply clicking on add here and then clicking on Scraping Browser. Now, scraping browser includes a Captcha solver as well as connects to a proxy network. What that means is that it will automatically give you new IP addresses and cycle through those,

so that you can simulate as if you were a real user, kind of accessing a website. It also means that if there is a Captcha that occurs, it will automatically solve it for you. So you don't need to deal with being blocked by CAPTCHAs. So what I'm going to do here is just give this a new name. So we'll just call this AI scraper like that. There's a few other options, but we can just click on add. And once we do this is going to say yes okay. We want to go ahead and create this. And I'll show you how we can connect to this from our code. Now the main advantage for developers here is that this just works with the code that you already have. So in our case we're using selenium right. So we can just continue using selenium. But rather than doing the scraping on our own computer, we'll push that out to the cloud and use a remote browser instance, which is what this scraping browser is.

So it's rather than running the Google Chrome browser on our own computer, we let Bright Data run it for us, where it's connected to all of its tools, and it works exactly the same way. Now this works with Playwright, Puppeteer, Selenium, Scrapy, I believe as well, and then some other languages. So let me show that to you. Okay, so now that this is created, all I have to do is go to checkout code and integration examples. And if I do that, it's going to show me how to connect to this from our code. And what we'll do is we'll modify our code now to use Bright Data so that we're able to actually scrape at scale and to get past all of those blocks. So you can see that we have a few different options here like Node.js, Python, C-sharp, Java. Obviously, we're using Python. So we'll select that, and then we can choose what library we're using.

So in our case, it's selenium. So what we'll do is just copy the code sample that it has here. And kind of retrofit that to our current code. And you can also just test it, by the way, if you want to do that using this, okay. So let's go back here and let's just make kind of a new section of our code. Let me zoom out so we can see here. And you'll notice that really all we're doing is rather than kind of connecting to this WebDriver where we've downloaded ourselves on our own computer, we're connecting to a remote instance which is defined by this URL right here. So we have this speaker connection. We're connecting remotely to this remote browser instance. And then we can do the exact same thing that we did before. However, there's this code that we can uncomment. And this right here will actually solve CAPTCHAs for us. So if we think it captures going to occur on our page, whether it does or doesn't.

We can execute this command on the remote browser instance, and it will automatically solve that captcha for us. So you can see it will tell us what the status of that was, and then continue on with the rest of the code. Okay. So what I'm going to do is I'm going to kind of copy this that we've got, I'm going to remove the rest of this, and I'm going to paste that inside of here. So let's get rid of all this. Let's paste this code. And let's just take these three lines and put these at the top of our program, okay. So let's actually replace these with those because we no longer need them. And now if we go here, you can see that we have print launch in Chrome browser. We have this speaker connection. And then we are connecting as driver, and we're going to this website. So let's change example.com to say website. We can get rid of this first print statement.

We can run the Captcha solver. We can grab the page source. And then we can return the HTML this time rather than what we were doing before, which was what? Was it just printing it out? Okay. So now again, we have the exact same thing, except we're using Bright Data. Okay. So let's test this out now and just make sure that this works. And again, I want you to kind of focus on the fact that this was literally just a few lines of code change. Now we're just kind of using this rather than setting up our own WebDriver. And also, this is significantly easier than having to download the WebDriver, set that up, etc. So let me go back to my Streamlit app. Let's make sure that's running, by the way. So let's just quit this and rerun it just so that we refresh our code. And let's try to go to Amazon now. So we're going to go to amazon.com.

and let's see the results that we get. All right. So I'm just looking at the console here because the scraping has finished. And you can see that there's actually not a caption. Now even though I know it's a little bit difficult to navigate here. And we are actually able to access Amazon.com and grab the content from that site. Now, it's not popping up on our own computer because we're connecting to that remote instance. We could go in and debug that and view that if we want. But the point is, we're getting all of this content and we're not being blocked like we were before because now we've connected with right data. So that's great. Let's continue on now with the rest of the tutorial because we're able to actually grab this HTML source. Okay. So now that we have the HTML source, what we want to do is clean this up a little bit and make sure that we're not grabbing things like, script tags and style tags.

and all these other pieces of content that we don't want to pass to the LLN. The idea is we want to take just the textual content, pass that to the L11, allow it to parse it so we have the fewest tokens possible, and we can reduce the amount of characters or batches that we need to submit to our LLM to get a valid response. You'll see what I mean in a minute. But for now, I want to write a few helper functions that will kind of clean this HTML for us. So the first thing I want to do is write an extract body content function. And this is going to take in HTML content. And that's not what I meant to do. Let's go down here. What this is going to do is simply extract the body. So we're going to say soup is equal to beautiful soup. And we're going to actually import Beautifulsoup now, which is an HTML parser. So in order to import Beautifulsoup, we are going to say

from BS4 import BeautifulSoup like that. Okay. So now we will import BeautifulSoup. We are going to pass to this the HTML content, and we're going to specify that we want this to do an HTML dot parser okay. Next, we're going to say that the body content is equal to soup dot body. So we can actually just grab the body tag directly from this because it will parse it for us. And we're going to say if body content exists and I don't know why that keeps happening, then we are going to return the string of the body content. Otherwise, we are just going to return an empty string so that we don't potentially get any errors. Okay. Next, after this, we are going to clean the body content. So we're going to say define clean body content like that. We're going to take in body content as a parameter and we're going to clean it. So in order to do that, we are going to say again soup is equal to BeautifulSoup.

We're going to take the body content and then HTML dot parse here. So just read pass that again. And we're going to say for script or underscore style in soup. And then we are going to take script and style. Then inside of here we are going to say script or style dot extract. And what this is effectively going to do is it's going to look inside of our parsed content for any scripts or style, and it's simply going to remove them. So that's all this is doing, is just getting rid of those tags, because we don't care about the styling or all of the scripts, which are just unnecessary characters. Okay. Next, what we're going to do is we're going to say cleaned content is equal to soup dot get underscore text. And we're going to specify that the separator is equal to the back end character. So let's do that. So backslash I'm sorry this is the new line character which says

okay get all of the text  
and then separate it with a new line. Perfect. Now what we're going to do is say cleans underscore content is equal to and this is going to be backslash n dot join. And what I'm going to do here  
is write some fancy code that's going to effectively remove any backslash  
n characters that are unnecessary. So a lot of times when you grab the content from the HTML,  
you're going to get a bunch of empty text strings. I want to remove all of those strings  
so we don't have them in our text. So in order to do that, I can do the following. I can say line, dot strip for line in the cleaned content dot split lines. If line dot strip. Now what this effectively says is if a backslash n  
character is not separating anything, so if there's no text between it and the next thing,  
then we're just going to remove it. So that means get rid of all these kind of random  
backslash and characters

that aren't actually separating any text and just exist in the what do you call it? The, the text that we don't need. So stripping will remove that. I also remove any leading or trailing spaces. Great. And then after this, we're going to say return the cleaned content like that. Okay. The last thing that we're going to do is split this content up into batches. So what happens is when we want to use an LM, we have a specific token limit. Now that token limit is usually about 8000 characters. That means that if we have a really big web page, it's possible that the LM can't take the entire web page at once and parse all of it. So all we need to do is actually split our text into a bunch of different batches of whatever the maximum size is. So maybe seven batches of 8000 characters or something along those lines, and then feed those characters to the LM one batch at a time so that it's able to process them.

Again. That's because there's a token limit, and we're not able to submit every single character if we have too many. So that's what this function is going to do. I'm going to say define split Dom content. And we're going to take the Dom content. And we're going to take a max length equal to. And I'm going to set this to 6000. But I believe the max is typically eight. I'm just leaving it at six to ensure, we're not going to go over. I'm going to say return Dom, underscore content, and this is going to be eye to eye, plus the maximum length. And this is going to be for I in range. And then this is going to be a little bit of an advanced step. But I'll explain this. We're going to say range zero. The length of Dom content and then take the maximum length okay. So the way that this works is I want to create batches of 6000 characters.

So I'm saying okay. I want to take my Dom content from index I, which will start out at zero, up to I plus whatever the maximum length is. So in this case, it's 6000. So this would grab the first 6000 characters for me. Then what this for loop will do is it will step by the max length up to the Dom content. So after I grab the first 6000 characters, now I will be equal to 6000 because it will step forward one. So we'll then start at 6000 and go up to I plus the next 6000 characters. And we'll keep doing that until we reach the length of our Dom content. Okay, perfect. So that's it for this scraping. Now we need to connect these functions in our main.py. Okay. So how are we going to do that. Well first, let's import them. So we're gonna say from scrape import scrape website. We're going to import these split Dom content.

the clean body content and the extract body content functions. Now what we're going to do is call those functions and then kind of print out the result in Streamlit. So we can see what it looks like. So we're going to say the body content is equal to extract body content. And we're going to take the result. We're then going to say the cleaned content is equal to the clean body content. And we're going to pass in the body content. Then we're going to store this in the session for Streamlit. So we're going to say SD.dot session state.dot Dom underscore content is equal to the cleaned content. This way we can access it later. Then we're going to say width star expander. Now this is an expander text box that allows us to expand it to view more content. We're going to say Vue Dom content. And we're going to write inside of here that contents. We're going to say store text area.

And we're going to say Dom content. And then this is going to be the cleaned content. And we can give this a height of 300. So sorry. Let me clarify this. The expander is kind of like a button that will toggle what we're showing in here. So when we click on it, it will show whatever's inside of here. When we click on it again, it will collapse it. Then the text area is something that we can expand the size of. So we can view like a little bit of content or a ton of content. And that's why we give it a starting height. And then we say that this is kind of the title of the text area, and this is the content that we want to display. Okay. So now let's actually view this. So to do that, let's refresh our Streamlit app. Let's make sure it is running. And I believe it is. And now we can view my website again.

So let's do tech with Tim Dot net. Let's scrape this. It's going to take a second. And then once we get that content, it will show us that in the expanded view. Okay, amazing. So now we have this view Dom content window. And when I do this, you can see we get all of the textual content from the main page of my website. Perfect. So that is that. That's actually a lot of the hard work done. The next step now is to actually take this Dom content and pass this into an album that will pass it based on what we ask it to do. So now what we need is we need to actually ask the user for a prompt. So like, Hey, what information do you actually want from this site? And then once they give that to us, we can pass that a prompt as well as the Dom content to an element and have it extract that data. So we're going to say if Dom underscore content

is in the store session state. So if we've saved that, then what we're able to do is say parse underscore description is equal to store text area. And we can say describe what you want to parse. Question mark. Then we can say if t top button and this button can be parse content. And we can say if the parse description, then we can say st dot write, and we can write parsing the content. And inside of here, we can parse it. So in order to parse it, we're going to say that Dom chunks are equal to split. And this is going to be Dom content. And then we're going to take the St dot session state dot Dom content and pass that to it. Then, of course, need to pass this to an LM, which will be the next step. So we're going to take these as chunks, pass them to an LM, get some response, and then write that. That's pretty much it. But we need to obviously write the M component.

But anyway, so if we go back here, you can see now that if we refresh and we scrape the site again. So let's go here and scrape it. So check with Tim dot net. After we do this, it should pop up a text window asking us to enter a description, okay. So you can see now that it gives us this text box saying describe what you want to parse. And I got to fix that spelling mistake. And then we can write whatever we want in here and then parse the content. Perfect. So now we're going to move on to the Lem component. So we're going to make a new file here called parse.py. And let's start writing the code that we need. And then we're going to download Alama, which is what we're going to be using to actually execute the LLN. If you're unfamiliar with Allama, this is something that allows you to run an open-source LMS locally on your own computer.

so you don't need to rely on things like API tokens from OpenAI, and you don't need to pay. This is completely free. So what I'm going to do is let's import here. We're going to say from lang chain underscore Allama imports au Lama lm. I'm going to say from Lang chain. And this is going to be underscore core dot prompts. Import the chat prompt template. Now olam lm is because we're using a lama. But if you wanted to, you could use something like OpenAI. You could use Gemini. You can use whatever you want really for the LM. but Lang chain is something that allows us to kind of connect loops to our Python code, and that's why we're using it. Now, what we'll do is we'll say model is equal to Allama lln, and then we'll say model is equal to Lama three. Now I'm going to specify how you know what to put here in just one minute.

But I want to write all of the code first. Then we're not jumping back and forth between too many windows, okay. So now we have our model. This is only going to work once Lama is installed, which again, don't worry I'm going to show you in just a minute. But we're going to write a function now that will use this to parse our code. So we're gonna say parse with a lama. We're going to take in our Dom chunks and our parse description, okay. And then inside of here, we are going to create a kind of, what is it, a way of calling the LM with that code and the description. Now we know when we call an alarm, we need a prompt. Now we will have a prompt that comes here from Streamlit. So it's like, you know, describe what you want to parse, but we need to give it some more detail so that it knows what to do with that.

as well as the Dom content that we're about to pass it. So what I'm going to do is copy in a template here from my GitHub. So this is the template. Let's just make this a little bit smaller so you can actually read it. And you'll notice that inside of the template we have two variables Dom content and parsed description. If you aren't familiar with how templates work, all you can do is write a bunch of normal code in strings, which is what I'm doing. And then you can specify any variables that you want to be injected when you actually execute the prompt. In this case, we want to pass the Dom content and the parse description. Now I won't read the entire thing to you, but it pretty much says, you know, you're extracting specific information from this content. Here's a description of what the user wants, and then it just gives some, more specific instructions to make sure that, what do you call it here?

We get a decent response from the, you could improve this prompt. You can change it around, but this is one that I found worked fairly well. So that's why I'm copying it in here. Now, if you want to find that prompt yourself, you can go to GitHub. You can go to parse.py. And you can simply copy it directly from here, which is exactly what I just did. Okay. So now we're going to go into the parser the Lama function. So what we're going to do is say prompt is equal to chat prompt template. And then this is going to be dot from not messages dot from template. And we're going to pass the template that we wrote. We're then going to say the chain is equal to prompt pipe and then model. This is the straight-up pipe on your keyboard if you're familiar with it is on the far right-hand side, typically beneath the delete key. I just say that because a lot of people

aren't familiar with this character. It's usually with the backslash. Okay, anyways, now we have a chain. This means that we're first going to go to the prompt and then we will call the model. That's kind of how it works with long chain. We're then going to create an array here or a list called Parsed Results. And what we're going to do is pass all of the different chunks to our LM and then grab the results and store them in here. So we're going to say for I, chunk in enumerate and we're going to enumerate over the Dom chunk. So we're going to start equals one. What do you say start equals one. That just means that I will start at one rather than zero. So then we don't need to add one to it. If we're going to be printing out the value, it doesn't actually change the enumeration. It just means that I will start counting at one rather than zero.

Okay. Next, we're going to grab a response. So we're going to say response is equal to chain dot invoke. Now, this is how you call the LM. And when you call it, you need to pass two variables. And they need to match the variables that you specified here. If you change the prompt and add other variables, well, then you need to pass those as well. So we're going to pass the Dom underscore content, which is equal to the chunk, because we're passing one chunk at a time. And then we're going to pass the parse description, okay. And then pass description like that, okay. Now we have a response. And what I'm going to do is just print out some kind of logging information. So I'm going to say past batch. And then notice that I'm using an f-string here so that I can embed variables directly inside of curly brackets. And I'm going to say I of and then Len of my Dom chunks.

and put this inside a variable. The reason for this is so I know how many chunks are passing, because sometimes this will take a fair amount of time. And I want to have some kind of output so I know that something's going on. Then I'm going to say my parsed results dot append. And I'm going to append the response that I got from the Latin. Lastly, I'm going to return a backslash n dot, join my parsed results, which is going to take all my results and just join them with a new line character. Okay, so that is how this works, right? What we're doing is we're creating a prompt template. We're using that for our model. We are then just taking all of the chunks from our Dom. Right. So all of those 6000 character chunks and we're going to pass them in to the prompt. So we pass them as Dom content. And then our line knows what to do because of the instructions that we've given it.

As well as that parse description that we provided from the Streamlit UI, we parse one chunk at a time. We then take those, append those into the results, and then we join them and return them. That's it. Now obviously you can do this with other LMS, and the faster your computer is, the faster this is going to execute. There's ways to paralyze this, add asynchronous code, and to make this run a lot faster. But for the simplicity of this video, this is what I'm going to be doing. Okay, so this is great, but this only works once Allama is installed. So let me quickly show you how we install Lama. It's pretty straightforward. You're just going to go lama install on Google, and you'll see that you have a download. Oh Lama link. So I'm just going to click on this one right here. And notice that you have four options. Or three options are Mac, Linux, and Windows.

So obviously click the appropriate one and download a Lama. Now once Olam is downloaded, you need to run it. And in order to run it, you're going to open up your terminal or your command prompt, and you're going to type the Lama command. Now when you type Allama, you should get something that looks like this. And now what we need to do is pull in Allama model. So the Allama module is what we're actually going to be using. We need to download the model locally before we can execute it. So in order to do that, actually let's go to the Allama GitHub quickly, and I'll show you a bunch of different options that you have. So if you go to Allama GitHub, you'll see that if you scroll down here on this page, it shows you all the different models that you can use. Lama 3.1 Mistral Code Lama. You got a bunch of different options so you can pick what you want.

but notice the size of them, okay. So, obviously, the bigger the model you download, the more performant it's going to be, but the harder it's going to be to run. And it will specify down here how many gigabytes of RAM you need in order to run these various different models. For example, if you want to try to run, you know, Lama 3.1, this has 405 billion parameters. You're going to need a very intense computer. I don't even think my computer, which is an M2 Max, will be able to run that. So, pick an appropriate model based on the specs of your computer, and then you can run it. So, for example, let's say you just want to do Lama 3.1, okay. What we can do is we're going to go to here to our terminal, and we're going to type Lama pull. And then the name of the model, which is Lama 3.1.

Now I already have this. So I'm not going to run it. But you would run this code. It would then download the model for you. And then you just need to wait for that to finish. And then you're able to actually use it. Now in order to use the model, you can say lama run. And then something like Lama three. When you do that, it's going to give you a prompt window. You can say something like hello world. And then you can actually use this model. So I have Lama three. Obviously, if you have Lama 3.1, then you would change this to Lama 3.1. And I think I want to exit, so I can I do exit. Yes. And then that will get me out of that. Okay, I know I went fast there, but I'm just trying to quickly show you how to get a lama on your system. Now that are lama is on your system, we can start using this code.

So let's go back to Main.py and let's import that function. So we're going to say from pass imports. And then this is going to be pass with a lama. And now all we need to do is say result is equal to pass with a lama and then pass the Dom chunks and the pass description. Now the pass description is right here okay. So that's perfect. And then we can just say start right. And we can write the result to our terminal. That's it. So now let's test this out and see if it works. So we're going to go back here. We'll do tech with him dot net just to check my website. So let's refresh this and just make sure that this is working. So we'll go https colon slash slash tech with Tim dot net. And we'll wait for that to scrape. And then we'll just give it a prompt like can you describe this site or something and see what it tells us.

Okay, great. So we've got the Dom content here. So now I'm going to say, Hey, can you put all of the main titles from this page in a table for me? I don't know exactly what kind of result we're going to get from that, but let's see what happens if we do this. And you see that it shows us the main titles that are found, which actually are correct: tutorials, courses, community gear shop, donate software development program, myself, development program, and tech with tip. Boom, just passed that content for us. Now we can try with a more complex site. So let's use one of the real estate sites I showed you before. Okay, so here's a property finder site that just has a bunch of different properties from Dubai Marina. This is the same example we looked at at the beginning of the video. And this is interesting because I'm actually going to be moving to Dubai shortly.

So I might actually want to make a web scraper that can organize a bunch of these properties for me. Anyways, point is, I put that link here, let's scrape it, and then I'll give this a prompt and we'll see if it can organize that for us. Okay. So the site is scraped. And if we go here I'll just say can you please collect all the relevant information and organize it in a table. And let's see what that gives us. And you'll notice if we go back here, by the way, to the terminal, that it should be showing us the batches. So here it said, you know, past batch one of one, and this one actually let me just click it again. I don't think that button worked. it should pop up and tell us, you know, parsing batch one person, batch two, parsing batch three, as it goes through there. Okay. And there we go. That actually happened quite quickly.

And it gives us a table here of this property information suite. So, that is pretty much it. For this tutorial, I wanted to create a simple AI web scraper for you, and I want it to leave this so that you can extend it and make it more customizable to whatever it is you want it to do. Obviously, this isn't going to work perfectly, and there's a lot of tweaks that you could make, but you can imagine the potential you can have with something like this, right? You can use this to scrape real estate. You could use this to scrape e-commerce. You could use this to scrape really anything that you want. And if you knew the kind of website you wanted to scrape, you could customize this even more, give it a better prompt, give it more information, and you could even make this kind of go back and forth where you can ask multiple questions and just keep going and refining the result that you want to get. So much potential here.

With this type of AI web scraping, this really just scratches the surface. And I hope you enjoyed this video. If you did, make sure to leave a like, subscribe, and I will see you in the next one.

Python AI Web Scraper Tutorial - Use AI To Scrape ANYTHING

Dive into the world of programming, software engineering, machine learning, and all things tech through my channel! I place a strong focus on Python and JavaScript, offering you an array of free resources to kickstart your coding journey and make your mark in the software engineering and programming fields. My mission is to deliver the finest quality tech and programming tutorials online, ensuring you receive top-notch education right at your fingertips.