In this video, we will build an AI crawler that can navigate through almost any website. Then, it will retrieve and return the data in a well-structured form. All this will be done with my favorite automation tool, and you will never guess which one it is. But how are we going to achieve this? Let's break it down into details. Our crawler will use AI agent Noe in na10. It will be equipped with two tools. The first one is the URL tool, which will retrieve all the links from the specific page. The second one is the text tool to give the agent the ability to retrieve the text and the context of the page. With those two tools, the agent will be able to autonomously navigate through the website and decide which pages should be visited to retrieve the desired data. Let's say you want to retrieve the data from a specific page, for example, contact details. It can be a page like workflow.com. It's a really great page. Our agent will look through the links available on this page and navigate to the one that has the highest chances to have this.

information that you want to retrieve in this case. It can be, for example, contact sub-page. So, our agent will visit this page and try to retrieve the data such as phone numbers, emails, and so on. This sounds amazing, but what are the downsides? While this kind of crawler is very easy to set up and can be used in a variety of projects, it comes also with two major downsides. Since it operates on HTML code of the website, it's not able to perform actions on the website such as typing or clicking. And when it comes to the extracting of the data, it relies fully on AI, so we should be prepared to have a percentage of hallucinations or parsing issues. With that in mind, let's dive into the project and see how it works. All right, so let's jump to the first workflow that we have here, and this one is responsible for retrieving social media profile links from the websites. Here we have in the input and in the output super base notes, and basically, we will use super base.

as our storage here in the input table, we will have the websites that should be scraped, and in the output, we should have stored the enriched outputs. Basically, when I execute this workflow right now, you should see that the agent is using two tools. The first one is the text tool, and the second one is the URL tool. The agent is currently going one by one through those websites and trying to find those social media profile links. As you can see, it has been successful. So, right now in the super base, we should see the outputs. And yeah, this is how it is. We have here the three records that we had in the input table. However, those are right now enriched with the social media column. And we click for the first record for this CL column. You can see that those are the social media platforms that have been extracted from the table page. And the same is for the retool, for example. And also, we have the similar ex

execution output for um for na10 and we can we can easily retrieve those information uh from from the websites using uh the craing agent um however let's jump more into details of those text and URLs tool okay so this is the text tool and this one is responsible for retrieving the whole text from the specific page. This is very useful for the agent because it gives more context on the page that is currently visiting and what more is used for retrieving the actual data. So we when we want to for example retrieve the profile of the company it's also used um in this case uh so basically what I have here is I have pinned the address of my landing page which is workf close.com and yeah this is this landing page and what we want to do here is to retrieve the whole text as you may remember I have also on my YouTube channel the video about retrieving the text with Puppeteer. We have uh we do hear uh something similar here uh but 100% low code so basically here uh we have have the query from the

agent which is the domain, uh, and we want to retrieve right now the text. So, when I execute the HTTP request note, uh, you can see that, okay, I have the successful execution here, but this is the HTML. So, this one is, uh, quite unuseful right now for the agent, and it's very long. So, I need to convert it to the actual test text, and for this purpose, I use here the markdown note, uh, in na10, and, uh, when I executed this, uh, notes, uh, by giving it, uh, the input data, the HTML of the page, you should see that, yeah, I retrieved the whole text, uh, from this page, and it's very well formatted. Eventually, I need to send it back to the agent, so I used the response variable, and I passed it data, which is the formatted text. When performing the scraping, it's quite good to have some kind of proxy because our calls can be blocked by captchas or other security systems. So, uh, what I use here is basically a web unlocker from Bright Data, which is quite handy. It comes with

capture solving, and um, it's also very easy to set up in na1 because in this HTTP request node, basically, I create an endpoint with a username, password, and the host post of our proxy, which I took from The Pride data dashboard. And uh, we can see that basically when we execute this uh very simple workflow uh for the first call to the example page, which is amazon.com, uh we have the output for non-proxy puff that the call has been blocked by some kind of capture or other security system. And for the one that we used web unlocker, uh we have the full content of the page. So uh, with this kind of tools, with web unlocker, we should be able to uh increase the accuracy of our agent and simply scrape more websites. And this is the URL tool, which works very similar to the text Tool, uh the only difference is that it returns all the links from the page in the Json format. Uh so uh, what we have here is I have also pinned uh my website for an exit execution example execution, and until this moment, it works exactly the

same. The difference comes here because we use here HTML node in na10, and basically, this one is responsible for retrieving all links from the page. The thing is, it's a bit unclean, I would say, because we have also things like hash and other things that are completely not usable for the agent. So, we need to clean it a bit. For example, we need to filter out all the empty links that have been retrieved by this node, and also, we want to remove the duplicated ones. So, after those operations, we come with about 13 links from the page, and finally, we can aggregate them to send back to the agent. And as you can see, among those links, we have also those with social media platforms and so on, so this should be easily recognized by the agent. The cool thing in na10 is that we can connect many agents into one chain of action. So, basically, we can scrape a lot of more data than

just the social media links. Uh, so here is a bit more extended workflow, and I added two more agents. Let me start executing those agents, and in the meantime, I will tell you what they actually do. So, the first agent is the one that you already know. This is the social media agent, and it will return the social media links from the pages. The second one is the company profile agent, and it will create a quick summary of the company that the agent is currently visiting. And the third one, the third one is the app features agent. Because in the input database, we have three apps, which are Airtable, RLE, and NA10. They have some app features, so we may want to extract them and parse them in a JSON format, right? So, basically, those agents are currently working. The app features agent is working, and in a few seconds, we should have an output table in Supabase, our results of the scraping. So, as you can see, the agent

uh and the workflow is uh has finished the work, and the new rows have been inserted. And when we go to the super base, we can see that uh not only the social media column is uh filled, but also the company profile um records are present. And we have also the very well unified Json uh for up features for air table, for uh retool, and also for uh na10. So basically, those are the very uh handy uh handy things to do. And yeah, that's all for today. Thank you very much for watching. If you like this video, please give a thumbs up, subscribe to my channel, and to my newsletter. Link in the description, and see you in the next video. Bye.

Autonomous AI crawler

Autonomous AI crawler

Scripts