so I recently did a video on how to build your own web scraper and a part two in which we display the data that we scraped. However, I got a few comments asking how to scrape data of websites that blocked bots from running web scrapers. For example, Amazon can easily detect if an action is being executed by a scraper bot or through a browser by a manual agent. In addition to this, some sites make scraping harder by changing that structure, making your node.js web scrapers obsolete. What I mean by this is see how they have nested the title element in this div. Well, it is common for websites to change this up, meaning you would have to recode your node.js web scraper app and keep track of any changes made to the Amazon website. An additional reason it can be difficult to scrape a site is when pages are created dynamically, like in React, and will change based on certain variables or even browser sizes, kind of like for Airbnb or Etsy, for example. In this video, I'm going to show you two ways to scrape.

data from Amazon as an example. One is the free version, in which we build our own node.js scraper that will get us some of our data but will come with limitations, such as having to manually go from page to page and run the scraper over and over, but in time delays as to not get picked up as a bot, as well as have us keep an eye on if the website structure changes. This means that if we are using this data for an app, it could break our app. The second way is a paid way, but as a more professional approach to getting data that you might see at a startup, for example. It will retrieve the data back for us while accounting for any of the website's structural changes. It will also keep this data fresh, meaning our apps will be able to run smoothly without any issues. This second solution is produced by the video sponsor Bright Data, so that is the tool we will be using for the more professional approach. Heck, you could even use it to scrape literally every single item on Amazon if you wish so in

this video. I'll be showing you how to get all the products from Amazon search that will return the results from all the pages continuously using these few lines of code and also show you how to get to titles from an Amazon search of one page manually by writing some custom code with limitations. Okay, so what are we waiting for? Let's do it. Okay, so let's start off with building the Amazon web scraper in node.js fast. I am just on my IDE. I'm using WebStorm, but of course, if you're using anything else, just please get that up so we can start a new project. This is basically going to be us doing the manual version of the more professional approach that I mentioned that I'm just quickly going to show you. So please head over to the Bright Data platform. If you use that tracking link, that will really help them track how well this video is doing. It's not an affiliate link for me. It's just for them to know. I've already signed up, so here is my user dashboard, and then I am

simply going to create a data collector. So, let's click on that. Develop a self-managed data collector. I'm going to start with a template, so an Amazon template, and essentially use this code to keep getting the freshest Amazon data for us. So, that took barely any time, and at this point, we're ready to start collecting data. So, please go ahead and get to this point as well, so we have both sets of code up. Once you are at this point, let's continue. So, I'm just going to select new projects, and I'm going to call this Amazon web scrapper and create. So, there we go. Now, this is a node.js project. So, the prerequisites that you're going to have to do are install node.js. So, please go ahead and do that now if you haven't already. Just visit this URL right here, node.js.org, and please download whichever one you need. I am using Mac OS, so it's detected that. So, I would just go ahead and install this one. I've already done it, so I don't have to do anything.

so make sure you have that installed on your computers, and once you do, let's carry on. Now, the commands we're going to have to use first are npm init to initialize our project. npm is a command that will come with the package that we just installed. You might need to do some extra configuration to get this. If you are struggling with this, please visit this URL here and follow these instructions. I do this in a few of my videos, so apologies to those who already know how to use node.js. If you don't, hit me up in the comments, and I will be happy to help you out. Okay, so making sure we are in the directory we just made, so Amazon web scraper, just type npm init, and I'm going to ask you just to enter, hit enter, just use all the default values that they're giving. Is this okay? And hit yes, and this will essentially create a package.json file that will get created thanks to that command and put into your Amazon web scraper. So let's just go ahead and wait for that to

happen and great. There's our package JSON file and everything that we press enter to has essentially populated all these values. So our main file that we're going to be reading from is an index.js file. So let's go ahead and just create that. We're kind of working backwards by doing this, so index.js. You could have named it whatever you wish, but just make sure to change this file name if you choose to call your files something else. Now we're going to have to use two packages. So this example is actually given by Smashing Magazine. We're of course going to be adapting it to scrape a list of our choice, but shout out to Smashing Magazine. I don't want to take credit for this because I did figure out how to do it thanks to this article. So we're going to need two packages, as I said. One of the packages is axios, and another package that we're going to need is Cheerio. So Cheerio will essentially let us pick out certain elements from a web page and work with them. So let's go

ahead and essentially import those. So please go ahead and do npm install axios and Cheerio. So just using that command right there, that will essentially install these two dependencies into here. So there we go, along with their versions. So if you're having any troubles and you're watching this in the future, then perhaps revert back to these versions. You just have to change it like so and install them again by running npm install. I'm, of course, not going to do that, but that is how you do it if you need. So great, let's also write a script to start this. So start. And because we don't want to run this once as it's a tutorial, I'm going to want to keep restarting it. I'm going to use nodemon to listen out for any changes on the index.js file. So let's go ahead and install the package nodemon as that will help us listen out for changes in our index.js file and stop us having to re-run the file over and over again. So great, now we've installed these.

packages. Let's go ahead and write our function. It's a function that is going to help us fetch all these games from this URL right here. Okay, so I've just literally typed in games into amazon.com, and this is what returns back. We're going to get all the titles. Now, this is done by essentially inspecting elements. Okay, so I'm gonna inspect, for example, this title element, and I'm going to use the class names in order to find those elements in the document. This is assuming that all the titles have the same class applied to them, and I mean, they should, right? Because they all look the same. I expect a developer would do that, but just go ahead and check. So, that can of course be quite problematic because websites do change this up quite a bit. Okay, and that would render what we are doing obsolete. So, for example, if you're using this code for an app that you want to keep fresh all the time, you're going to have to figure out a way to, you know, check that your app isn't broken because this website

decides to change its structure because that is very possible. So, that is one thing that you need to look out for. So, let's write our function fetch games, right? So, I'm going to use a functional expression for this, and we're going to use try and catch. So, we're gonna catch any errors, and I'm just going to console error the error so we can see if there are any errors. Now, we're going to use axios to get essentially that URL. So, let's go ahead and provide this URL. This is the URL we want to scrape. Just make this a little bit bigger for you so you can see it's exactly the same. And this is an async function. It returns a promise, so we're going to have to await its response and let's save it as response. And as we've used the await keyword, we need to make this an async function, right? So, that we go. If you don't know what I'm doing, please watch my tutorial on async JavaScript. This is like a little mini-series that you

should be able to find on my channel. So once we get the response, well, that response data is essentially the HTML of that document. And now we're going to use Cheerio load to load in all of that HTML, and I'm going to save it as this variable. So this is just taken from the Cheerio documentation. Now we're going to essentially look for the class of each element. We're going to look for each element, okay, this whole thing, and we're going to pick the title. But having this code, you can actually pick out the price and everything else because we're going to grab this whole element by its class name. So let's grab all of this, and actually, we don't need all of this. So in fact, I'm going to take this class because all of these should have that. So let's check that out. We can specify that this is on a div if we wish, and let's just get this second class, so that one right there, maybe this one too, just because you never know how many items in here are gonna have

the same this next one as well, this one too, and this one right here. So, I think that should be enough. And now for each of those elements, so this comes with Cheerio. That is not standard JavaScript. We're gonna get its index. I'm gonna get each element, and then we're going to get that element. We're going to pass it through like so and save this under game. And once we get each game, we're going to look inside each game element and literally look for the class that holds a title, which is this. And it exists on a span, so it's a class that exists on a span element. So, let's grab the span, and then we're going to do that. We're going to get the class name, okay? And once again, just link up all the classes where they dot, so just like that. I'm going to get its text, great. And let's save it under the cons title. And next, we need to save all of them somewhere, right? So, I'm going to make an array like so. I'm going to use Games.

push to push in the title wonderful, and once that's done, we'll just return all the gains when we call the fetch games function. So, fetch games then, because this reads a ton of Promise, it's an async function, which is why we need to do some chaining games, just going to consult log out the games. So now, if we look in the terminal, let's get up our terminals. So here's mine, I'm just going to do NVM Run start, and ta-da, we get all the games. Okay, however, you will notice that it's just the titles from this one page. Okay, I would then have to essentially go to the next page like so, copy that, replace the link, but of course, after having to store this data. So already, this isn't ideal for if you want to fetch all the games from an Amazon search, but if you're happy to do it manually, this is of course a great way to do it. Just be aware of its limitations in case you are using this for something serious like building an app and not just a personal project.

All right, now let's do the same thing but with a continuous scraper that we don't have to manage or worry about the code or worrying about being stopped by a bot. So let's do it. So this time I'm just going to go over to the Bright Data platform. So here we go. So let's continue where we left off. And once the IDE comes up, this is the template code that will get us products. I'm simply going to search for games just like we did when I searched games on amazon.com. I'm going to hit preview, and then it should return all of that page for us in here. So there we go. So that's really all I need to do. I'm going to finish editing, and here are my outputs. So I don't just get the title; I get the search URLs. I basically get everything, the rating, review that comes with each Amazon product. Okay, and it goes over page by page as you saw on the code here. It's literally doing the pagination for us. Now to get this, I'm

going to initialize by API, and here we have the code for us. We are going to need our API token, but essentially Bright Data is going to give us all that data, and we're just going to access it with a token. So, let's go ahead and grab our token. Under settings, account settings, scroll down here, and create a token. So, let's go ahead and do that. That is my username. I'm just going to hit save, and it will generate a token for me once I put in my verification code. So, that was just sent to my email, and I'm just going to paste it. Okay, and once we have our API token, we could just replace it in this code right here. So, replace it here, get up your terminals, paste in that bit of code. As you can see, I have already replaced the bit that says API token with my actual API token. Okay, so this should come back with this, with an ID, and then we actually have to grab all that data. So, once again, grab this piece of code, replace the API token with your actual API token. Okay, so once

again, I've replaced the string of API token so the words with an actual token here, and now I'm going to press enter. Whoops, that was way too quick. Let's run that command again, and today we get all of the products, literally all the products under games. So, this scraper has not only got the products from the one page but actually gone from page to page and got all the products under the games category. So, there should be seven pages left. The main reason why the date is able to collect all this data in seconds is because it works through a proxy network, collecting and returning data from thousands of IPs at once. Cool. And of course, the benefit of this is that this code right here behind the scenes is completely managed by the team at Bright Data. So, for example, they always keep track of what exactly will return a product. So, that is one of the perks. You don't have to do this, and you can choose to get this data continuously, or you could choose to get this data manually. Of course, if you have an app.

you probably want to choose to get this continuously, and you don't have to worry about it. You don't have to worry about the code. You don't have to worry about being blocked by a bot. And you don't have to worry about Amazon changing the structure on you, because that's what we pay bright data to do. Okay, so hopefully that was useful. Once again, we have the way that we did it with our own code in node.js, but of course, we can only scrape one page, and we might be blocked by Bots, resulting in messages like this. And of course, it takes time, and we need to monitor this. Or we could use a paid version, which will continually seem monitor to the code for us, update it, and make sure that every time we make a call, we definitely get by products, and not just from one page, but many pages. So by now, you would have seen two ways to scrape data from a website: a basic way that requires a lot of love and attention, and a more professional way that is much more hands-off, as well as

fast. We also didn't cover that using tools such as this one can help overcome captures and web lockers, which is a big one, as well as help developers and companies with scalability and speed thanks to the ready-made templates and functions with out-of-the-box solutions. You also do not have to worry about proxy infrastructure as they take care of it, as well as provide you with structured data that is accurate and can be accessed via an API. And of course, we have already covered the coverage you can get using an out-of-the-box tool where we looked at the limitations of pagination with our custom code. All these points are crucial when building an app that you plan to sell that requires live data. This data can be used for an e-commerce site, a travel comparison app, for example, or even a web testing and AD Tech. Okay, so wonderful. I hope that helps, and I'll see you again soon.

🛑 How to Scrape UNSCRAPABLE data! (super simple!) Node.js + API

🛑 How to Scrape UNSCRAPABLE data! (super simple!) Node.js + API

Scripts