hi everybody. My name is Raphael. Um, let me tell you a little bit about myself. Um, I've been with Right Data for over six years now. I've been a solution consultant, solution architect, and I've been helping clients, uh, basically, uh, achieve their goals in collecting the data for a very long time. And um, I want to share some of the experience with you. And uh, the new thing is now serverless scrapers, uh, scraping functions as we call them. And uh, let's dive into that. So first, I want to go over the modern pipeline of what data collection looks like today. I mean, with years, things got much more complicated. I don't know if any of you have been scraping, um, you know, like maybe seven years ago. Like it was much more simpler. The antibot didn't exist. Pages were more open, and data was more accessible. Now there's a whole industry trying to block public data from being accessed by scrapers. So we are going to be discussing on how to actually make it work. And uh, the main goal here is how we can make it work.

with actually without having the infrastructure right. So, as we are talking about serverless scraping, right? So, the current data pipeline looks like this, right? So, first, obviously, you need proxies, IP rotation. It's very easy if you're having one IP. You'll access a few pages, and you're going to get blocked. So, proxies are a must. Next, unblocking, right? So, that's the second step. The website is trying to block you. It's trying to detect irregular traffic, Cloudflare, Akamai. I'm sure some of you are familiar with that. So, you need to unblock it, right? So, next, you need a browser. If it's a more complex text, you need to interact with the page, do a search, do an input, and then get the results, get the HTML page, parse it. Then, you need to qualify the data, right? Quality of the data is very important, especially if you're dealing with large scales, right? So, if we're dealing with a data set of a million records, 10 million records, 100 million records, validation is a must, right? So, process the data.

validate the data, then you have to integrate it, store it. So, it's just a very big chain of things that needs to be done, and I want to talk about the first three steps, right? So, site scraping is something that's replacing all of these steps and just making it much simpler. You don't have to have the infrastructure to support the browsers. You don't have the infrastructure to run the browsers on the servers, validation, processing, all of that is done in the cloud, right? So, it's literally saving you a lot of time, and the scalability, the reliance, it's much higher. So, let's start with evaluating different models of how the web data collection can be. Of course, you can do everything in-house, right? So, let's say that you know you want to hire a few dev teams, you want to buy servers, set everything up, maintain everything in-house. Honestly, I don't see a reason for that anymore. I've done a lot of scraping projects with big companies out.

there and even the biggest players out there. Nobody really does that in-house anymore. People are renting servers, right? So AWS, for example, uh, just there's no reason to build everything in-house. Like I always use this example. Imagine if you want to build a car, right? You don't have to invent a wheel. You don't have to create a wheel. Buy a wheel that's already been built, right? You're still building the car, but you don't need to actually create all the pieces. So in-house, it gives you flexibility because you have full control over everything, the server, the OS that you're using. But anything breaks, you have to fix it yourself. Anything goes wrong, you have to figure it out. So the maintenance is very high, and uh, it's a very deaf resource. Like you need a lot of developers to actually maintain this. Um, the next is hybrid. As I said, you don't have to invent the wheel, rent it, buy it, get it from somebody else. So this lowers the how many people you actually need on your team, right? Instead

of having Dev teams, you might have need one or two developers, uh, depending on the task, and basically you're using APIs and other scraping tools that are helping you unblock the websites, uh, and get the data right. So unlocking currently probably one of the hardest tasks, especially if you're dealing with complicated websites. Um, I'm not sure how many of you actually do scraping, uh, but for those that have built scrapers and have written code yourselves, you're probably familiar with spending maybe a day or two on just getting a response from the website. Sometimes it can take even longer. Sometimes you just might not be able to achieve it at all. Then when you do achieve it, and then you run it on scale, all of a sudden everything stops working as well. So hybrid gives you the option of actually using um other tools that help you unblock the website, and then you kind of more concentrating on already uh processing the data and uh validating the data more than uh making the website respond with the data. And the third one is that as a service in

this day of age, a lot of companies out there are scraping data, collecting data, and already have the data that you might need. So, this is the advice I give people all the time: if your core business is to process the data, don't create a department to collect the data. It's going to be more costly than buying it, and um, obviously, you will never be able to create a department as big as the companies whose core business is to collect the data. For example, us and Bright Data, we have a team of over 20 people to just unblock websites, right? So, we have an unblocking team that is working constantly over the, you know, day and night to make sure the websites are working, meaning that they're not blocking our requests. So, imagine just to have that, you will have to have 20 people for un, which is really ridiculous. If your core business is to analyze the data or do some kind of machine learning on top of the data, so just don't go there. Outsource it. Data as a service is the way to go. You don't need developers.

don't need anything storage, right? You need some storage where to store the data and then concentrate on processing the data. So, serverless scraping, right? In my opinion, this is the future of scraping. I personally, if I will have a project or when I have projects, this is the way I would approach it because A, it's easy; B, I can do it on my laptop; and C, I can just literally run thousands of browsers on my laptop. All I need to do is connect, write a code, and it's scalable, elastic, it's cost-efficient. A lot of times, I found that it's saving you money instead of wasting money on developers, wasting money on servers, infrastructure, and instead of like, it just saves you a lot of money, and it's simple. It's very simple. We're going to talk about some of the products that can literally create a collector in 20-30 minutes, and we will, I will show you an example of such a thing. So, the other day, I was reading Reddit, and I

came across this, uh, this thread, and it was, it's true, right? So, this guy was trying for two months to, uh, scrape by using Lambda, and after two months, he just gave up. Lambda is an amazing place if you want to do simple things right. It's, uh, it's a cloud infrastructure where you can run scripts, but when things get more complicated, when you need to run a browser, when you need to keep a session, when you need to have functions interact with each other, it becomes a mess. So, as scraping, like serverless scraping, is amazing, Lambda actually gives you a little bit of a headache. So, if you want to go serverless scraping, I do not recommend it. And as you can see, and if you read about it, there, it gets very complicated on a large scale. I just wanted to throw it out there that, uh, you know, if you like happiness and work-life balance, don't try to do it. There are better ways of going about it. So, orchestrating unlocking and interaction is very

complex F. This is basically the diagram of how things should go right. So, you start with your input. It needs to have a browser fingerprint, right? So, you need to make sure that it looks like a real browser. Otherwise, it's going to get blocked. Referral headers are very important for those who don't know what referral headers are. It's basically a header that tells the website where you came from. A lot of times, if just by adding, let's say, a referral from Google search, it literally increases the success rate. I've seen adding google.com as a referral increasing success rate by 20-30%, just as simple as that. Handle cookies. Cookies are very important. Session cookies, different authorization cookies. It's important to get those cookies. So, sometimes just going to a homepage, loading the cookies, and then going to the product page can increase success rate. Managing user agent. User agent is the most important part because if you're not putting a user agent, then most likely you're using either a default user agent. If you're sending a

request like using a library like Axium. Then the user agent will be Axium. If you're using a Chromium user agent, it will be Chromium. So, you need to make sure that the user agent matches the real browser if you don't want to get blocked by those detection systems. Different G GEOS, right? Uh, you want to see what the website looks from different GEOS, or maybe the website is geo-restricted, so you need to make sure that you are using the correct geo. If after all of this you still got a captcha, you got to solve the captcha. There are websites where captcha is 100%. It doesn't matter what you do, you need to solve it. Retry with a new IP, by the way. It's in cases where captcha is not always. Sometimes retrying is easier than solving the captcha. So, my opinion is if you get a captcha, try with a different IP, see if it helps. Then you need to render JavaScript and uh, interaction script. So, for example, if you want to do a search, you need to do type, you need to

bring out some kind of a table that you want to scrape. So this is and to orchestrate all of this. It's a very complex test just like uh we saw oned before on Lambda. It's practically impossible because there's so many different uh uh moving parts and uh this is where I want to introduce scraping function. Scraping functions is basically a seress infrastructure uh the bright data created in order to help people collect data right. So we basically took everything that we know about proxy, everything we know about unlocking, and we put it into one place uh to basically help you create scrapers fast, cheap, reliable, and scalable. So you run your scrapers 100% in the cloud. That means that as I said before, you can literally do it on the go, on your laptop, connect to the IDE, create it, run it, execute it. Uh there's pre-made templates to speed up the development process so you can load up a template, edit it, modify it. Uh automatically when you create a collector, you get a dedicated

API with it, meaning that once you build a collector, you can initiate it by sending an API call to it, and of course, built-in proxy unblocking infrastructure. I think this is the number one thing that this product has because when you don't get blocked, collecting data is so much easier. Autoscaling for concurrent sessions, meaning that if you want to scrape a thousand pages at the same time, it's possible. And of course, live preview, run logs, debugs, and more. So how it works, it's pretty simple, right? So we have the ID where you are developing the scraper, the crawler, and on proxy and unblocking infrastructure is already built into it. You also get a monitoring dashboard, so you'll be able to monitor success rate, you'll be able to see the history of what has been done, how much you've been spent per job, different types of configuration, delivery configuration, output configuration, and I will show you that live right in a bit where we will actually build a collector, review a collector together, and of course, each collector has a delivery preference.

whatever you are working with, uh, we're supporting, uh, pretty much everything out there. So, scraping functions, key components. We have an interaction code where you're basically interacting with the page, get post, and you're navigating, you're clicking, you're typing until you get to the data that you want to collect. And then we have the parse code. It's basically once you have gotten to where you want and the data is there, you trigger the parse code and, uh, parse HTML. Uh, it supports jQuery, so it's pretty simple to write a parse script. Uh, also, we're supporting multiple stages. If you want to run, write a spider crawler, right? So, you start from the main page, you collect all the URLs, then you open up all the URLs, uh, and then you go further and further. You can create multiple stages, and, uh, this gives a very huge flexibility for building Crawlers on our platform. Different types of Agents, right? So, for example, some websites you need to interact with, and then you need a browser, but some websites you just want to pull the HTML and parse it, right? So,

code worker is basically like sending a curl and getting the HTML back. I don't know how many of you are using our products now. Unlocker is very similar to what this is. It's basically a request to the server: bring me the HTML back. You get HTML, parse it, it works very fast, and obviously, it's the cheapest solution because you're not using so much resources of the servers. But of course, there are websites where you need to interact with the website, you need to solve the capture, and so on. So, browser worker is for more sophisticated jobs that require more interaction with the website. Built-in scraping frameworks, right? So, our IDE has built in so many different functions, and I'm going to show you some of them, but some convenience functions that I think they're amazing. Close popup. So, at any point, you can specify what is the selector for the popup. At any point of time, if that popup pops out, it will automatically close it. Wait for visible.

element, it's standard, right? You're clicking, you want to wait for something to load, wait for it to be visible, and one of my favorite ones is solve capture. Out of the box, we're solving about 25 to 30 different captures just by typing self capture. So, if you navigate to a page and you hit capture, all you got to do is add a command self capture. And so, basically, the key benefits are it's a scrape, it's a rapid scraping development. Like, you can literally, I've built scrapers in 20-30 minutes. The longest part is usually the parser, right? So, if you want to extract, let's say, 100 data points, there's nothing really to go, you know, you still need to write every line. So, name is selected, that title is selected, that price is selected, that. So, this is usually the longest part, but we have chat GPT, we have different types of AI that can literally, like this, help you create a parsel code, built-in proxy management, reduce maintenance, and infrastructure.

cost, and now I want to go into the live demo, and I want to show you exactly what I'm talking about. So, this is my demo account. I'm not sure how many of you have accounts with us. I hope all of you do, because it doesn't really cost you anything to create an account. You can already build these scrapers without really paying anything, right? Add your card, get a bonus, work with it. So, I have only one collector here. It's an Amazon search collector. And let's go over a little bit what the in like what the interface is what we're looking here at. So, once you build a collector, you will have some something like that here. We have output configuration, we have delivery preference, we have different properties, who created, and so on. We have statistics from the previous runs, and we're going to jump in that a little bit further. I want to go into the code, and let's review how simple it is to build an Amazon collector that basically does the following. And I just want to show you really quickly on the

website. So what this collector does, it goes to amazon.com. We do a search, for example, Xbox. In this collector, I will be generating this URL here. There's no reason to really go to the homepage; we can literally navigate here. So I'm generating this URL, then I'm checking how many pages of the products we have here. Then I'm opening up all those pages, and I'm collecting the URLs of the products. And then I'm going inside each product, and I'm extracting the data from this page. And this is how it works, right? So the first step is product search. I want to review each line by line. So country US. This is how I specify that I want to use an IP from the United States. If at any point of time I want to use a different IP, for example, GB, Great Britain, or anything, this is basically all the proxy and unlocking. It's one line of code. We're going to use US next. It's for those who are not familiar with JavaScript. I do apologize. Our coding face currently is only in

JavaScript. Uh, there is uh, on the roadmap, other language support, but it's very simple. You don't really need to be a professional uh, JavaScript developer in order to use this interface. It's kind of like, you know, it's more like English, right? So, and you don't really need all of this. This is more for just like uh, the pagination. I made it a bit more complicated than it could be. It could be much more simpler, but here what we do is we're creating the URL, right? So, as you can see, it's with- s. Let's go back to the search page, right? And then we have the keyword, right? So, here I'm creating the K with the input search. Here I'm creating search for iPhone, right? This is my input variable. And then I'm just basically navigating to the URL that we built. So, this URL here referral, remember always use a referral, because if we are going to the search page and you check the headers of this request, you will see that the referral is from amazon.com. That means

that we came from this from the amazon.com page, right? So, if you do a search here, referral is amazon.com. It's important, increases the success rate everywhere. So then, I check the max page, right? So here, I need to parse from the search how many pages there are. So the only thing that I want to extract from this page is this component here, how many pages there are. So in jQuery, I'm using the selector to basically extract how many pages, and if there isn't, I'm just doing one. And then I'm creating a loop in which I'm sending to step two, right? So next stage here, it's what data I'm sending to the next stage. I'm sending more pages, right? So more pages is the new URL that we are creating. So let's just give it an example. So let's say iPhone, and I want to do four pages here. We're running a preview, you can see it's navigation, it's parsing, boom. So children is what goes to the next step, step step two, and here we have page one, page two, page three, and Page

four. So let's go to page step two. So this step right now is receiving four inputs. That means it opens all four URLs at the same time, right? So we're not doing it async where it's one after another. All four opens at the same time. So what we do now is we are checking for the product URLs, right? So currently, we are on this page. We have four open: one, two, three, four. And we want to collect all the product URLs. So all product URLs, we're creating an array, and we're mapping basically all the products that are on that page. So for example, let's open up page one, right? So we're navigating to page one, and we are extracting all the products that are on that page. And again, please pay attention to this. This is six lines of code, including brackets. This is another eight lines of code. This is nothing. It's so simple. You can create all of this in Visual Studio, but the amount of work that you will need to put into it is so much higher.

just really no point. So, here we have already all the URLs of all the products, and we can just, you know, we can actually take a URL and navigate to it. Boom, product URL perfect. So, what we do next is we are navigating to the page, we're extracting the URLs of all the products, and then I'm creating a for loop and sending these URLs individually to the next step. So, if in the four search pages we're getting how many results per page, we're getting 16. So, here we got 16 results, right? So, four times 16 is 64, and basically step three will be open 64 times at the same time, right? And what is step three? We're basically navigating to the product URL and parsing from this page. I didn't really spend much time on the parser code. I mean, the goal of this webinar is not to teach you how to choose the right selectors, right? This is something, again, you can use AI these days, just chat GPT. You can literally

ex take this HTML page, throw in chat GPT, tell hey can you build me a parser to extract ABC, and you can then paste it in here. It works simple um so here we're basically I only set up title price and description and again let's check one of those inputs that were from the previous uh steps s to this to step three it's taking a bit longer let's try it again okay so this took a little bit longer uh again we're using real residential IP so sometimes a proxy could be a bit slower than usual but on average this should take a few seconds so here we already have the title we have the price we have the description and uh let's just review it quickly again we have 12 lines of code with three lines of code 15 we have another six and eight and we have another it's literally like 40 lines of code that does the following search goes to every page goes to every product and extracts data for those who have built scrapers in visual code or any other uh coding.

interface can see the benefits here. This is why you can build a collector in 20 minutes. Now, let me give you a little more detailed information on what actually happens here because all of these functions might seem very similar to you. For example, we have click, we have type, and we have everything else that you might need to collect data. But when you do click, let's say in Selenium or Puppeteer, the mouse instantly teleports to the selector, right? So, there's no mouse movement, it's a teleportation. If you're dealing with a website that is using trackers, I'm not sure how many of you are familiar there, but there are websites that literally send a packet of information every few seconds, even if you're not doing anything, right? Amazon doesn't really do it, but there's other websites out there that you will see requests being sent. Oh, here we have something coming in, but that's not the packet, but it just keeps coming in, coming in, coming in, and you're not doing anything, and you're wondering, What is that? This

is basically the website recording what you do, your mouse movement, how long you spend on the page, how long you scroll down, and so on, right? So, it's kind of like checking your metrics. So, when you do click in our IDE, it's basically, it has a lot of pre-recorded different mouse movements to imitate a real human type, same thing, mistakes, typos, delays, different types, speed, and so on. So, when you're writing a code in this coding interface, it imitates a real human behavior. So, for example, all these steps that you have here will not be executed at the same time. There's small delays between them. So, if you're dealing with a website, sophisticated website that is using ACMI or something that is literally like trying to on-scale monitor, okay, this is, you know, the behavior of people, and when they see thousands of people doing the same behavior, it thinks that it's a bot and it blocks us. This one is randomizing everything, so it's an amazing tool to bypass blocks. It's an amazing tool to quickly create scrapers, and it's just

and you don't need anything, just your laptop, right? Build it here and then execute it outside. So once we build the collector, we save it. Let's go back to the other window before. This is what we get, so output configuration, right? Since we don't have a huge parser code, it's a bit simple here, but we have the input parameters and we have the output schema. Let's say at one point you want to change title to name. You don't really need to change the code. All you need to do is you go to Output schema and change it to name. Let's say you want to change price. Let's make sure that it's money and let's make sure it has a dollar sign, right? So boom, just like that, the price will now have a dollar sign in it. And description, right? So you can literally modify the output, how it looks. You can also add additional data that we have pre-built. You want a timestamp? Sure, let's enable the timestamp to make sure you know when it was scraped, right? So you want to see the

input as well. Let's enable that. Um, you want a screenshot? Sure, let's take a screenshot of the page. You want the HTML of the page? Sure, let's do that as well, and so on, right? So just like that, I added to extract HTML, I take a screenshot. It's very simple, and it helps you work like it's just, it gives you flexibility on what to do, and it's fast, it's easy, and you don't always have to open it up. Add code, codes of line modifications can be done here also. You have the delivery preference, so right out of the box, you have different formats that the data can be delivered today. You want CSV too, or you want a JSON, just by clicking these buttons, and JSON, EX, whatever the standard, the format that you want is easily switchable by clicking these buttons. We also have two types of collectors. We have a real-time collector or on a job. So what is the difference? The difference is very simple. Let's say that you have a platform where

people interact, click, and you want to have a button that sends an API call to our collector to extract the data and send it back as soon as possible. This is real-time, right? So, the collector is on standby, always waiting for input. As soon as it comes in, it processes it and provides you the data. Or you have, I don't know, a list of 1 million inputs that you want to just throw in there. Once it's done processing the data, it gets delivered to you, whatever the workflow is. This is it. If you're working with a batch, I recommend splitting the data, right? So, if you get, I don't know, a 10-gigabyte file with 100 million lines, it might be a bit hard to process it. A lot of computers just can't handle it. So, you can actually split the data in batches of 10,000 records instead of one file with 100 million records or whatever the preference is. API delivery, right? So, obviously, you trigger The Collector with an API. You can download the data.

through API that works, but let's say you want to create a web-like automation where a script is executed on a data delivery. So, let's say that the data is done collecting; it gets delivered to your webhook, and then you have a script that processes it, does some kind of machine learning, and so on. Or it can trigger another collector, right? So here you can build a whole flow of different collectors doing different things on top of machine learning. We have clients with very complex setups where one collector collects one data, it gets processed, then there's another API call to collect another collector, and then in the end, they have a whole big data set that they build with different collectors and INR from different sources. So you can do this as well. Everything is very simple: set it up webhook. We have Amazon A3 if that's what you prefer, Google Cloud, Microsoft Azure, we have SFTP, even Alibaba Cloud, right? So if something is not here, which I really doubt it, we can always add it, right? We are always trying to cover all

the types of uh storages that people work with. I think this pretty much covers everything. Obviously, we have different types of. We have a development and we have a production stage. So, if you are in the production and you want to work with the The Collector, and you want to experiment with it, you can set it as a development stage, and then you are working with just the development mode. It's not in the production. The production is running on the other version, and then once you're ready, you publish it to the production. Uh, different notifications. All of this is automatically built for you, just so you understand, right? In order to build all of this in uh, in a code, you can develop all of that, right? But it will be thousands and thousands of lines of codes, and it would take you to develop a y. Here, it is already available for you. So, obviously, you can initiate it by API. Here's an example of how it's done, right? So, basically, you're sending in the data, the parameters that we need to do.

search here. We have two parameters, as I showed you before. We have the search and how many pages we're interested in collecting, and that's pretty much it. Once we trigger this, you get a job ID, and then with a job ID, with a collection ID, you can actually extract the data, uh, if it's, uh, downloaded via API. Here we can see previous runs. I've run this collector a few times, as a matter of fact. Let's give it a try. Let's run it. I want to collect iPhone, two pages. Let's do Xbox, three pages. Um, let's do Samsung, four pages. I don't know, let's PlayStation, five. Let's do one page, right? So you click Start. We can see here we have four inputs, right? And let's see, it's going to start with four pages, right? The first step is four search pages. Then we're going to add up on the pages how many pages I selected here. You can download, by the way, the input file, just in case if you forgot what you ran it with. And also, while it's doing that, we have a

subscription. Let's say that you want to run inputs every day, right? So, 9:00 every day. Select the calendar, pre-select what inputs you want, and create a schedule that the collector will follow. It's pretty simple. If you want to run it once a day, it works. If you want to run different inputs on different days or times, then my recommendation is to utilize the API. There's a lot of other services like trigger API trigger services with the timers that you can utilize. And let's see here, obviously versions, you can report issues. So, we're here to help you if you run into any problems. We have records coming in, failed crawls. Don't be afraid of failed crawls. Pages do fail, but we have a retry mechanism under the hood that automatically retries failed requests. So, as you can see, it's collected 26 out of the 38, and these two fails will be retried once the circle is done. We can already preview some of the

data with a quick view. So, here we have, uh, I didn't remove the spaces in the description. The description has, uh, uh, empty spaces, but you can add that Trim in the code if you want. Let's quickly review how it's done, uh, for those who know jQuery. Oh, we do have trim. Maybe we need to adjust it a little more, uh. I would assume that because I'm extracting this right and the structure is paragraphs, right? So, we have, it would be probably better to extract it as, uh, an array of objects. Again, I'm not here to teach you the parser. The goal here is that we can get to the data without getting blocked. We can get to the data fast. We can scale it very fast. And, uh, just let's go back to statistics really quickly. We already got 38 records. There's more 90 records. It's collecting. It's working. This is the goal here, right? So, I literally had projects where a client needed data fast, in half an hour, on the call with him. We build a collector. We ran it, and we collected.

million records within a few hours, and this is the goal. If you have a project and you needed the data yesterday, and I'm sure a lot of you have been there where I need the data yesterday, this is an amazing tool to get your data fast, and it's not expensive, right? So, in a lot of cases, I've seen this come out cheaper than using a residential proxy with a browser. It's because here you don't pay for bandwidth. The price is per page load, just for you to understand, right? Page load. So, do you know how much a page weighs in on Amazon? So, let's quickly clear cache and cookies, and let's look here how much bandwidth is actually transferred when you are opening the page. Already 10 megabytes, and this is just one product page. 12 megabytes of data, 13 megabytes, 14, and so like, it's a lot of data. All of this would be one page load. So, in a lot of cases, it comes out cheaper, and so basically, this is the

product I recommend. You can register with us, play with it. It doesn't cost you anything to create these collectors if you're going to be running them on scale, of course, yes, but to create, to run the code, all of this is open for you. You can see the preview here. Remember we spoke, I told you about browser and code worker. Here's where you choose that. You can take a screenshot. You can see everything. Strongly recommend, especially for people that like to create collectors. It's an amazing tool that just makes it work. So, I also want to mention a couple of other things that Bright Data has for people that don't really want to be involved in creating the collectors. We have scraper APIs already pre-built for a lot of websites. So, you can literally send an API with an input and get the data. We have data sets where you can actually already purchase the data set. So, don't always need, you don't always need a department to collect data. Look out there, there's a lot of sources where

you can purchase this data. You have different types of tools that you can use. You don't need service. You don't need infrastructure. And it's just more reliable because when you have a team fully dedicated to collecting data, it's much more reliable than when you have a team to process the data that's trying to collect the data. It doesn't match. So at this point, I do want to open up our Q&A, and I want to see if there's any question questions that I can address, something that can't come up, or something that hasn't been addressed. Um, guys, anything that should be addressed? Let's see. Um, so there's nothing yet unanswered. I'm sure people are inputting new questions as we... There we go. H, what are the regulations that one needs to be mindful of when scraping? So one part I did forget to mention, and thank you for bringing up this question. So under the hood, our coding interface and um, let me stop sharing my screen so you guys can see me, and let's have a

little conversation um under the hood. Our ID has built-in limitations, right? So, we are constantly monitoring the websites that we are scraping, or you are scraping through our coding interface, to make sure we do not do them. So, by using our services, you're already making sure that the website is not going to get damaged. We're going to slow you down a little bit if we're seeing that the response of the website is dropping. And, uh, you can only collect public data, right? So, this is the main thing. We are only helping you to collect publicly available data. So, that means if you will try to use our service to log in, we will block this request. You will not be able to access private data. So, basically, when you're using this, you can't get in trouble. We are preventing you from doing something that can get which cause you legal issues. Uh, can is the preconfigured scraper for Zillow. Can it be modified? Um, I'm not sure which templates are available. I'm sure I know that there's a few out there. I just checked. We do have

perfect, so there is a Zillow template that you can start with. Um, keep in mind, guys, templates, they do get updated from time to time, but it doesn't mean that it's going to work out of the box. Mainly, selectors change, so if you load up a template, check that the selectors are still what they are. But overall, the flow can be, should be the same. How does the cost running server scrapers compare with more traditional server-based solutions for high-volume scraping? So, okay, um, in order to have a server scrapers with like a um scraping with servers, and you need to run a browser, you need to get, I don't know, an EC2 instances, maybe five, ten of them, depending on how many browsers you want to run. I mean, how many of you have tried running a Puppeteer Selenium on your own computer? For example, on my work laptop, I can run about 15, 20 browsers. I've checked that with 15, 20 browsers collecting 100 million records is going to be forever. I wouldn't even try.

so now, in order to collect that much data, you need to have at least a thousand, maybe 2,000 browsers. So now, you need stronger instances, and they cost a lot of money. So just by needing the servers, you can be spending a few thousand dollars on just the servers, without the proxy, without any unblocking, without any development, just renting the servers because you need to process a lot of data. Serverless scraping removes all of that. You're basically paying, as I said in our example, per page load. Even without the servers, it sometimes comes out cheaper versus using the proxy, right? So I compared it. I didn't even touch the infrastructure, right? I compared using residential proxy with this interface. We're not even talking about the prices, so it comes out a lot cheaper in the end because you're only paying for page load. In our instance, I'm sure there's other providers that are charging you for maybe bandwidth and so on, but with us, you're just paying per page load.

one page pay for one page. Doesn't matter how many requests on a back, so it comes out a lot cheaper than having to have this whole structure, infrastructure with developers, with maintenance, and then when something fails here, you open a ticket there, you pay a developer to spend a week figuring it out. So, it's again, it's just not. If your core business is not collecting data, don't start creating a whole department to do it. It's just not cost-efficient. Do you have scrapers from medical lectures, Google SC color? Um, I do. I do not know if we have. I do not believe we have any preset scrapers. Uh, again, you can build it yourself or we can build it for you. We also have a service where we can build these scrapers for you. So, feel free, Reach Out, open up an account. We can help you collect the medical data. Um, some of the sites we mine data from are actually paid subscription sites. We must sign in to access the data. So, unfortunately, when you sign in, you

accept terms and conditions and this becomes already private data. Bright data does not help you collect private data. This is not something that we will be able to assist you with due to regulations and the GDPR compliance and other regulations. We just don't go behind login, sorry. LinkedIn scraper, yes, publicly available. We have a template, feel free to create load. As a matter of fact, you can literally just click navigate to a URL of LinkedIn and it will open it because the unlocking technology is so powerful. You don't get blocked. So, if you want to collect LinkedIn public data, all you need to do is country us navigate input URL. The input URL is a profile or a company, and it will open it up and load it up for you. Just then parse it as you want, really. I think our time is up. Thank you so much for joining us here today. Thank you, Rafa, for taking the lead and giving this awesome demo. And if anybody has any follow-up questions, you can reach out to Rafa on LinkedIn.

find him or send him an email rafhael bata.com, and I'm sure he'll be happy. There we go. Can the QR code? Don't be a stranger. Um, everybody, thank you for coming. Appreciate your time, and I hope you find it useful. Uh, register, open an account, play with it. As somebody who writes code, I wrote a lot of scrapers in Visual Code. I enjoy building collectors in IDE. It's fun because the most frustration you get is from blocking, and when you don't get blocked, it's just a different experience. Give it a try, really. Thank you, everyone. Have a good day.

[Webinar] Deploying Serverless Scrapers - Web Scraping API

[Webinar] Deploying Serverless Scrapers - Web Scraping API

스크립트