Building a Simple Email Scraper with Python Scrapy

The scraper effortlessly extracts emails from any website, enhancing email automation for local businesses.

### Building a Simple Email Scraper with Scrapy
* 📧 This tutorial demonstrates how to create a straightforward scraper to **extract email addresses from any website**, particularly useful for those seeking to **enhance their email outreach to local businesses**.
* 💻 The process begins with a simple Scrapy script that takes a website URL and retrieves a **list of email addresses** found on that site, even if they are **not visible on the homepage**.
* 🔧 To set up the scraper, users need to install Python and Scrapy, then create a new Scrapy project called **"email scraper"** where the spider scripts will reside.
* 🌐 Using Google Maps data, the video shows how to **identify and input URLs of businesses** to scrape for email addresses, demonstrating the process with a **specific example**.
* 📝 The scraper is designed to **avoid duplicate emails** and can be modified to **crawl all pages of a website**, ensuring that it captures emails that may be located on **contact pages or other linked pages**.
* ⚠ Caution is advised when scraping multiple URLs to avoid IP blocking; using **proxy servers** is recommended for **extensive scraping tasks**.
* 📊 The tutorial also includes a promise to **expand the script's capabilities in the future** and mentions additional resources for **scraping LinkedIn profile URLs**.
* 🔗 Viewers are encouraged to **access the code provided in the description** and engage with the content by **liking and commenting on future automation requests**.

I'm going to show you how we can build a really simple scraper to extract any email from any website. I get a lot of people reaching out to me with URLs they get from either Google Maps or LinkedIn, and they want to be able to figure out

how they can get the email off the website that they just scraped. So this video will show you just that. This is great if you want to expand your email automation efforts and reach out to more local businesses. I have links to where you can get all the code down below, and

also, if you want to work with me in any kind of AI automations project, make sure to book a free time down in the description below. But anyway, let me show you how this works. So how this scraper works is I basically have this simple scrapey script here, which allows me to

put in a website URL, and then it will just pop out a list of emails. So I can go over to my Google CSV data with a bunch of Google Maps data here, so we can come on over to a website here. We'll grab the domain, I'll paste this into our

start URL here, and then I'll copy over the allowed domain right there. And if we give this a run, you can see it'll go through through all of the connected domains to see if there is any emails, and then if it finds anything, it puts it

in our little emails JSON here, so we can see we got this Susie email from her business right there. We can scrap basically any email from any website, if it exists on their website. You'll notice here it doesn't even show the email on the first page, it's getting it from

somewhere else on the website so let's get into building this thing I'm over on VS Code and we're going to be using a series of Python libraries in order to get this to work and the big one is going to be Scrapy, it's quickly

becoming one of my favorite Python libraries to scrape stuff off the internet and this is what we're going to be using to build our website scraper to get started let's head on over to ChatGPT and see if we can build out this scraper so

I'm going to say make me a simple Scrapy script to scrape any email off a URL and then put it in a JSON file I'm on ChatGPT 4 for this and we're just going to send it so Scrapy is very similar to

other frameworks like Beautiful Soup or Selenium or Puppeteer, I talk about a lot of the other stuff on my channel but for my experience Scrapy has been one of the fastest scrapers to use especially when trying to crawl stuff on websites so we're going to use this

really simple scraper here um but first let's kind of follow the instructions in order to get this set up on our machine so if we don't already have Python installed first we got to make sure we have at least the latest version of Python over on the Python website here and

then we're going to want to make sure we install Scrapy with Pip install Scrapy so I'll just paste this in my VS Code terminal in here and it will download Scrapy for us and in the meantime we can come back over here and we need to make

a new Scrapy project so we could do Scrapy start project email scraper awesome so now I'll just come into here and we'll just paste in this here and there we go we have our new email scraper folder so to get into this we're

going to CD email scraper and enter in there so now we're inside of our folder and then we have to navigate into our email scraper and into our spiders directory so if we go into our email scraper folder here our spiders directory is where we're going to put

our web scrapers that we're going to run we have to touch any of these other python files unless it specifies us to so we're going to want to put our new scraper inside of our spiders folder so I'm going to come back over here and I'm

going to copy this script for spiders.py and we should be all ready to go if I come in here I'll make a email_scraper.py this will be our new python file and we're all ready to go also if you have python installed on

your computer but you don't see the python Run tab up here or the python files make sure to install the python extension over on the extensions tabs on the right here you can grab it from right here and that will allow you to run python files inside of VS Code so

now we just need a website to test this out for this I'm going to grab a website from a new example of my SAS app which you can get on the wait list down description below we have this Google Maps bus search here which allows us to

kind of search around on Google Maps here and we can pick out URLs we want to scrape so I'm going to come into Washington DC here for this example and we're going to save this current map URL link right here and then we're going to

want to enter in some keywords I'm just going to do lawn for this example but you can obviously put in whatever kind of business you're looking to search for you can also kind of add in a couple more URLs if you want we'll do three for

this example and then we could just search these three URLs and if we head on over to our tasks here we'll have a new Google Maps search right here for the time March 20th you can see I've already done a couple already and this

is still testing if you're interested in using this tool make sure to go sign up for the wait list down in the description below it's going to be released probably at the end of March here I'm really excited for you guys to use it if we give it a

refresh here you'll see we have 433 businesses we found and we can just come in here download the CSV so now if we open up this Google Maps data in Google Sheets here you can see we got everything like the business ID the city

the description the Friday hours full address latitude longitude the Monday hours the name the phone number the ID the Google Maps link the price rating the review count the Saturday Sunday hours what it's currently at right now time zone the types the website and the Wednesday hours as well so it's all

sorted by A to Z, and we also have the 434 including the header rows here for all of our leads. Awesome, we're going to use Michael's Lawn Service here because I notice that they have a URL on their website that we're going to test out to

make sure this works. Head on over to our scraper that we just built and then kind of put it in right here, and then I'm going to take the URL also and then put it in our allow domains. Now we're going to be crawling for this URL to find any

emails on this specific page here. What we can do from here is I'll make a new terminal and then I'll come over to chatGPT and we need to run a specific script called Scrapy crawl email spider, and that will run our web scraper based off

our name right here, so it's not by the file name, it's by the name right here. First, we got to make sure we go into our email scraper folder and then we can run our Scrapy crawl email spider command, and that will go through and

hopefully if all goes well. I don't know if this printed the email file, I don't see any email data. So let's try to edit this a little bit more. I'm going to say, okay, so it's going to tell us to modify the script slightly to get our emails.

And let's try this new one out instead: Scrapy crawl email spider, and there we go. We have our emails: Jason right here, and it looks like it gave us two different emails we can pick from on the page. So we want to do a couple of things to this

first we want to make sure we're not giving duplicate emails so I'm going to edit this more I'm going to say can you make this so it won't give me duplicate emails just one if it finds multiple and this will make sure we don't get more

than one email right here cuz these are both the same email so looks like it's going to add this emails found set here to make sure it's only unique variables and we can copy this over and and let's give this a test email spider and now we

only have one email awesome that's fantastic so now what we want to do is this let's say for instance we go on this guy's website and his email's actually not on this website a lot of the times they're on like the contact page or something like that for instance

I found this law site here that doesn't have an email on the first page but if we go over to contact you'll notice there's actually an email on the contact page but if we were to put in only this base URL into here let's see what it

will find so if we just replace this and we put this in here and then if we give this a run we don't get any emails cuz there's no emails on the page let's take this and let's put this back into ChatGPT and then we'll say can you make this

crawl every possible page to see if there's emails on the website so you can kind of see here that Scrapy has this Harse method here which allows us to find all the links on the page and then it will go through all those links and

then check to see if there's emails as well let's take our new code slap this back in here and let's see if it fixes it Scrapy crawl email spider you can see it's going through all of the different URLs on the page one thing I do want to

mention is you got to be careful that you don't go through too many URLs cuz if you do try to do too many requests on your own local machine you could get IP blocked so just make sure you're using proxy servers or something like that to

make sure you're not wrecking your own internet you can see it picked up the email right there look at that let's try with a couple more different URLs I don't see any emails on this page but maybe we can find some we come into here

and replace this URL give it a run look at that we got longare Gallow gmail.com right there awesome now so I just ran this scraper from here and it printed out the emails right there so maybe I'll probably just do that from now on here

let's try a couple more Cowboy Cuts if we want to get their email as well give it a run and we got three different emails here Cowboy Cuts Rose and James look at that as I mentioned before I'll have a link down the description below

to where you can get all this code for free so make sure you give this video a fat like and leave a comment let me know what kinds of things you want me to automate next in the future I'm going to be expanding on this script to scrape

more websites and get more data to help you supercharge your business with AI and web scraping tools. But in the meantime, if you want to learn how to scrape LinkedIn profile URLs, you can check out this video here where I show you how to build a really simple scraper

to scrape unlimited LinkedIn profile URLs for free. So I'll see you guys over there.

Scrape All Emails From ANY Website For Free

supercharging everything with ai and automation tools!
professional copy & paster



Mike Powers

Introduction and Project Overview: Building an Email Scraper

The video demonstrates building a simple scraper to extract emails from websites.

The scraper is useful for expanding email automation efforts and reaching local businesses.

The scraper takes a website URL as input and outputs a list of emails.

A demonstration using Google Maps data and a website URL is shown.

The scraper uses Python libraries, primarily Scrapy.

Setting up the Scrapy Environment and Project

Uses ChatGPT to generate a basic Scrapy script.

Scrapy is compared to other frameworks like Beautiful Soup and Selenium.

Instructions on installing Python and Scrapy using pip.

Creating a new Scrapy project using `scrapy startproject email_scraper`.

Navigating to the spiders directory to create the scraper script.

Using a Google Maps business search to obtain URLs for testing.

The Google Maps data includes various business details.

Testing the scraper with a specific URL and allowed domains.

Running the scraper using `scrapy crawl email_spider`.

Improving the Scraper: Handling Duplicates and Crawling Multiple Pages

Modifying the script to avoid duplicate emails.

Addressing the issue of emails not appearing on the main page.

Using ChatGPT to modify the script to crawl all pages.

Using Scrapy's `parse` method to find and check all links for emails.

Caution about excessive requests and potential IP blocking.

Provides a link to the code and encourages likes and comments.

Mentions future expansion of the script and other scraping projects.

Promotes a related video on scraping LinkedIn profile URLs.