파이썬 스크랩으로 간단한 이메일 스크레이퍼 구축하기

### 스크랩으로 간단한 이메일 스크레이퍼 만들기
* 📧 이 튜토리얼에서는 모든 웹사이트에서 이메일 주소를 추출하는 간단한 스크레이퍼를 만드는 방법을 보여드리며, 특히 **지역 비즈니스에 대한 이메일 홍보를 강화**하고자 하는 분들에게 유용합니다.
* 💻 이 프로세스는 웹사이트 URL을 가져와 홈페이지에 표시되지 않더라도 **해당 사이트에서 찾은 이메일 주소 목록**을 검색하는 간단한 스크랩 스크립트로 시작됩니다.
* 🔧 스크래퍼를 설정하려면 사용자는 Python과 스크래피를 설치한 다음 **스파이더 스크립트가 상주할 "이메일 스크래퍼"라는 새 스크래피 프로젝트**를 만들어야 합니다.
* 🌐 이 동영상에서는 **Google 지도 데이터를 사용하여 이메일 주소를 스크랩할 비즈니스의 URL을 식별**하고 입력하는 방법을 구체적인 예시와 함께 보여줍니다.
* 📝 스크레이퍼는 **이메일 중복을 방지**하도록 설계되었으며 웹사이트의 모든 페이지를 크롤링하도록 수정하여 **연락처 페이지나 기타 링크된 페이지에 있을 수 있는 이메일**을 캡처할 수 있습니다.
* ⚠ IP 차단을 피하기 위해 여러 URL을 스크랩할 때는 주의가 필요하며, **광범위한 스크랩 작업에는 프록시 서버를 사용하는 것**이 좋습니다.
* 📊 이 튜토리얼에는 향후 스크립트의 기능을 확장하겠다는 약속과 함께 **LinkedIn 프로필 URL을 스크랩하기 위한 추가 리소스**에 대한 언급도 포함되어 있습니다.
* 🔗 시청자는 **설명에 제공된 코드에 액세스**하여 향후 자동화 요청에 좋아요를 누르고 댓글을 달아 콘텐츠에 참여하는 것이 좋습니다.

스크레이퍼는 모든 웹사이트에서 이메일을 손쉽게 추출하여 지역 비즈니스의 이메일 자동화를 향상시킵니다.

I'm going to show you how we can build a really simple scraper to extract any email from any website. I get a lot of people reaching out to me with URLs they get from either Google Maps or LinkedIn, and they want to be able to figure out

how they can get the email off the website that they just scraped. So this video will show you just that. This is great if you want to expand your email automation efforts and reach out to more local businesses. I have links to where you can get all the code down below, and

also, if you want to work with me in any kind of AI automations project, make sure to book a free time down in the description below. But anyway, let me show you how this works. So how this scraper works is I basically have this simple scrapey script here, which allows me to

put in a website URL, and then it will just pop out a list of emails. So I can go over to my Google CSV data with a bunch of Google Maps data here, so we can come on over to a website here. We'll grab the domain, I'll paste this into our

start URL here, and then I'll copy over the allowed domain right there. And if we give this a run, you can see it'll go through through all of the connected domains to see if there is any emails, and then if it finds anything, it puts it

in our little emails JSON here, so we can see we got this Susie email from her business right there. We can scrap basically any email from any website, if it exists on their website. You'll notice here it doesn't even show the email on the first page, it's getting it from

somewhere else on the website so let's get into building this thing I'm over on VS Code and we're going to be using a series of Python libraries in order to get this to work and the big one is going to be Scrapy, it's quickly

becoming one of my favorite Python libraries to scrape stuff off the internet and this is what we're going to be using to build our website scraper to get started let's head on over to ChatGPT and see if we can build out this scraper so

I'm going to say make me a simple Scrapy script to scrape any email off a URL and then put it in a JSON file I'm on ChatGPT 4 for this and we're just going to send it so Scrapy is very similar to

other frameworks like Beautiful Soup or Selenium or Puppeteer, I talk about a lot of the other stuff on my channel but for my experience Scrapy has been one of the fastest scrapers to use especially when trying to crawl stuff on websites so we're going to use this

really simple scraper here um but first let's kind of follow the instructions in order to get this set up on our machine so if we don't already have Python installed first we got to make sure we have at least the latest version of Python over on the Python website here and

then we're going to want to make sure we install Scrapy with Pip install Scrapy so I'll just paste this in my VS Code terminal in here and it will download Scrapy for us and in the meantime we can come back over here and we need to make

a new Scrapy project so we could do Scrapy start project email scraper awesome so now I'll just come into here and we'll just paste in this here and there we go we have our new email scraper folder so to get into this we're

going to CD email scraper and enter in there so now we're inside of our folder and then we have to navigate into our email scraper and into our spiders directory so if we go into our email scraper folder here our spiders directory is where we're going to put

our web scrapers that we're going to run we have to touch any of these other python files unless it specifies us to so we're going to want to put our new scraper inside of our spiders folder so I'm going to come back over here and I'm

going to copy this script for spiders.py and we should be all ready to go if I come in here I'll make a email_scraper.py this will be our new python file and we're all ready to go also if you have python installed on

your computer but you don't see the python Run tab up here or the python files make sure to install the python extension over on the extensions tabs on the right here you can grab it from right here and that will allow you to run python files inside of VS Code so

now we just need a website to test this out for this I'm going to grab a website from a new example of my SAS app which you can get on the wait list down description below we have this Google Maps bus search here which allows us to

kind of search around on Google Maps here and we can pick out URLs we want to scrape so I'm going to come into Washington DC here for this example and we're going to save this current map URL link right here and then we're going to

want to enter in some keywords I'm just going to do lawn for this example but you can obviously put in whatever kind of business you're looking to search for you can also kind of add in a couple more URLs if you want we'll do three for

this example and then we could just search these three URLs and if we head on over to our tasks here we'll have a new Google Maps search right here for the time March 20th you can see I've already done a couple already and this

is still testing if you're interested in using this tool make sure to go sign up for the wait list down in the description below it's going to be released probably at the end of March here I'm really excited for you guys to use it if we give it a

refresh here you'll see we have 433 businesses we found and we can just come in here download the CSV so now if we open up this Google Maps data in Google Sheets here you can see we got everything like the business ID the city

the description the Friday hours full address latitude longitude the Monday hours the name the phone number the ID the Google Maps link the price rating the review count the Saturday Sunday hours what it's currently at right now time zone the types the website and the Wednesday hours as well so it's all

sorted by A to Z, and we also have the 434 including the header rows here for all of our leads. Awesome, we're going to use Michael's Lawn Service here because I notice that they have a URL on their website that we're going to test out to

make sure this works. Head on over to our scraper that we just built and then kind of put it in right here, and then I'm going to take the URL also and then put it in our allow domains. Now we're going to be crawling for this URL to find any

emails on this specific page here. What we can do from here is I'll make a new terminal and then I'll come over to chatGPT and we need to run a specific script called Scrapy crawl email spider, and that will run our web scraper based off

our name right here, so it's not by the file name, it's by the name right here. First, we got to make sure we go into our email scraper folder and then we can run our Scrapy crawl email spider command, and that will go through and

hopefully if all goes well. I don't know if this printed the email file, I don't see any email data. So let's try to edit this a little bit more. I'm going to say, okay, so it's going to tell us to modify the script slightly to get our emails.

And let's try this new one out instead: Scrapy crawl email spider, and there we go. We have our emails: Jason right here, and it looks like it gave us two different emails we can pick from on the page. So we want to do a couple of things to this

first we want to make sure we're not giving duplicate emails so I'm going to edit this more I'm going to say can you make this so it won't give me duplicate emails just one if it finds multiple and this will make sure we don't get more

than one email right here cuz these are both the same email so looks like it's going to add this emails found set here to make sure it's only unique variables and we can copy this over and and let's give this a test email spider and now we

only have one email awesome that's fantastic so now what we want to do is this let's say for instance we go on this guy's website and his email's actually not on this website a lot of the times they're on like the contact page or something like that for instance

I found this law site here that doesn't have an email on the first page but if we go over to contact you'll notice there's actually an email on the contact page but if we were to put in only this base URL into here let's see what it

will find so if we just replace this and we put this in here and then if we give this a run we don't get any emails cuz there's no emails on the page let's take this and let's put this back into ChatGPT and then we'll say can you make this

crawl every possible page to see if there's emails on the website so you can kind of see here that Scrapy has this Harse method here which allows us to find all the links on the page and then it will go through all those links and

then check to see if there's emails as well let's take our new code slap this back in here and let's see if it fixes it Scrapy crawl email spider you can see it's going through all of the different URLs on the page one thing I do want to

mention is you got to be careful that you don't go through too many URLs cuz if you do try to do too many requests on your own local machine you could get IP blocked so just make sure you're using proxy servers or something like that to

make sure you're not wrecking your own internet you can see it picked up the email right there look at that let's try with a couple more different URLs I don't see any emails on this page but maybe we can find some we come into here

and replace this URL give it a run look at that we got longare Gallow gmail.com right there awesome now so I just ran this scraper from here and it printed out the emails right there so maybe I'll probably just do that from now on here

let's try a couple more Cowboy Cuts if we want to get their email as well give it a run and we got three different emails here Cowboy Cuts Rose and James look at that as I mentioned before I'll have a link down the description below

to where you can get all this code for free so make sure you give this video a fat like and leave a comment let me know what kinds of things you want me to automate next in the future I'm going to be expanding on this script to scrape

more websites and get more data to help you supercharge your business with AI and web scraping tools. But in the meantime, if you want to learn how to scrape LinkedIn profile URLs, you can check out this video here where I show you how to build a really simple scraper

to scrape unlimited LinkedIn profile URLs for free. So I'll see you guys over there.

Scrape All Emails From ANY Website For Free

supercharging everything with ai and automation tools!
professional copy & paster



Mike Powers

소개 및 프로젝트 개요: 이메일 스크레이퍼 구축하기

이 동영상은 웹사이트에서 이메일을 추출하는 간단한 스크레이퍼를 만드는 방법을 보여줍니다.

스크레이퍼는 이메일 자동화 작업을 확장하고 지역 비즈니스에 도달하는 데 유용합니다.

스크레이퍼는 웹사이트 URL을 입력으로 받아 이메일 목록을 출력합니다.

Google 지도 데이터와 웹사이트 URL을 사용한 데모가 표시됩니다.

스크레이퍼는 주로 스크랩과 같은 Python 라이브러리를 사용합니다.

ChatGPT를 사용하여 기본 스크랩 스크립트를 생성합니다.

스크랩은 아름다운 수프 및 셀레늄과 같은 다른 프레임워크와 비교됩니다.

pip를 사용하여 Python 및 스크랩을 설치하는 방법에 대한 지침입니다.

스크랩 시작 프로젝트 이메일_스크래퍼`를 사용하여 새 스크랩 프로젝트 만들기.

스파이더 디렉터리로 이동하여 스크레이퍼 스크립트를 만듭니다.

실제 데이터로 스크레이퍼 테스트

Google 지도 비즈니스 검색을 사용하여 테스트용 URL을 얻습니다.

Google 지도 데이터에는 다양한 비즈니스 세부 정보가 포함되어 있습니다.

특정 URL과 허용된 도메인으로 스크레이퍼를 테스트합니다.

스크래퍼를 `스크래피 크롤링 이메일_스파이더`를 사용하여 실행합니다.

스크레이퍼 개선하기: 중복 처리 및 여러 페이지 크롤링하기

이메일 중복을 방지하기 위해 스크립트를 수정합니다.

이메일이 메인 페이지에 표시되지 않는 문제 해결.

ChatGPT를 사용하여 모든 페이지를 크롤링하도록 스크립트를 수정합니다.

스크랩의 `파싱` 메서드를 사용하여 이메일의 모든 링크를 찾아서 확인합니다.

과도한 요청과 잠재적인 IP 차단에 주의하세요.

코드에 대한 링크를 제공하고 좋아요와 댓글을 장려합니다.

스크립트 및 기타 스크래핑 프로젝트의 향후 확장에 대해 언급합니다.

LinkedIn 프로필 URL을 스크랩할 때 관련 동영상을 홍보합니다.