아름다운 수프와 요청으로 웹 스크래핑하기

### 파이썬의 아름다운 수프와 요청을 사용한 웹 스크래핑 소개
* 🐍 아름다운 수프와 요청은 **웹 스크래핑을 위한 필수 파이썬 패키지**로, 초보자도 웹 사이트에서 원하는 정보를 추출할 수 있습니다.
* 📦 더 고급 패키지도 있지만, 이 초급 시리즈는 **아름다운 수프와 요청에 초점을 맞추고** 있으며, 향후 시리즈에서는 **고급 기능**을 살펴볼 예정입니다.
* 📥 첫 번째 단계는 **`bs4에서 BeautifulSoup 가져오기`와 `요청 가져오기`**를 사용하여 이러한 패키지를 가져오는 것입니다.
* 🌐 다음 작업은 **HTML 데이터를 추출할 URL을 지정**하고 이를 변수에 할당하는 것입니다.
* 🔗 요청.get(URL)`을 사용하면 URL로 가져오기 요청을 보내고 **상태 코드가 200인 응답 객체**를 반환하며, 이 객체는 성공을 나타냅니다.
* 🥣 그런 다음 **Beautiful Soup을 사용하여 URL에서 얻은 원시 HTML 콘텐츠를 구문 분석**하여 특정 정보를 쉽게 탐색하고 추출할 수 있습니다.
* 🔍 다음 강의에서는 **`find`와 `find_all`**을 사용하여 태그, 가변 문자열, 클래스 등의 HTML 요소를 쿼리하는 방법을 다룹니다.
* 📊 마지막 강의에서는 **웹페이지에서 데이터를 추출하여 판다스 데이터프레임에 저장하는 미니 프로젝트**를 진행합니다.

이 강의에서는 패키지 가져오기와 HTML 검색에 중점을 두고 웹 스크래핑을 위한 Python의 Beautiful Soup과 요청을 소개합니다.

hello everybody. In this lesson, we're going to be taking a look at Beautiful Soup and Requests. Now, these packages in Python are really useful. These are the two main ones that I used when I was first starting out with web scraping. It can get a lot of what you want done in order to get that information out. Now, of course, there are other packages that you can use that may be a little bit more advanced. But again, this is just the beginner series. In a future series, we'll look at other packages as well that have some more advanced functionality. So what we're going to be doing is we're going to import these packages and then we're going to get all of the HTML from our website and make sure that it's in a usable state. And then in the next lesson, we're going to kind of query around in the HTML, kind of pick and choose exactly what we want. We'll look at things like tags, variable strings, classes, attributes, and more. So let's get started by importing our packages. What we're going

to say is from bs4. This is the module that we're taking it from. We're going to say import, and then we'll do beautiful soup. Then we're going to come down, and we're going to say import requests. Now let's go ahead and run this and hit shift enter, and it works well for me. Now if this does not work for you, you may potentially need to actually install bs4. So you may have to go to your terminal window and say pip install bs4. I'll just let you Google how to do that if you need to do that because it's pretty easy. But if you're using Jupyter notebooks through Anaconda like how we set it up at the beginning of this Python series, then you should be totally fine. It should be there for you. The next thing that we need to do is specify where we're taking this HTML from. So what we need to actually do is come right over here to our web page, and we need to get the URL. So we're going to go here.

going to copy this URL, and I'm just going to put it right here for a second. And what we're going to do is we're going to be using this URL quite a bit, so we just want to assign it to a variable. So to say, URL is equal to, and then we'll put it right in here. Now we can get rid of that. So now this is our URL going forward. This is where we'll be pulling data from. Let's go ahead and run this. Now we're going to use requests, and what we're going to do is we're going to say requests.get, and then we're going to put in URL. Now this get function is going to use the requests library. It's going to send a get request to that URL, and it's going to return a response object. Let's go ahead and run this. As you can see here, I got a response of 200. If you got something like a 204, or a 400, or 401, or 404, all of these things are potentially bad. Something like a 204

would mean there was no content in the actual web page. 400 means a bad request so it was invalid. The server couldn't process it and you don't get any response. If you got a 404, that might be one that you're familiar with. That's an error that means the server cannot be found. The next thing that we're going to do is take the HTML. Now if you remember, we come right back here and we inspect this. We have all of this HTML right here. Now on this web page specifically right now, it's completely static. It's not a bunch of moving stuff or anything like that. But usually when you're looking at HTML, if you're looking at something like Amazon, and those web pages can update, but when you actually pull that into Python, you're basically getting a snapshot of the HTML at that time. So what we're going to do is bring in all of this HTML, which is our snapshot of our website, and then we can take a look at it. So we're going to come right down here and now we're going to

say beautiful soup. So now we'll use the Beautiful Soup package or library. So when you say Beautiful Soup, then we're doing open parenthesis. We're gonna do two things. There's two parameters that we need to put in here. First, we need to put in this get request. So we actually need to name this and we'll call this page. We'll say page is equal to. And let's run this. And now we're going to put that page in here. And what we're going to say is .text. So the page is what's sending that request. And then the .text is what's retrieving the actual raw HTML that we're going to be using. Then we're going to put a comma here. And what we need to specify is how we're going to parse this information. Now this is an HTML. So what we're going to do is HTML just like this. This is standard that's already built into this library. So we don't need to go any further, but it's basically going to parse the information in an HTML format. Let's go.

ahead and run this. Let's see what we get. And as you can see, we have a lot of information. And as we scroll down, I'll try to point out some things that we've already looked at in previous lessons. Um, something like this th tag, that should be very similar, that's the title. Then we have these TD tags. And then of course, if we scroll down even further, we'll have things like ATR tag. So these are all things that we looked at in that first lesson when learning about HTML. Now again, we want to assign this to a variable. So we're going to say soup, that's going to say equal to this information right here. Now I'm not going to go into all the history behind Beautiful Soup. What I will say is the guy who created this Beautiful Soup Library, what he said was is that it takes this really messy HTML or XML, which you can also use it for, and it makes it into this kind of Beautiful Soup. So I just that was kind of funny, but that's why we're calling it soup.

right here, and we're gonna go ahead and run this, and we'll come right down here, and we'll say print soup. And let's run it, and now we have everything in here. So we have our HTML, our head, we have some href, and some links in here. And let's scroll down a little bit more, and then we have our body right there. And of course, we have a bunch of information in here. Now, in the next lesson, what we're going to be doing is learning how to kind of query all of this to take specific information out and basically understand a lot of what's going on in this HTML to make sure we can actually get what we need. Now, if this looks really kind of messy to you and it just doesn't make a lot of sense, there is one more thing that I'm going to show you, and we'll come right down here. So we'll say soup. Dot printify. And if you've ever used a different type of programming languages, protify is very common in a lot of them.

we'll just make it a little bit more easy to visualize and see. You'll notice that it kind of has this hierarchy built in, whereas if we scroll up, there's no hierarchy built in. It's all just down this left-hand side. So if you kind of want to view it and just kind of visually see the differences, this does help a lot, but it doesn't actually help a lot when you're, you know, querying it or using, you know, find and find all, which is what we're going to look at in the next lesson. So that is our lesson on Beautiful Soup and requests. In the next two lessons, we're going to be looking at find and find all, as well as really diving into things like variable strings and tags and classes and all those things. And then in the last lesson, we're going to do kind of this mini project where we try to get all the data from this webpage that we've been using from that table and put it into a Pandas data frame. So thank you guys so much for

watching, I really appreciate it. If you like this video, be sure to like and subscribe below, and I will see you in the next one. [Music] Thank you. [Music]

BeautifulSoup + Requests | Web Scraping in Python

Alex The Analyst

파이썬의 아름다운 수프와 요청 소개

아름다운 수프와 요청은 웹 스크래핑, 특히 초보자를 위한 필수 Python 패키지입니다.

이 패키지는 웹 사이트에서 HTML 데이터를 추출하여 추가 조작에 사용할 수 있도록 하는 데 사용됩니다.

이 과정에는 Beautiful Soup 및 요청 모듈을 가져오고, 특정 URL에서 HTML을 가져온 다음, 다양한 HTML 요소와 속성을 사용하여 원하는 정보를 탐색하고 추출하는 작업이 포함됩니다.

Python을 사용하여 URL로 가져오기 요청 보내기

이 코드는 변수에 URL을 할당하고 요청 라이브러리를 사용하여 URL로 get 요청을 전송하고 응답 객체를 수신합니다.

200, 204, 400 또는 404와 같은 다양한 응답 코드는 요청의 성공 또는 실패를 나타냅니다.

그런 다음 웹사이트의 HTML이 검색되어 당시 웹 페이지 콘텐츠의 스냅샷을 제공합니다.

웹 스크래핑에 아름다운 수프 사용

Beautiful Soup 패키지는 가져오기 요청을 하고 원시 HTML 데이터를 검색하는 데 사용됩니다.

그런 다음 HTML 정보를 파싱하는 데 사용되어 태그와 요소를 쉽게 탐색할 수 있습니다.

Beautiful Soup은 지저분한 HTML과 XML을 처리하기 위해 만들어져 스크래핑을 위해 데이터에 더 쉽게 액세스할 수 있습니다.

아름다운 수프로 HTML 쿼리 배우기

이 튜토리얼에서는 아름다운 수프를 사용하여 HTML 구조를 탐색하고 이해하는 방법을 다룹니다.

다음 강의에서는 찾기와 모두 찾기, 가변 문자열, 태그, 클래스로 작업하는 방법을 중점적으로 다룹니다.

마지막 강의에서는 웹페이지에서 데이터를 추출하고 판다스 데이터 프레임을 만드는 프로젝트를 진행합니다.

시청자가 좋아요를 누르고 구독하도록 유도하는 것은 유튜버의 성공을 위해 매우 중요합니다.

충성도 높은 구독자 기반을 구축하는 것은 채널의 성장과 성공에 큰 영향을 미칠 수 있습니다.

시청자의 성원에 감사를 표하는 것은 크리에이터와 시청자 간의 관계를 강화하는 데 도움이 됩니다.