hi, welcome to applications of deep neural networks for PyTorch with Washington University. In this video, we're going to look at Python files, in particular, files that we'll be using for deep learning and machine learning, which includes image files, CSVs, and others. Here, you can see the material for this part of the module. I have a link to this notebook in the description of this video. Let's go ahead and open it in Colab so that we're able to actually run the code if we need to. And here we are inside of Colab. I'll run that introductory part that just detects if Colab is present. And we also are going to map my G Drive. You can see my G Drive here is going to be mapped under content drive. Since we're dealing with files, this will allow us to copy files to and from the Google Drive. Now here, you can see that it has to ask for additional permissions from another part of Google. I pick my account name there. Those are the permissions I'm granting, and we

click to allow. We'll just wait while that connects up, and it's done and ready to go. I also need to remove that old TensorFlow tag on here since we're not using TensorFlow. So, these are the types of files that we will deal with in this course: there's CSV, which contains tabular data; there's image files, which contain images. We do a lot of image processing in this course; text files, which are just raw unstructured text, often used for natural language processing; JSON, which contains semi-structured data. You will see JSON often when you're scraping data from APIs and getting it from other sources. You might see H5; that's kind of a binary form of JSON; audio files as well. We don't really work with those in this course, but audio files can be thought of as a sort of time series because you have an audio wave that is going up and down, just like time series. These files that you get, they will come from either your hard drive or your Google Colab G Drive, the internet, or your G Drive, the cloud.

so let's first look at reading a CSV file. You'll see a lot of these HTTPS CSV files throughout the course. I have a location called data.heatonresearch.com where I put all the CSV files for this course, so you can download them right from here. If you were to copy and paste that URL and put it into a browser, it would just download the CSV file. Sometimes, you may want to do that. You can then take a look at it in Excel or some other CSV viewer. But if we just run this, it is going to actually reach out across the internet, and it's going to get us this data frame. Now, this is using pandas. I recommend using pandas for CSV files, especially if they fit completely into memory. Now, sometimes you'll get crazy big CSV files, and then you can use the built-in CSV package in Python that allows you to stream one so that you just have one record coming in at a time. But now we've got the data frame loaded, and if you display the first part of it, you can see

that this is the classic iris data set. It contains four measurements of three different types of iris flower, and you can see the four columns here, plus this is the target. This is what you would be trying to classify. Most likely, you could, in theory, try to classify sepal length based on the other four if you so desired. Not usually how that data set is actually used. Now, if you have a large CSV file that is just not going to fit in memory completely at once, there's a variety of ways that you can handle this. Everything from Spark to using large data warehouses like Snowflake. Many, many different ways to go about it. But here, what we're going to do is just re-stream it, read it in, and read one by one by one element. Here, I have the code that is used to actually open that connection. We're not loading it into pandas this time, though. We read this, we specify the CSV, we know that it's UTF-8 because that's how I saved it. UTF-8 is the most common format, but never completely assume that.

especially if you're dealing with international files. And what we're going to do then is this command next that moves us past the header role. It returns an array of the of whatever row read. You might want to use that, or you might not. And then we're going to sum up each of those four values. So I'm creating a sum that is an array of zeros. So this four means create an array that has four values, all of which are zero. And then we're going to count the number of rows. So we're going to loop over for every line in the CSV file. We're going to convert each row to a NumPy array. You do it that way so you've got the first line that's a string of an array of strings. And then line two, we're going to convert each of those elements in lines. So we converted the line, which is an array of strings, into an array of numbers. This says just take the first four, so location zero, one, two, three, because those are the first four Irish measurements. You don't want the species.

because we can't sum it, it's not a number. And then we're going to convert that type to floating point because it would have come back as a string normally, but these are floating point numbers. And then we are going to skip empty lines if they happen to exist. I think there's maybe one at the end of that file. And we simply sum this up. So the sum, this is a linear algebra sum, so it's taking the four values that we created of the zeros here and we're simply adding to them the four numbers from each row in that CSV file. And then we also have a counter that counts up how many lines we have. Then we do a final calculate the mean. So we take the sum and again this is linear algebra. So this is for values in that vector divided by count. So it's going to divide each of the four values in the iris data set by the count and get the average. So you can see this first measurement, the supple length, I think is much bigger than the others and

then with the two in the middle kind of three-ish and then low uh around one for the other we can read a text file. We're going to read this text file right across. It's a sonnet by a famous sonnet by William Shakespeare. Ah, I found a small problem, so let me uh, you'll, I'll add this to the uh file before you download it. I'm not sure how I managed that. And now you can see that it has loaded it again. I'm using codecs to load it that way I can specify the encoding like utf-8. And how do I compare the to a summer's day by William Shakespeare? It's just raw text. We read it in line by line, and I am using a right strip so it strips off any trailing space. Not necessarily needed for visualization, but I like to do that just because there does tend to be trailing white space often on lines and text files. Reading an image, you can read the image in by using the pill package in Python.

lets you work with images. I'm using requests to pull it in across from an HTTP address, so I'm loading it from this file, and we pull it in, and I convert it using PIL to an image. And now you've got a picture of the iconic Brookings Hall from Washington University. Okay, that's a quick introduction to how to read images, the types of images that you are dealing with in this course. We'll see other ways that you can take images directly into structures that PyTorch is able to make use of, but this is how you kind of go directly at them. Thank you for watching this video, and please subscribe to my YouTube channel if this was helpful or you'd like to follow along with the course. And please smash the like button if this video was helpful to you.

Python File Handling for PyTorch (1.4)

Python File Handling for PyTorch (1.4)