Understanding Artificial Neural Networks and Batch Processing

The practice session focused on training an artificial neural network to approximate the cosine function using regression and batch processing.

### Implementing Batch Processing in PyTorch for Training an Artificial Neural Network to Approximate the Cosine Function
* 🧠 We will implement an artificial neural network to **approximate the cosine function** using the PyTorch library and train it through **batch processing**.
* 📦 Batches are used to **speed up the learning process** by processing **multiple data points simultaneously**. In this practice, we will set the **batch size to 25**.
* 🔄 By using the DataLoader function in PyTorch, we can easily **divide and shuffle batches**. Preparing the training tensors and then passing them to the DataLoader with the specified **batch size** and **shuffle option** will automatically apply the batching function.
* 📉 The loss steadily decreases during training and **converges to 0**, indicating that the model is learning to **approximate the cosine function effectively**.
* 📈 When visualizing the results obtained from each neuron in the network, individual graphs are drawn that do not resemble the cosine function. However, when summing up the outputs of all neurons, the final output **closely matches the cosine function**.
* 📚 Understanding the principle of **batch processing** and how to implement it in PyTorch is essential for **training neural networks efficiently**.
* 🎓 "Since we are studying, we have delved a bit more specifically into the **batch principle**. Probably the training is **all done**."
* 🔗 The ability to leverage batch processing effectively will become increasingly important as the **size of models and datasets** in deep learning projects continues to grow.

Hello, in this practice session, we will learn about artificial neural networks. The first artificial neural network practice will demonstrate the process of approximating a common function using the optimization method we learned in the previous session. What I mean is, first, if you look here, you can see a formula we learned in our theory class. This formula we learned in theory represents the algorithm of artificial neural networks. So, we will use this principle to approximate the cosine function here. Currently, this function is not a cosine function. We will structure the algorithm like this and train it to approximate this cosine function. Therefore, we will practice learning this function to make it look exactly like a cosine function. So, today, we will practice training to make this function resemble a cosine function.

This practice will be about regression. What is regression? It's when the output of a certain model becomes a specific real number. It should be the same as the cosine value of that real number. In our next class, we will learn about classification, which is different. In classification, it should output an integer, whether it's 0, 1, or 2, representing certain values. In this practice of ours, since it's regression, the real number coming out of this function itself will be the output. So, let's start right away. First, I will import the necessary libraries like this. If you look below, I have written quite a bit. It might be good to read it once as a review of the things we learned in the theory class. There are many mathematical notations, so it might be a bit difficult, but once you read it, you'll understand.

It shouldn't be that difficult when you look at it. Of course, for those who are not familiar with math, it might be helpful to just skim through it quickly and then move on to coding practice. Let's define the model right away. In the previous practice, do you remember when we defined the linear regression model as a class? We used class functions in our linear regression model to implement it. It's similar here. But the key point to be careful about here is this part. You need to understand this well to be able to understand how we implement the artificial neural network model, not the linear regression model. You need to understand this part well. Focus a bit on the values inside here and try to understand. Let's see what they mean and explain while we go. We will use a variable called ... and create it using PyTorch.

I'm going to create a parameter using PyTorch Live. I will fill in the values of the parameters randomly using PyTorch. I will create this parameter matrix shape as 1 x N. This means I will create the parameter matrix shape as 1 x N. The parameter matrix shape will be in the form of 1 x N. Similarly, I will create another variable, let's call it B, using PyTorch Live to fill in the parameters randomly. I will fill in the parameters randomly and shape the matrix as 1 x N. This means I will shape the matrix as 1 x N. Also, for V, I will again use PyTorch Live to fill in the parameters randomly. I will fill them randomly, and the shape of this parameter matrix will be in the form of N x 1. However, here, you see, N is mentioned below here as news.

I'm going to initialize with a neuron and news, I will initialize with news. I'm going to do a parameter initializer with news. Here, what's going in now, if you look, what 'n' means is the number of nodes, in other words, the number of neurons. Since it's an artificial neural network, nodes are also called neurons. Anyway, because it mimics the brain, sometimes nodes are referred to as neurons. Anyway, it means the number of nodes, which is the number of nodes within an artificial neural network. We learned this in our theory class. But then, why did we define each parameter, wbv parameters, in this way? Why did we define it as 1n, 1n, n-1? It's written above here. I've explained it in detail above. The process I'm trying to implement involves these calculation processes.

I will explain it through the slides. First, I will explain why the parameters in this model, corresponding to wbv, are defined in matrix shape. When I explain X times V, the rest should be easy for you to understand. Here, as you can see, we will input the training data in the form of a one-dimensional vector. What does this mean? It means that when one input is entered, one output is produced, and this process will be repeated a thousand times. In other words, we have a thousand pieces of data, and we will input each of the thousand data points into the model one by one. So, we will input a total of 1000 times. Now, we have hyperparameters that we need to set, such as the number of nodes or neurons. Let's say the very first input comes in. In the very first input from this training data, there is -2 pi. As you can see from the training data above, the very first input is -2 pi.

- 2nd layer - The 2nd layer receives input - When the 2nd layer receives input, there are a total of n nodes, right? So, starting from the first node, there will be nodes up to the nth node. However, in the 2nd layer, inputs are given to each node separately. Each node receives its own weight. So, the first node has w1, the second node has w2, and the last node has wn as their respective weights. Also, we need to add bias, right? We need to add bias, so each has its own bias. When the inputs are all in the nodes, the output here will be wn * w1 + B1, here will be wn * w2 + B2, and here will be wn * wn + BN. That's the process now.

And this process includes everything up to what is behind it. By using matrix operations, vector operations, or matrix operations, we can mathematically represent the drawing I have shown you. However, in nodes, it doesn't end with just this process. After that, what we will do is, for each node, we will pass the values that come out through some activation function, then we will multiply it by another weight parameter in front of it. These are the operations within each node. How many times do we repeat this? We repeat it a total of a thousand times. Why? Because we have multiple input data. When one input data is entered, as I explained, each node receives input and performs these operations one after another. After that, when we add them together, the desired output will be obtained. We repeat this process a thousand times.

Repeating a thousand times can be burdensome. Just performing calculations for one piece of data already involves a considerable amount of computation. However, we need to repeat this a thousand times. The data we use in deep learning is not at the level of individual items that you will use in your practice later; it's not just a hundred or a thousand items, it easily goes beyond a million. So, does it make sense to repeat this process a thousand times? It seems unreasonable, too challenging, right? That's why we need to use batches. By using batches, we can still reduce the number of computations to some extent. We can reduce the number of these calculations to some extent. Now, let me briefly explain how using batches helps reduce the amount of computation. In our practice, we will set the batch size to 25. In other words, we will input 25 data sets at once. Let me explain this slowly.

Earlier, we performed an operation on a matrix with input data, right? But this time, it means we will perform operations on the matrix for the number of batch sizes. So, this M probably represents 25, right? In our practice, we will set the batch size to 255, which means we will input 255 pieces of data at once. It's a bit different from inputting just one like we did before. What does this mean? Earlier, we input data that we knew as -2 pi into each node and passed through activation functions and operations to get an output by summing them all up. If we operated on the -2 pi data to get an output, this time, instead of -2 pi to -2 pi, we will input 25 at once. Therefore, naturally, we will get not just one output from the beginning but 25 outputs from the first to the 25th.

It will be possible. What's good about this is that earlier, we had to input -2 parts and repeat it 1,000 times to get 1,000 outputs. This time, by inputting 255 parts and putting in 25, if we repeat it 40 times with 1 divided by 255, it will work. Since the need for batching decreases from 1 repetition to 40 repetitions, there is no reason not to use batching. Then, how does the inference process continue? Through matrix operations like this above, using this matrix concept, we can implement it. If earlier we put in one input data and obtained one column, this time by putting in m input data and performing matrix operations, we can obtain a matrix of this size, m by n. When we put in one input data earlier, we obtained a 1 by n vector or matrix, but by putting in m data, we can get an m by n matrix. And this is much more.

It's efficient, so there is no reason for us not to use batches. If you understand that, then how are intercepts calculated? Well, when one input data is entered, the intercept is calculated. So, when multiple input data are entered, the intercepts for each row should be correctly summed. However, this can be easily done using the broadcasting function. Later, when you see the code, you will understand, but basically, we will have an input of size N by 1. We will receive an N by 1 matrix input, and we will multiply this by a 1 by N matrix. Then, we just add the 1 by N matrix.

If it's about casting, even if you define it as 1 by n, you can understand that it automatically applies the concept of broadcasting to matrix operations. Even if the data becomes two-dimensional, the principle remains the same. The principle and the process of operation are the same, so it's not difficult. When I explained during the theory class, I mentioned that the shape of the input data and the shape of the weight parameter w within each node should be the same. What does this mean? It means that if the input data x comes in as a two-dimensional vector, then the W parameters within each node should also be two-dimensional. Since we defined the number of nodes as n, there are a total of n nodes. So, what happens is this: we have input data X1-1, X1-2, like this, two elements like this.

A 2-dimensional vector is inputted, then there are a total of n nodes. When there are n nodes, from the first node to the nth node, they are continuously inputted. Each node has parameters like w1-1, w1-2, w2-1, w2-2, wn-1, wn-2. Shall we erase a few? Without any objections, within each node, these parameters like W exist in this way. Then this 2-dimensional vector is inputted into each node for computation. This is how the computation proceeds. Next, it goes through the activation function, then similarly, the weights are multiplied in the front, and added. The principle is the same, just aligning this and that. It's not difficult at all. Even if a 2-dimensional vector returns, and even when the concept of batch is applied, it's the same. The concept of batch is not applied.

Yes, this is when the concept of batching is not applied, and only one piece of data comes in. When only one piece of data comes in, if this kind of operation is performed, if the batching concept is applied, it is like this, grouping these into matrices. Originally, if input in the shape of 1 by 2 came in, this time if input in the shape of 1 by 2 comes in, then if an m by 2 input comes in, a matrix of shape N by 2 comes in, then it is multiplied. The weight matrix W, the weight parameter matrix, remains the same. And the shape of the output becomes n by n. Earlier, it was 1 by n, right? 1 by n. Just stacking the data together into one matrix and putting it in, so the data is just combined and output. Applying the concept of batching makes it the same. Even if the concept of batching is applied to input for a two-dimensional vector, the principle remains the same. Just understand that. This is how the operation has been going on until now.

You have seen the process, and then we pass through the activation function like this, and after going through the matrix operation process we just explained earlier, we pass through the activation function, and then this matrix needs to undergo matrix operations. In this practice, after passing through the activation function, we additionally multiply by some parameters, so we need to multiply by a matrix of the shape n by 1 at the end to go through the process of addition. By doing this, we can perform the process of artificial neural network computation all at once. Therefore, we define the matrix shape as n by 1 to perform the operation. Once we finish this operation, we obtain an m-dimensional vector.

Put the m-dimensional data into the input. Now, we group the m-dimensional individual data into batches for input. Since we have grouped the individual data into vectors for input in batch units, the output should naturally be individual data as well. If you confirm this far, it should be fine. You can represent the calculation process above in one line. By simply performing operations in one line, you will be able to recall the detailed calculation process I have explained so far. It's simple. This is the power of PyTorch and matrix operations. It's great to be able to perform what seemed complicated in one line. Also, if you look here, there is an 'n'. 'n' now represents the number of nodes. Let's understand this meaning and move on. So, this 'n' is a hyperparameter. I have explained this before, but a hyperparameter allows us to determine the number, such as 100 or 1,000.

If you look, x x Self Point w is given, and the matrix is chosen. Then, Self Point B is added. Just adding Self like this can be operated correctly through broadcasting. Then, it was mentioned that it can be activated by passing through an activation function. In this practice, we will use a sigmoid. Because there may be a problem with the sigmoid gradient vanishing, we often use ReLU instead of the sigmoid in hidden layers. However, since this practice is very simple and we only have one layer, we can just use the sigmoid. The performance is good even when using the sigmoid, so we will use the sigmoid. So, after passing through the activation function sigmoid, the value is finally.

If we do self and the last self, now if we just do the main matrix, then we get the output we want. Here, if I want to create a deeper neural network by increasing the layers, then I just need to put this returned value back in and add one more line. Like this, uh, 2 and this output, what, here, just uh, no, not here, I should put it here, just keep returning it like this, then if we do this, uh, we have added one more layer. But of course, uh, what, this self dot V, we need to define one more thing with a different one, right? Uh, no, no, right now, uh, with this self dot W and self dot B, we need to define one more thing, because uh, for another layer, we need to add more here, anyway, in this way, we can just stack layers.

Please understand up to the fact that stacking layers is not a problem. I wanted to mention that it can be implemented very simply. However, just because a neural network is deep doesn't mean it's all good. It's definitely not good just because it's deep; gradient vanishing can occur, so making it deep is not always good. Instead, it can lead to bad results where computations increase but performance doesn't improve. Our practice is simple, so just stacking hidden layers like this can already yield good performance. Also, below here, we have functions that make it easy to call parameters in classes. Our loss function here has a slightly different function name, but it should still be the same. And our...

I will use the gradient descent function that I implemented and the gradient step function from the previous practice as they are. Our input data will be divided into values between negative 2 pi and 2 pi. We will use the divided values to create a vector using reshape, making it a vector of 1000 by 1 shape. We will input this vector as our data. Also, we will divide it into batches for further consideration. For comparison, we need the actual values, the true values. We need a function to extract the correct answers for comparison and training. Therefore, for comparison, we will also draw the actual cosine function. Our goal is to train the model to draw a graph similar to the cosine function we aimed for earlier.

The function is a function that actually draws the cosine function and fixes it. It declares an instance for us. I will try it once with one epoch and execute it. Let's draw the graph once. The blue function here is the blue graph of the real cosine function, and the orange graph is the model we defined above. If you look, they are not matching closely now. Why are they not matching now? Because we haven't trained the model yet. So, the parameters in the model are just randomly initialized. Since it's randomly initialized, of course, a graph like this, without any meaning, will come out. Right. Then let's try training once. But the training loop we are looking at now is slightly different from the one we saw in the previous practice because now there are two doors.

We have this set up because we are going to adapt the batches. What was a batch? It splits the input data into batch sizes to train all at once. What is the effect of batches? It speeds up the learning process. Of course, this practice is very simple because we only have 1,000 data points, so the effect of these batches is not significant. Even if we train one by one a thousand times, the performance will probably be good. However, as we are studying, in future projects, our model size will increase. When the model size or the amount of data increases significantly, batches are essential. If we don't use batches, the performance will not be good. Now, we define the batch size as 25, set the learning rate, and then.

We need to look at this change in Los Angeles because of the cold weather, so I will declare a list to gather Los Angeles. If you look, there are two loops. I will explain why this loop is implemented twice in the slides later. I will explain this part of the code. First, let me explain this part step by step. We will run the epoch a thousand times. First, we will create an index list, and this index list will be listed as long as the length of the training data. Earlier, we created 1000 training data. We will create indexes like this length. Then, 1,000 indexes will be listed in a NumPy array list like this, and we will shuffle it randomly. Next, let's go into the second loop.

We will repeat as many times as the batch size. Earlier, we defined the batch size as 25. So, how many batches are there? Earlier, the train was 1, and the batch size is 40, meaning we will loop 40 times according to the batch size. This is the first loop. Now, if you see, what is the batch count? It's 0. Since it's the very first loop, it's 0. So, if we substitute 0 for the batch count here, it will be multiplied by 0 in the front and 25 in the back. So, the batch size is 25, and the batch count is 0. This means we will slice the first 25 from the shuffled indices we had earlier. Let's name this batch index. Earlier, we randomly shuffled 1000 indices. So, since we sliced only 25 from those 1000 shuffled indices, the batch index has a length of 25.

There is an index list. More precisely, it is a list of NumPy arrays with a length of 25. The element now is 943, and what was the function F? It was the cosine function. When you input the cosine function with a batch, it can define the true answer like this. Let's try the second loop. This time, the batch count is 1. It's 1 for the batch count. So, if you input 1 here, there are 25 from before, and this one is 25, and then this one is 25 plus 1, so it will be 50 here. This means that we will slice from the 25th to the 49th index of the indices we mixed earlier. Next is the same. Since we sliced from 25 to 49 of the previously mixed 1000 indices, this 25-length NumPy array list is obtained, where 50 is index 25 and 49 is index 49. Then, in the same way, at the corresponding index.

We will extract the relevant data from the training data. So, in this second batch, these kinds of data will be extracted. When we input this, the truth, in other words, the correct answer, can be defined. How many batches did we say this batch size is? 40, right? So, with this batch size of 40, we will repeat it 40 times for batch processing. Up to this point, we have seen how the batch function was implemented. Now, we will put the divided batches and the truth values obtained from the true/false function into tensors, converting both the correct answers and the divided batches into tensors because our model is implemented in PyTorch now. After converting them into tensors, we will input these batches into the model, and after inputting them into the model, we will calculate the loss using the loss function with the prediction values and the true values. Our loss function uses Mean Squared Error.

We did that, and we need to initialize the gradient once because the gradients accumulate. So, we need to initialize this accumulated gradient once, and then we can use automatic differentiation functions to calculate the derivative values. After calculating the derivative, we proceed with gradient descent. This is no different from what we learned in our previous practice, right? In our previous practice, we did the same thing. You can try learning like this. Shall we try training once? As you can see, the loss is steadily decreasing. Yes, it's steadily dropping. Since we did 1,000 epochs, it will probably iterate up to 1,000 times. Also, regarding this, what's worth mentioning is that the process of implementing batches was quite complex and difficult. However, in fact, PyTorch provides batch functionality easily through libraries.

I will help you see, I have written down some references. By using the library called TensorFlow Dataset and the data loader in this library, you can easily divide and shuffle batches. First, prepare the training tensors and then use the function of TensorFlow Dataset within the dataset to insert the training data we have prepared. You need to put it into the data loader, specify the batch size here, and also indicate whether to shuffle here. Then the data loader will automatically apply the batching function. It would be good to know the data loader function as it will be widely used in projects. Implementing the batching function directly in projects will probably be complex, but if you use the data loader function, batching will be implemented very easily.

Here, if you look, there's this batch dancer, and ah, batch tensor. Batch tensor is just declaring true tensor as a batch, then it will be divided directly into batch size and true tensor units. By extracting batches in the data loader, it becomes simple, right? Then you can see that the code has become very simple. Below, it might look the same, but if you see, the code becomes very simple in just one line. This batch function is being implemented. However, since we are studying after all, we have delved a bit more specifically into the batch principle. Probably the training is all done. Since the training seems to be done, let's draw a graph once to check if the training is done properly. If you see, you can see a perfect match. The endpoints are slightly different, but here, it seems a little different, but almost.

You can see something matching the cosine graph, which is a huge difference compared to what we saw earlier. Let's press this and take a look at the loss change. It drops steadily, reaching 0, and you can even see it converging to 0. Below is mine, and below that is mine as well. I just added a reference once. What is this? Shall we press this to see? This is each angle, so I mean, I graphed each result obtained from each node. What does this mean? Currently, we are calculating our process and summing up each of these. We need to sum them all up. Before adding them up, we have these. Before adding them up, we graphed each one of them one by one, one by one, one by one.

It looks like there is no connection with cosine function. Graphs that have no relation with cosine are being drawn. However, when we add up all the node results, it turns out like this. If you look now, these graphs drawn in various colors represent values obtained at each neuron. Once you arrange them, when we used neurons as inputs to the model earlier, you can see a total of 9 graphs. These graphs drawn in different colors and solid lines, totaling 9 graphs, but if you add them all up, you will get a black dotted graph. In other words, only when you add them all up, you finally get the desired output value. Through this graph, we have experienced the principle of a neural network.

By appropriately training each neuron and summing them up, we can obtain the desired model output. So, let's finish the practice here. Thank you.

2024 OUTTA 부트캠프 딥러닝반 실습 3-1 인공신경망1

OUTTA official

Practicing Approximation of Cosine Function with Artificial Neural Networks

Artificial neural network practice involves approximating a non-cosine function to resemble a cosine function using regression.

The practice focuses on defining model parameters and their shapes using PyTorch, with an emphasis on the number of nodes in the neural network.

The goal is to implement a specific calculation process to train the function to resemble a cosine function.

The Explanation of Model Parameters and Batch Processing in Deep Learning

Model parameters in matrix shape are used for X times V, representing the input of training data as a one-dimensional vector.

The process of inputting a thousand data points into the model, with the need to set hyperparameters like the number of nodes, is repeated for each node separately, including the addition of bias and the use of batch processing to reduce computations.

Performing Matrix Operations for Batch Sizes

The batch size is being increased to 255, allowing for 25 inputs to be processed at once, reducing the need for repetitive batching.

This results in the generation of 25 outputs simultaneously, improving efficiency.

By using the broadcasting function, the intercepts for each row when multiple input data are entered can be accurately summed, simplifying the process of operation.

Computation Process of 2-Dimensional Vectors in Neural Networks

2-dimensional vectors are inputted into n nodes for computation, with each node having specific parameters and undergoing an activation function.

The concept of batching can be applied, which involves grouping the individual data into matrices for input and output.

The process involves matrix operations and multiplication by a parameter matrix of shape n by 1, ultimately resulting in an m-dimensional vector.

The use of 'n' as a hyperparameter allows for the determination of the number of nodes in the neural network.

Building a Simple Neural Network for Cosine Function Approximation

A single-layer neural network is used with a sigmoid activation function to approximate the cosine function.

Increasing the layers in the neural network is mentioned for a deeper network, but the potential issue of gradient vanishing with a deep network is also highlighted.

The practice involves dividing input data into batches and training the model to approximate the cosine function.

Understanding the Training Process for a Cosine Function Model

The initial graph of the cosine function and the model show a mismatch due to the random initialization of the model parameters.

The implementation of batch training with a batch size of 25 is explained, emphasizing its importance in speeding up the learning process as the model and data size increase.

The process of creating and shuffling batch indices for training the model in a loop is outlined, highlighting how it allows for efficient utilization of the training data.

Implementing Batch Functionality with PyTorch and TensorFlow Dataset

The process involves slicing a list of NumPy arrays to create batches for batch processing, with a batch size of 40 being used.

The divided batches and truth values are then converted into tensors for input into a PyTorch model, where the Mean Squared Error loss function is applied.

The gradient is initialized and automatic differentiation functions are used for calculating derivative values before proceeding with gradient descent.

PyTorch provides an easier way to implement batch functionality through the TensorFlow Dataset library and the data loader, allowing for the simple division and shuffling of batches with specified batch sizes.

Understanding Batch Processing in Neural Networks

Batch tensor simplifies the data loading process by dividing true tensors into batches, making the code more efficient.

The practice demonstrates that by training and summing the results of each neuron in a neural network, the desired output value can be obtained, showing the principle of a neural network in action.

The process highlights the importance of appropriately training each neuron and combining their results to achieve the desired model output.