Training Neural Networks: Activation Functions, Preprocessing, and Initialization

Batch normalization improves network training by ensuring unit Gaussian activations through mean and variance normalization at each layer.

### Training Neural Networks: Activation Functions, Data Preprocessing, and Weight Initialization
* 🔘 "Activation functions are used to **introduce non-linearity** to the neural network and **determine the output of a neuron**." Common choices include the **sigmoid, tanh, and ReLU functions**.
* The sigmoid function "suffers from the **vanishing gradient problem** due to its flat regions, while tanh is zero-centered but still has **saturation issues**."
* 🚀 ReLU is "the most widely used activation function due to its **fast convergence, efficiency, and biological plausibility**."
* 📊 Data preprocessing typically involves "**zero-centering the data to address bias issues** and **normalizing it to ensure features are in the same range**."
* 🎚 "Weight initialization is crucial to **break symmetry** and prevent the **vanishing gradient problem**." Initializing with small random values helps in shallow networks but leads to the **"vanishing gradient problem"** in deeper networks.
* Common techniques to address the vanishing gradient problem include using "**leaky ReLU, parametric ReLU, and ELU**", which provide solutions while maintaining **efficiency and faster convergence**.
* 📉 "The dying ReLU problem occurs when neurons in the negative regime never **activate and update**." This can be mitigated by using variants like **leaky ReLU**.
* 📷 In image data preprocessing, "zero-centering by **subtracting the per-channel mean** is commonly used, as images already have a **relatively uniform scale and distribution**."

### Training Neural Networks: Best Practices and Tips
* 🔥 Use ReLUs as activation functions for **faster convergence** and to **avoid the vanishing gradient problem**.
* Subtract the **mean** from the data as a preprocessing step to **center the data distribution**.
* 🌱 Use **Xavier Initialization** for weight initialization to prevent the gradients from vanishing or exploding.
* 🥛 Apply **batch normalization** to each layer to **stabilize and speed up training**.
* 👶 "Babysit" the learning process by checking and adjusting **hyperparameters, such as learning rate**, regularly.
* 🎯 Perform **hyperparameter optimization** using techniques like **cross-validation** to find the **best set of hyperparameters**.
* 📈 Monitor the loss curves, training and validation accuracy, and weight update magnitudes to diagnose and address issues like **overfitting, underfitting, and exploding gradients**.

Lecture 6 | Training Neural Networks I

The Stanford School of Engineering has been at the forefront of innovation for nearly a century, creating pivotal technologies that have transformed the worlds of information technology, communications, medicine, energy, business and beyond. 

The faculty, students and alumni of Stanford Engineering have established thousands of companies and laid the technological and business foundations for Silicon Valley. Founded in 1925, the school has a tradition of pursuing multidisciplinary collaboration aimed at solving the most pressing global problems.

Stanford University School of Engineering

Training Neural Networks: Details and Processes

Assignment deadlines and project proposal due dates are announced, with a reminder to start thinking about projects.

The upcoming lectures will cover activation functions, data preprocessing, weight initialization, batch normalization, training dynamics, and hyperparameter optimization.

The lecture will focus on the trade-offs between different activation functions, starting with the sigmoid function and its problems with gradient saturation, especially in the very negative region.

The discussion will also include the interpretation of sigmoid as a firing rate of a neuron and the upcoming focus on more biologically plausible nonlinearities like ReLU.

Problems with Sigmoid, Tanh, and the Superiority of ReLU Activation Function

The sigmoid and tanh activation functions suffer from the problem of killing gradient flow in saturated regions and being non-zero-centered, leading to inefficient gradient updates.

In contrast, the ReLU activation function does not saturate in the positive region, making it superior in preventing gradient flow problems.

Additionally, ReLU is computationally efficient due to its simple max operation, which is a significant advantage over sigmoid and tanh.

The Superiority of ReLU Activation Function in Neural Networks

ReLU converges six times faster than sigmoid and tanh, making it more biologically plausible for neurons.

Despite its issues with being zero-centered and causing dead ReLUs, it is still widely used in training networks.

The introduction of leaky ReLU has provided a solution by offering a slight negative slope in the negative regime.

The ReLU activation function is computationally efficient and converges faster than sigmoid and tanh.

Other variants like Leaky ReLU, PReLU, ELU, and Maxout Neuron offer different benefits such as zero mean outputs, flexibility in the negative regime, and avoidance of the dying problem.

In practice, ReLU is the most commonly used, but experimenting with other activation functions can be beneficial depending on the specific problem.

Data preprocessing, including zero centering and normalization, is also essential for optimizing the training process, especially in machine learning problems.

Normalization of pixel values is not heavily emphasized in image-related tasks, as the features already have a relatively comparable scale and distribution.

Image Data Pre-processing for Convolutional Networks

Image data pre-processing, specifically for convolutional networks, involves zero mean pre-processing by subtracting the mean image or a per-channel mean, which is applied to both the training and test data.

This helps in maintaining the spatial structure of the original image and is commonly used in networks like VGG.

The pre-processing solves the zero mean problem for the first layer of the network, but as the network gets deeper, it may not be sufficient, leading to non-zero mean problems.

The Problem with Zero Initialization of Weights

Initializing all weights to zero in a neural network causes a symmetry problem where all neurons perform the same operation, receive the same gradient, and update in the same way, preventing them from learning different features.

This lack of symmetry breaking leads to the neurons behaving identically, which is not desirable in the network's learning process.

The gradient may vary based on the loss in the last layer, but the overall symmetry in

Improving Neural Network Training with Small Random Weights

Using small random weights from a standard Gaussian distribution with a scaled standard deviation can break symmetry in small networks, but it leads to problems in deeper networks.

The issue with small random weights in deeper networks is that it causes the activations to quickly shrink and collapse to zero, leading to the vanishing gradient problem.

Attempting to solve this by using large weights results in the saturation of the tanh activation function, leading to another set of problems.

Weight Initialization and Batch Normalization in Neural Networks

Neural network weight initialization is crucial for proper training, with the Xavier initialization providing a good rule of thumb by scaling weights based on the number of inputs to address the issue of vanishing or exploding gradients.

However, this method breaks down when using ReLU activations, requiring an adjustment by dividing the scale by two to account for the variance reduction caused by half of the neurons being deactivated.

Batch normalization is introduced as a method to force unit Gaussian activations by empirically calculating the mean and variance of each batch of activations and normalizing them during the forward pass of the network.

This approach aims to maintain the desired activation distribution at every layer throughout the training process, offering a functional way to achieve the goal of unit Gaussian activations.

The Concept of Batch Normalization in Neural Networks

Batch normalization involves computing the mean and variance for each feature dimension in the input data, normalizing the data, and then applying scaling and shifting factors.

This process improves the gradient flow, making the training more robust and allowing for a wider range of learning rates and initialization methods.

Additionally, batch normalization acts as a form of regularization by producing non-deterministic values for a given training example and tying together all the inputs in a batch.

The Flexibility of Batch Normalization in Neural Networks

Batch normalization allows for the flexibility to control the normalization of neurons, which is useful for non-ideal situations like the tanh activation function.

It is a common practice in standard convolutional neural networks and can still be effective with small batch sizes, although the normalization for different types of networks is still an active area of research.

Batch normalization does not force the inputs to be Gaussian, but rather shifts and scales the data, preserving the structure, especially in convolutional layers.

Understanding Batch Normalization in Neural Networks

Batch normalization layers use the empirical mean and variance from the training data during test time.

The process of monitoring training involves zero-meaning the data, choosing the architecture, and initializing the network.

Adjusting the regularization and training with a small amount of data helps in determining the appropriate learning rate.

Choosing a suitable learning rate is crucial, as a rate that is too small may result in minimal change in loss, while a rate that is too large can lead to NaNs in cost.

Optimizing Hyperparameters for Neural Networks

Adjusting the learning rate is crucial for controlling cost explosion during training.

Cross-validation is essential for finding the best hyperparameter values, typically done in stages with a wide range of values at first, followed by a finer sampling.

Detecting explosions in the cost function can help in real-time adjustment of hyperparameters.

It is advisable to optimize in log space and ensure that the range of values covers the optimal range for the hyperparameter.

Random sampling of hyperparameters is more practical than a grid search for hyperparameter optimization.

Tips for Neural Network Hyperparameter Optimization

More samples of important variables provide a more useful signal overall in hyperparameter optimization.

Hyperparameters to focus on include learning rate, decay schedules, update types, regularization, and network architecture.

Monitoring loss curves can help identify the right learning rate and potential issues with bad initialization.

Tracking the gap between training and validation accuracy helps in adjusting regularization strength and model capacity, while also monitoring the ratio of weight updates to magnitudes aids in debugging potential problems.

The key takeaways for hyperparameter optimization include using ReLUs, subtracting the mean for data preprocessing, employing Xavier Initialization, using batch normalization, and randomly sampling hyperparameters.