Why left would be preferred?
Why the right is better?
The single best prediction of the parameters
A good estimator is a function whose output is close to the true underlying thetha that generated the data
Bias
Estimator’s expected value= so the ouput of our thetha estimate and the true value of that parameter
Bias comes from not being able to model the real model in a correct way
Variance
So if you use a different training set or a different split how different are the learnt NNs parameters. If differ a lot then high variance
High variance & low bias, your model on average get it right but it has a high variance because it wants to model all noise, so overfilling, it spread all over the place
High bias and low variance, in average it does not even get it correct so underfitting
It will learn to recognize that the img contains hourses not by the hourses but the watermark. This is seen in the heat-map where the point of attention is in the watermark. This is overfitting
To avoid overfitting we use regularization
Dash line is where the data samples equals the number of parameters
So in the top right image even if we have increase the number of hidden layers and our training error is zero then our test set is still decreasing that is weird.
So here we then presume that bigger model will have lower error.
Before we use to have a curve upwrads from the tipical bias/variance curve in the prev slide, but the weird thing is that where the dash line meets then this error starts to decrease again.
Two answers: smoothness and regularization
In the x axis you have the number of times SGD was proceed, so the amount of units of texts akak ‘words’ were processed
We can see that with larger models I require fewer samples to reach a lower test loss
Also when they reach stability they can provide with a test loss which is the best shot they can give and this depends on their size, larger models give more accurate results
Also it shows that the quicker models learns quickler than the smaller model. So while the smaller model for a given number of tokens the large model learns more quickler.
This is Language models still not applicable for vision.
We also can say that (in language models) in terms of flops is more efficient to learn large models for fewer steps than to learn small models for larger amount of steps
In practice if you increase the number of neurons you may be closer to overfitting. For that we need regularization
Here we reduce the complexity of a NN and avoid ovverfitting
Referred to weight decay or Ritch regression in the linear case
Omega is proportional to how large the weights are
Minimizing this is also the same as if you assume a Gaussian prior on your weight, here you assume the weights are Gaussian distributed
The L2 loss, also known as the Euclidean loss or Mean Squared Error (MSE), is a common loss function used in regression problems. It measures the average squared difference between the predicted values and the actual values.
The formula for L2 loss is given by:
\(L2\_loss = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2\)
Where:
Here’s a simple example in Python using NumPy:
import numpy as np
# Generate some example data
true_values = np.array([2, 4, 5, 4, 5])
predicted_values = np.array([1.5, 3.5, 4.8, 4.2, 5.2])
# Calculate L2 loss
l2_loss = np.mean((true_values - predicted_values)**2)
print("L2 Loss:", l2_loss)L2 Loss: 0.12400000000000003The L2 loss is calculated by taking the mean of the squared differences between the true and predicted values. The smaller the L2 loss, the better the model’s predictions align with the true values.
Lamda tells you how important the regularization is, if:
Lambda during trainnig is fixed, if you want to find the best one you need to run lots of experiments
Isotropics means is equal in all directions
Now if you take the gradient of this, now if you take derivative of w, because you want to optimize then it end ups a 1/2. So you end up with a constant factor in the loss that keeps getting substracted. So what happens is that we end up substracting a constant to our weights.
So it substract a bit from the positive weights and adds a bit to the negative weights and pushed them towards zero
L1 leads to sparce weights, so that means more weights will be closer to zero
If alpha here increases then that means weights become zero
L2 regularization has basically a circular constraint area because you have w1^2 + w2^2 needs to be constant so all these combinations would be a circle
So the contours of the loss function in red will intercept the constrains regions at an axis. What that means is that if you are trying to find in this case the optimal loss, then it touches the constraint region where one of the values is set to zero while with L2 regularization there is no particular point where you could make one of the weights zero
This is because it needs to have the sum of the squares to be a small value but there is no particular motivation to have a similar weigth dimension to be equal to zero, so there is no reason to have sparse weights in L2
Here the alpha goes from strong to weak
In L1 it puches some weights to zero and then stay zero aftwer a while and it is not like all of them get smaller at the same time but some of them stay quite high for a loong time, and now increasing alpha here you are making individual weights close to zero.
L2 Regularization (Weight Decay): Encourages smaller weights but does not force them to be exactly zero. It smoothens the weights but doesn’t induce sparsity.
L1 Regularization: Promotes sparsity by adding a penalty term based on the absolute values of the weights. This can lead to some weights being exactly zero.
*Typo: with better test set error
The model at this stage have low variance because they are not overfitting
Here weight decay they mean by L2-regularization
Weight decay, also known as L2 regularization, is a technique to prevent overfitting by adding a penalty term to the loss function that is proportional to the squared magnitudes of the weights. This regularization term discourages the model from learning very large weights and encourages a smoother and more generalized solution.
Early Stopping:
Early stopping is a regularization technique used during the training of a machine learning model, typically in the context of iterative optimization algorithms like gradient descent. The idea behind early stopping is to monitor the model’s performance on a validation set during training and stop the training process when the performance on the validation set starts to degrade, even if the performance on the training set continues to improve.
Weight Decay (L2 Regularization):
Weight decay, also known as L2 regularization, is a technique to prevent overfitting by adding a penalty term to the loss function that is proportional to the squared magnitudes of the weights. This regularization term discourages the model from learning very large weights and encourages a smoother and more generalized solution.
Key Differences:
In practice, these techniques can be used together to enhance the regularization effect and improve the generalization performance of a machine learning model.
You switch the activations to 0, and now say with Bernulli you have 50% neurons working
During testing you are not learning so you use all the neurons
Bagging, also known as bootstrap aggregation, is the ensemble learning method that is commonly used to reduce variance within a noisy dataset. It is train in parallel
Bagging:
Dropout:
Putting data into a common shape without distorting tis shape
Here basically if we have in same scale then our weights will not be elongated like in the elipse, now they would be able to take the same step size in the correct direction
This we apply in the input stage:
Normalization is a linear operator so you can put this back into the NN after you have trained if you want to
Here we talked about the normalization within the NN
Batch Normalization (BatchNorm) is a technique used in neural networks to improve the training stability and speed by normalizing the inputs of each layer. It was introduced by Sergey Ioffe and Christian Szegedy in their paper “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.”
Here’s a high-level overview of how Batch Normalization works:
Normalization:
\(\hat{x}^{(k)} = \frac{x^{(k)} - \mu}{\sqrt{\sigma^2 + \epsilon}}\)
Where:
Scale and Shift:
\(y^{(k)} = \gamma \hat{x}^{(k)} + \beta\)
Where:
Training and Inference:
Batch Normalization has become a standard component in many deep learning architectures due to its effectiveness in improving training stability and convergence speed.
Because some layers will push ouputs in one direction, then other layers will not use this. With batch norm we centered data so that all layers train around these centered inputs
There is no answer here
The important thing is that when you go to test time, you dont have batches (you dont want anything that is depended on how you construct the batch) because that means if you take another batch is not the same which then is not reproducible.
So what we usually do is keep a moving average of the mean and variance during training, and then at test time you plug them
Basically you extract the mean and the variance form the training and use it in test data
Better explained here
Here mean and variance are not computed across batch but across all channels and spatial dimensions.
So now the statistics are independent of the batch size because now they depend on the feature dimensions see example below:
Layer Normalization is a normalization technique similar to Batch Normalization but operates on a per-sample basis rather than per-minibatch. It normalizes the inputs of a layer across the features (dimensions) for each individual sample. Here’s an example of how Layer Normalization is typically applied:
import torch
import torch.nn as nn
# Assuming input has shape (batch_size, num_features)
input_data = torch.randn(32, 64)
# Layer Normalization
layer_norm = nn.LayerNorm(normalized_shape=64)
output = layer_norm(input_data)
# Display input and output shapes
print("Input shape:", input_data.shape)
print("Output shape:", output.shape)Input shape: torch.Size([32, 64])
Output shape: torch.Size([32, 64])In this example:
input_data is a random tensor with shape (32, 64), representing a batch of 32 samples, each with 64 features.
nn.LayerNorm is the Layer Normalization layer provided by PyTorch. The normalized_shape parameter specifies the number of features in the input tensor.
The output tensor is the result of applying Layer Normalization to the input data.
Layer Normalization normalizes the values along the feature dimension independently for each sample. This means that each feature in a sample is normalized based on its mean and standard deviation across the entire sample, rather than across a batch as in Batch Normalization.
Layer Normalization is useful when the batch size is small or when working with sequences of varying lengths, as it normalizes each sample independently. It has been widely used in natural language processing tasks and recurrent neural networks.
Basically, we have that our mean and std will be computed across dimensions, so on the columns, not on each input. So then each sample (so each row) we make it with this mean and std to be distributed between 0 and 1.
This is great for RNN, or other stuff that requires small batch sizes.
Here the same operations happens at training and test time.
So now instead of normalizing across data samples now we basically i.e. if the input is an image that each color should be roughly be ocurring the same amount of spread across all the image, because now we normalize it across channel dimensions
Instance normalization now you do layer normalization but per channel and per training example.
So now the network should be agnostic to the constract of the original iamge and of the constrast whithin the channels
Not used that often
Here we compute the mean and var per sample but not per channel.
Instance Normalization is a normalization technique similar to Batch Normalization and Layer Normalization but operates on a per-instance basis. It normalizes the activations of each individual sample independently. Here’s an explanation of how Instance Normalization works:
Input Tensor:
Calculate Mean and Variance:
\[\mu_c = \frac{1}{H \cdot W} \sum_{i=1}^{H} \sum_{j=1}^{W} X_{n,c,i,j}\]
\[\sigma^2_c = \frac{1}{H \cdot W} \sum_{i=1}^{H} \sum_{j=1}^{W} (X_{n,c,i,j} - \mu_c)^2\]
Normalize:
\[\hat{X}_{n,c,i,j} = \frac{X_{n,c,i,j} - \mu_c}{\sqrt{\sigma^2_c + \epsilon}}\]
Where:
Scale and Shift:
\[Y_{n,c,i,j} = \gamma_c \hat{X}_{n,c,i,j} + \beta_c\]
Where:
Instance Normalization is often used in computer vision tasks, especially in scenarios where the batch size may be small or when normalization across instances is desired.
We are gonna group different certain channels toguether. So we are not normalizing per channel but per groups, for instance 5 channels.
If you have only one group then you recover layer nomalization because layer normal normalizes across all channels and not per channel
If you have more number of channel group toguether i.e 3 channels group together then you do Instance normalization. meaning you compute the mean per sample across each channel
In Grouped Convs, you are basically separating the hidden layers or the hidden channels in hidden groups which makes computations more easier
This is better than batch normalization for small batches <32
You can thing the weights like a vector, you can have its magnitude and its direction. Now with this g can learn this parameter which tells you how long you want to go in that direction
To achieve convergence you need tehse two equations:
You also make a warmpup learning rate so you start by going up in a linear fashion
We do all of the above,
Architecture, which model do you use?
Progress is not only in the architecture side, for example if you want to develop better algorithms, you want to have just a neural network which can identify between cats and dogs, there is multiple ways:
for classifying 8 classes, the loss should be -log(1/8) and you check whether is true, if it gets worst performance that randomly guessing so something is wrong
