Batch normalization

But how do we do that when our neural net now in a forward pass estimates the statistics of the mean understanding deviation of a batch? The neural net expects batches as an input now.

So how do we feed in a single example and get sensible results out? And so the proposal in the batch normalization paper is the following.

What we would like to do here is we would like to basically have a step after training that calculates and sets the BatchNorm mean and standard deviation a single time over the training set.

And so I wrote this code here in the interest of time, and we're going to call what's called calibrate the BatchNorm statistics.

And basically what we do is torch.nograd telling PyTorch that none of this, we will call a dot backward on, and it's going to be a bit more efficient.

We're going to take the training set, get the pre-activations for every single training example, and then one single time estimate the mean and standard deviation over the entire training set.

And these are not trained using backpropagation.

These are trained using this janky update, kind of like a running mean update.

and instead we're going to analytically derive using math and paper and pencil the gradient of the loss with respect to the logits and instead of back propagating through all of its little chunks one at a time we're just going to analytically drive what that gradient is and we're going to implement that which is much more efficient as we'll see in a bit Then we're going to do the exact same thing for BatchNormalization.

So instead of breaking up BatchNorm into all the little tiny components, we're going to use pen and paper and mathematics and calculus to derive the gradient through the BatchNorm layer.

So we're going to calculate the backward pass through BatchNorm layer in a much more efficient expression instead of backward propagating through all of its little pieces independently.

So that's going to be exercise three.

And then in exercise four, we're going to put it all together.

Okay, next up, we have dhPreact, and we want to backpropagate into the gain, the bnRaw, and the bnBias.

So here, this is the bash norm parameters, bnGain and bias, inside the bash norm, that take the bnRaw, that is exact unit Gaussian, and they scale it and shift it.

and these are the parameters of the BatchNorm.

Now, right now, for our model as is, this doesn't actually do anything because the only thing inside this model is this nn.embedding. And this network would behave the same in both evaluation mode and training mode.

We have no dropout layers, we have no BatchNorm layers, etc.

But it is a good practice to think through what mode your neural network is in, because some layers will have different behavior at inference time or training time.

There's also this context manager torch.nograd, and this is just telling PyTorch that everything that happens inside this function, we will not call .backward on.

And we also see that the network is starting to get big enough that our train loss is getting ahead of validation loss.

So we started to see a little bit of overfitting.

And our... generations here are still not amazing but at least you see that we can see like is here this now grief sync like this starts to almost look like english so um yeah we're starting to really get there okay and the second innovation that is very helpful for optimizing very deep neural networks is right here so we have this addition now that's the residual part but this norm is referring to something called layer norm so layer norm is implemented in pytorch it's a paper that came out a while back here and layer norm is very very similar to batch norm so remember back to our make more series part three we implemented batch normalization and batch normalization basically just made sure that across the batch dimension any individual neuron had unit Gaussian distribution so it was zero mean and unit standard deviation one standard deviation output So what I did here is I'm copy pasting the BatchNorm1D that we developed in our MakeMore series.

And see here, we can initialize, for example, this module, and we can have a batch of 32 100 dimensional vectors feeding through the BatchNorm layer.