Layer Normalization - EXPLAINED (in Transformer Neural Networks)

Another great video - I like the structure you use of summarising the concept and then diving into the implementation. The code really helps bring it together as some others have commented. I look forward to seeing more of this series and would love to see a longer video of you deploying a transformer on some dummy data (perhaps you already have one - still going through the content)!

very clear and sound explanation for a complex concept, thumb up for the hard works!

the diagram is GOLD

Nice series on transformer. Really liked it. Btw interesting design choice for the video to use a landscape layout of the transform architecture during intro :D

Great! Since you’re covering transformer components, I would love to see TransformerXL and RelativePositionalEmbedding concepts explained in the upcoming videos! ☺️

best explanation ever!

I really like your voice and delivery. It’s quite reassuring, which is nice when the subject of the videos can be pretty complicated.

Great lectures.

Thanks Man ,
you deserve more than a like

if our batch has 2 vectors of 2 words x 3 embedding size say:
[[1,2,3],[4,5,6]] and [[1,3,5], [2,4,6]]
For layer normalization,
Is nu_1 = mean(1,2,3,1,3,5)
and nu_2 = mean(4,5,6,2,4,6)
Just wanted to clarify. Keep up the great work, brother. Like the small bite sized videos.

Great video! I find these very informative, so please keep them going! Question on the output dimensions though. In your transformer overview video the big diagram shows that after the layer normalization, you have a matrix of shape [batch_size, sequence_len, dmodel] (in the video 30x50x512 I believe.) However here you end up with an output matrix (out) of [sequence_len, batch_size, dmodel] (5x3x8). Do we need to reshape these output matrices again to [batch_size,sequence_len,dmodel], or am I missing something? Thanks again for all the informative content!

as per my understanding and from the LayerNorm code in pytorch, in NLP for an input of size [N, T, Embed], statistics are computed using only the Embed dim, and layer norm is applied to each token in each batch. But for vision with an input of size [N,C,H,W], statistics are computed using the [C,H,W] dimentions

Beautifully done video but isn't layer normalization essentially batch normalization layer?

Wow. Amazing channel

EXCITED!!!!!

the python example really helped to solidify understanding.

Thank you very much friend, very good!!

Thanks! This vedio helps me a lot

10:30 when layers normalizations are "computed across the layer and also the batch", are their means and std connected as if the batch boundaries aren't there? So that means there's a different learnable gamma and beta parameter for each word?

Amazing your expiation are so clear. Can this help with exploding gradients?

Have an actual question this time! While trying to understand the differences between layer and batch normalization, I was wondering whether it’s also accurate to say you are normalising across the features of a vector when normalising the activation function - since each layer is a matrix multiply across all features of a row, would normalising across activation functions be similar to normalising across the features?
In the same thread, can/should layer and batch normalization be run concurrently? If not, are there reasons to choose one over the other?

🔥🔥🔥

Great Video !! Just a question, why do we need to swap the dimensions and all that other stuff ? Why can't we do something like this
# assuming inputs is of format (batch_size,sentence_length,embedding_dim)
mean=inputs.mean(dim=-1,keepdim=True)
var = ((inputs - mean) ** 2).mean(dim=-1, keepdim=True)
epsilon = 1e-5
std = (var + epsilon).sqrt()
y = (inputs - mean) / std
y

Where can I find the reason why we need to calculate mean and standard deviation over the parameter shapes? In Pytorch, they just caculate over the last dimension, hidden size.

Is pytorch better for NLP task than Tensorflow?

Value Matrix should be on the right side of the multiplication with Attention Weights matrix

Given that we have 8 heads, Is it 512 / 8, which is 64? Are we actually going to split the 512 into 8 equal 64 length parts?

I am actually not sure why it is okay to reshape your input from (B, S, E) to (S, B, E) @ 10:10. It doesn't matter in this case as B=1, but in general wouldn't you be changing the your data? In particular, you would change the number of words in each batch, i.e., changing the maximum sentence length.