Blowing up the Transformer Encoder!

If you think I deserve it, please consider liking the video and subscribing for more content like this :)
Some corrections in the video: 2:38 the dimensions of each one hot encoded vector is max_sequence_length x vocab_size (I mentioned the latter incorrectly in the video)

A hidden gem on youtube, these explanations are GOATED. Thank you!

I have struggled to find a good explanation of transfomers and your videos are just amazing. Please keep releasing new content about AI.

With every new video from your Transformer series, I still keep learning something new, especially in clarifying some aspect that I didn't fully comprehend before.

You have a talent for reducing complex issues to the essentials and also illustrating them super. I was able to learn so much. Thank you for that! 🤓

This is a superb explanation! Your videos are immensely helpful, and are undoubtedly the best on YT.

Physics students pondering the forth dimension
Computer Scientists casually using 512 dimensions

Wow ! Excellent explanation ! Couldn't find this content anywhere except your channel. Thanks

The best explanation on internet. Thank you. Keep it up!!

Thanks really very helpful resource for me!
Keep rocking Ajay.

Absolutely amazing series! Thank you so much for explaining everything over these videos and especially the code and visual examples! I'm very excited to learn about the decoder when you're ready to cover it.
Perhaps for the descriptions of Q, K, and V, it might help to distinguish V not as "what we actually have" (I think) and instead as "what we actually provide". So "what we want," "what we have to offer," and "what we actually provide." That's at least how I understand it.

This is wonderful presentation! I finally understand more deeply about transformer. Thanks!

Well nicely structured and clearly explained. Thanks a lot. You deserve lot more subscribers. Once again thanks for putting so much time and efforts for making this playlist.

You are a very unique tutor. I love the way you explain everything from start in your every video. It helps us understand and learn the concept in so much depth that it won't be easy to ever forget these concepts

Absolutely the best detailed and visual explanations. None better.

Thank you so much! I finally have an intuition on how encoders work thanks to you😀

You are truly amazing! Thank you so much for your well-elaborated explanation.

bro you made me cry again. Thank you for this wonderful content

Thanks, very helpful. For me, I go over various sections more than once which is ok on line but would irritate you and others in a live class--but it helps me learn. What an exciting time to be doing neural networks after decades of struggle.

Good One Ajay

Hats off, man.

we scale weights by 1/sqrt(d_k) to avoid variance problems (q, k have variance of roughly 1. so q @ k.T will have variance of d_k (head_size). in order to make its variance 1 we divide by sqrt(d_k)) - otherwise softmax will have really high values (higher values, when passed into the softmax function, will converge to a one-hot vector, which we want to avoid :))

Best explanation i ever see, really

Thank you so much for this sir. Learning so much here.
I know many might disagree with the philosophical aspect of Mind, and dont mean to shoe-horn in but, I think these Transformer Networks are humanity's successful building of a Mind. Expressing intangible thought into semi-tanglible objects that one can piece together to wind up a thinking machine. Yet doesnt exist in the same 3D plane as physical objects, as Math doesnt exist in this plane, its in the Thought/Mind non-spacial dimension.

This video is really phenomenal! Thanks for all the hard works! Is it possible for you to share your diagram with us? 😀

Hello Ajay, another awesome video! I may have missed some parts. May I ask why running this 12 times as you said in the last part of video? Thanks.

can you do a deep dive into the embedding transform?

Thank you! What do you use to make your drawings and record your setup?

I think i finally understand transformers. Especially the qkv part.
In the first skip connection you add the positional encodings only, but i the original drawing it seems they are adding the (positional+base) embeddings in the residual connection. Can you please elaborate about that?

great content!!!!!!!!

Thanks a mil for your explanations! I have a little request. Do you think you could share the little "not so complicated" diagram you showed at the beginning of the video?
Thanks a mil!!!!

Thank you very much!

great

Hi. This video is an extremely perfect one.
at @5:30 the dimension of output from the qkv linear layer is 1536*max_seq_len? and each qkv matrix is 512*max_seq_len .

Hi CodeEmporium Team , Thanks for such great content. One question I have - When we use Transformers Encoder to encode any sequence to generate embeddings what loss function does transformer uses. For e.g. I am using Transformer Encoder to encode a sequence of user actions in a user session to generate embeddings to be used in my recommender system. Kindly answer

thanks for the video. I have a question about Wq, Wk and Wv. you mentioned that Wq is like the encoded original input 'My name is Ajay'. Then what about the Wk and Wv, as you mentioned Wk is what can it offer and Wv is what actually offered. does Wk, and Wv also represent 'My name is Ajay'? Thank you

The first layer where you talked about MAX_SEQ_LEN , Does that mean length of each one hot encoded vector is equal to vocab size.

Thank you for this great explanation. I think the multi-head explanation is inverted ( on purpose for simplicity i guess)
But i guess the idea is to start with a 64 dimensional QKV and then concatenate them to n heads in your case it s 8 heads. Also this way we can have the possibility to concatenate them or just get the mean of the 8 heads.

This is great! Can you please share the encoder arch diagram file you are explaining here. please....

-----> BERT -------> 👏👍

Thanks so much for this video, just wondering if there's a difference in encoding in the Vision Transformer model

Hi Ajay, thank you so much for these transformer breakdowns, they're great! One thing that is confusing me about the 'initial encodings' step, whereby you transform the input tokens to their respective one-hot vectors; your diagram shows that as a SLx SL vector. My question: is this encoding trying to preserve positional information or is it trying to uniquely identify the token? I had thought it was the latter, which would mean it shouldn't be SL x SL, it should be SL x Vocabulary such that the one hot encodings can represent any token in the 'language' not just those in the input sequence.

can you make a video on vision transformers please

do i need to decoder in image classification task, or i just need to encoder part ?

Does anyone know where the drawing in the video?
I really need that to take a deeper look myself

Sir plz start yolov8 plz

Two questions for the Input:
1) If you do One Hot Encoding: Is the matrix size really "Max Sequence Length x Max Sequence Length" - or shouldn't it be "Max Sequence Length x Dict Length"?
2) Is it really necessary to do One Hot Encoding for the Input? I mean the words are encoded/embedded in this 512 dimensional vectors, so it doesn't matter how they are - initially - referenced, no?

I used think that all the heads take entire feature vector of token as input.
Now I understood it just takes part of a feature vector

Hello some one please help me if my max sequence length is different for input and output. For example if I am applying this text summarising. The input length of text for encoder is different which 4 times the summary length so where should I change the max sequence length after multi head attention of encoder or after normalisation or after feed forward network. Please suggest idea about it.

2:38: Do you mean Max Sequence Length x Dictionary Size? (the one-hot vectors must be able to encode every single token in the dictionary)

why is the embedding size max_seq_len X max_seq_len ? shouldn't it be max_seq_len X vocab_sizze

can i understand the heads here as the kernels in CNN

In the summarized diagram there is no “skip connection” for positional encodings but for values. Just after you explain residual connections you tell about an add operation and I then expected that that would be the value, because that is what is in the summarized diagram, but in your expanded diagram it is the positional encoding. And you never have a + for the value in your expanded diagram. What does this mean? 1. Is the summarized diagram leaving out details (forgetting the positional encoding skip connection) or 2. did you accidentally forget to draw in the value skip connection or 3. did you confuse values with positional encodings because the expansion is so huge? I was very confused about that part. But very nice presentation overall!

you are the best 🥵❣

Is it 12 or 6? I think we use 6 encoders and not 12.

Shouldn't it be MAX SEQENCE LENGTH x VOCAB SIZE?

Looking forword to get simillar video for decoder.

Your video is 99.9% informative, please provide the image you are showing to make it 100%

I was able to nod along and pretend I understood until 19:14. "We actually execute all of these kinds of roles multiple times over [...] like 12 times [...] cascaded one after the other". Do you to say that the entire block is composed with itself? I'm struggling to understand why the encoder would be applied like so: (f (f (f (f (f (f (f (f (f (f (f (f x)))))))))))), or f^12(x).
Is the dimensionality of the embedding decreasing with each step, like the gradual down-sampling an image in a diffusion model? Or is it something else? Is there any intuition here?

Am I being dumb or do you need to perform a values = values.permute((0, 2, 1, 3)).reshape((batch_size, max_sequence_length, 512)) instead of just a reshape? The thing is this would not put the words back together in the right order after multi-head-attention, would it? Some code I ran to test this:
def f(x):
... return x.reshape((x.shape[0], x.shape[1], 8, -1)).permute((0, 2, 1, 3)).reshape((x.shape[0], x.shape[1], -1)) # compact version of the shape / ordering transforms happening in attention (attention itself doesnt change the shape: initial_v.shape = values.shape)
...
>>> def g(x):
... return x.reshape((x.shape[0], x.shape[1], 8, -1)).permute((0, 2, 1, 3)).permute((0, 2, 1, 3)).reshape((x.shape[0], x.shape[1], -1))
...
>>> v = torch.arange(120)
>>> v = v.reshape((1, 3, 40))
>>> torch.all(v == f(v))
tensor(False)
>>> torch.all(v == g(v))
tensor(True)

6:40 if you're implying that the batch dims communicate with eachother, that's wrong as far as I know.

Ajay I guess dk is 64. And square root of it is 8. It is done to stable the gradients.

Might you consider creating a Discord guild?

houu ơ du