Transformer Encoder in 100 lines of code!

If you think I deserve it, please consider hitting the like button and subscribe for more content like this :)

This is the best explanation I have gone through

Superb and so love these classes! Will watch all of them one by one

I watched the entire series and it gave me a deeper understanding on how all of this works. Very well done!!!! Takes a real master to take a complex topic and break it down in such a consumable way. I do have one question: What is the point of the permute? Can we not specify the shape we want in the reshape call?

Next level video *especially* because of the dimensions laid out and giving intuition for things like k.transpose(-1, -2). Likely the best resource out right now!! Thanks for all your work!

Very clear, useful and helpful explanation! Thank you!

This video was really informative. Thank you for all the detailed explanations!

It's really helpful that you are going through all the sizes of the various vectors and matrices.

This is the best content on youtube

bro you're a legend!

Just amazing!!!

@CodeEmporium
The transformer series is awesome!
It is very informative.
I have one comment, It is usually recommended to perform dropout before normalization layers. This is because normalization layers may undo dropout effects by re-scaling the input. By performing dropout before normalization, we ensure that the inputs to the normalization layer are still diverse and have different scales.

This is the most detailed Transformer video, THANK YOU!
I have one question, the values is [30, 8, 200, 64], before we reshape it, shouldn't we permute it first? like:
values = values.permute(0, 2, 1, 3).reshape(batch_size, max_sequence_length, self.num_heads * self.head_dim)

Thank you, I going through all your videos. great work!

Really friendly for the beginners!😁

thank you!

bro... i love how u dive deep into explanations. You're a very good teacher holy shit

You are awesome .The way you teach is incredible.

Appreciate your work! As someone else mentioned, hope you can do an implementation of training the network for a few iterations.

Thanks for the great series. Would be very helpful if you'd attach the Colab.

Wonderful explanation

you are awesome bro

Awesome content as always ! Are you planning to demonstrate a training example of training for the encoder for the next video ? For example on a wikipedia data sample or something like that ?

awesome content! thanks a lot!!

Hats Off to you for explaining such a complex topic with simplicity and understanding. Thanks a lot. Is there any course you're offering besides these awesome videos on youtube? Want to learn more concept from you.

Please, blow up the decoder as well hahaa !!
Thank Ajay, these videos were very helpful for me.

Hi Ajay. I think, we need to make a small change in the forward() function of the encoder class. We should be doing `x_residual = x.clone() # or x_residual = x[:]` instead of `x_residual =x`. This will ensure that x_residual contains a copy of the original x and is not affected by any changes made to x.

thank u a lot

Amazing video series! At 39:07, why does the layer normalization just consider 1 dimension, the length of parameter shape, but not consider the batch size? Your previous video about the layer normalization mentioned layer normalization should consider both. Am I missing something?

Great video!!!!best content for transformer... Ca n you suggest ways to implement transformer encoder for a time series data

Thanks!
Please do Cross Attention and maybe Attention visualizations next!

Thanks!

I am looking forward to see whether you will try to put all the bits of the transformer together i.e. the positional encoder before this "encoder" and then the decoder after. I wonder whether/how it will respond to the input text "My name is Ajay". Would it respond as though in a conversation "Hi, how are you" / "My name is Bot" or generate more text in the same vein e.g. "I am 28 years old" or translate it to another language or something else. To achieve an end-to-end use case I guess we will also need appropriate data to be able to train the models and then actually train the models, save the model weights somehow, etc. Am new to all this but your videos are gradually helping me understand more e.g. encoder input and output matrix being of the same size to permit stacking. Thanks 👍

What would be the best book to learn probability and statistics for Machine Learning?

Hi. Great video but i have a question. Referring to 19:31, why is the dimension of k found by using the code q.size()[-1], shouldn't it be k.size()[-1] instead. Thnx in advance:)

I understand how the forward way works, but not how the learning works. Basically all videos I have seen so far covering Transformers "only" explain the way forward, but not the training. For example I'd like to know what the loss function is.
Question 2: afaik an Encoder can work on its own and doesn't (necessarily) need a Decoder (for example for non-translation use cases). How does the training work in this case? What is the loss function here? (-> we don't have a target sentence)

Overall your explaination is great, But I little confiused. Actually i could not understand the difference between positinal encoding and Position-wise Feed Forward Network. Can anyone explain to me?

your code is pretty clean, except i more like "black" code formatting

I know it’s a lazy question, but can someone tell me why is multi-head better than single head for performing attention?

a video about how to code chatgpt in which the code is generated by chatgpt 😁

where did you get the 3 for 3 times 512 =1536? Is it 3 because you have query, key, and value?

I think you forgot to address in you MHA code to pass the mask value.. I think here you need ModuleList and can't use nn.Sequential

Yolov8

Did he just mimic what Andrej Kaparthy was doing. Explanation not even 10% as clear as what Andrej did. So bad.