Self Attention in Transformer Neural Networks (with Code!)

If you think I deserve it, please consider liking the video and subscribing for more content like this :)

It's one of the best videos I have watched. The concepts are explained very much, specially with codes.

Excellent , how well you explained. NO other video on youtube explained like this , Really done good job.

This is great! I've been trying to learn attention but it's hard to get past the abstraction in a lot of the papers that mention it, much clearer this way!

What's important is that for every token generation step we always feed the whole sequence of previously generated tokens into the decoder, not just the last one. So you start with the token and generate now new token, then feed + into the decoder, so basically just appending the generated token to the sequence of decoder inputs. That might have not been clear in the video. Otherwise great work. Love your channel!

I usually don't write comments, but this channel really deserves one! Thank you so much for such a great tutorial. I watched your first video about Transformers and the Attention mechanism, which was really informative, but this one is even more detailed and useful.

I have learnt so much between yourself, ChatGPT, and Alexander & Ava Amini iat MIT 6.S191. Thank you all.

Thank you so much, I searched so many places, this is the first place finally have a nice person willing to spend time really dig in step by step. I'm going to value this channel as good as Fireship now.

Extremely well explained. Kudos !!!!

Best video explaining this that ive seen by far

I was legit searching for self attention concept vids and thinking that it sucked that you didn't cover it yet. And voila here we are. Thankyou so much for uploading!!

very happy to see kannada here
Great 😍Love from kannadigas

Best contents ever I've seen. Thanks for this video.

This Has been a most wonderful series on this channel so far

Fantastic explanation! Wow! You have a new subscriber. :) Keep up the great work

waiting for your future videos. This was amazing. especially the masked attention part.

This is exactly what I needed! Can't believe self-attention is that simple!

This is pure gold man!
Transformers are complex but this video really gives me hope.

One video to understand them all! Dude thanks I've tried to watch like 10 other videos on transformers and attention, yours was really super clear and much more intuitive!

Wonderful works you've done! I really love your video and have studied twice. Thank you so much!

The best explaination on the internet, thank you!

What can I say, dude!
God bless you
This is the only content on the whole youtube that really explain the self-attention mechanism in a brilliant way.
Thank you very much.
I'd like to know if the key, query, and value matrixes are updated via backpropagation during the training phase.

Great! This is exactly what I was looking for in all of the other videos of yours

Thanks a lot for making the video!! This deserves more views.

Thank you for this very clear and helpful demonstration!

With all my heart, you deserve a lot of respect
Thanks for the content. Damn I missed my metro station because of you.

Awesome! Well illustrated. Thanks

Thanks! U r very deserved and underdog!

Thank you for making this concept so easy to understand. Can’t thank you enough 😊

Simply wow..

My god, that was amazing, you have a gift my friend;
Love from Brazil :D

Great walkthrough of the theory and then relating it to the code.

Thank you so much for taking the time to code and explain the transformer model in such detail, I followed your series from zeros to heros. You are amazing and, if possible please do a series on how transformers can be used for time series anomaly detection and forecasting. it is extremly necessary on yotube for somone!
Thanks in advance.

This is brilliant, I've been looking for a bit more hands on demonstration of how the process is structured.

u have saved me. thank u.

This is an incredible video, thank you!

I really like how you use Kannada language examples in your explanations.

Amazing work! Very glad I subscribed

Huge respect ❤️

Thank you for teaching this.❤

Absolutely brilliant! Thumba chennagidhay :)

Good video Bro.
Studying Transformer this week but still a little confused before I met your video.
Thanks

this is amazing!

you made me cry brother

Thx for your efforts!

Excellent video . If you can please make a hello world on self attention like first showing pca representation before self attention and after self attention to show how context impacts the overall embedding

ನಮಸ್ಕಾರ ಅಜಯ್, ನೀವು ಕನ್ನಡಿಗ ಎಂದು ತಿಳಿದು ತುಂಬ ಸಂತೋಷವಾಯಿತು! ನಿಮ್ಮ ವಿಡಿಯೋಗಳು ತುಂಬ ಚನ್ನಾಗಿ ಮೂಡಿಬರುತ್ತಿವೆ.

Amazing video! Thank you😊😊

Brilliant Mate

Great video. Thank you very much.

Thank you for explaining in such an easy way. It would be great if you could upload some codes on GCN algorithm.

this is very helpful

Dear Ajay. Thank you so much for your videos!
I have a quick question here. Why did you transpose the values in the softmax function? Also... why did you specify axis=-1? I'm a newbie at this and I'd like to have strong and clear foundations.
have a lovely weekend :D

you save my life!!!!!

Your videos are way better than Stanford lecture cs224n

Mashallah, man you are a rocket.

Thanks!

I stop the video sometimes and roll it back some seconds to hear you explaining something again and I am like: "No way that this works, this is insane", some explanations on AI techniques are not enough and yours are truly simple and easy to understand, thank you.
Do you collab with anyone when making these videos, or is it done all by yourself?

Thanks, @CodeEmporium / Ajay for the great explanation!
One quick question- can you please explain how the true values of Q, K, and V are actually computed? I understand that we start with random initialization but do these get updated through something like backpropagation? If you already have a video of this then would be great if you can state the name/redirect!
Thanks once again for helping me speed up my AI journey! :)

Ustad 🙏

Got it, thank you very much, but one question: What I still don't understand: We are talking about neural networks, and they are trained. So all the math you show here, how do we (know|make sure) that it actually happens inside the network? You don't train specific regions of the NN to specific tasks (like calculating a dot product), right?

Thank you very much for this great video! Can you please upload a video on Self Attention code using Graph Convolutional Network (GCN)?!

really great video, but why have the Q, K, V Vektors dim 8? i know its random in this video but what does the values in the vektors say about the word? or is it just to "identify" a word in a certain room like in word embeddings and give it a certain "id" ?

How did we get the values of q, k and v vectors after initializing them as randoms. Great video btw. Waiting for more such videos.

which phase does the model take the pretrianed info to decide the output?

why is it d_k in both Q and K in the np.random.randn ?

how do we actually choose the dimensions of Q, K and V? Also, are they parameters that are fixed for each word in the English language, and do we get them from training the model? That part is a little confusing since you just mentioned that Q, V and K are initialized at random, so I assume they have to change in the training of the model.

thx

why we need to stabilise the variance of attention vector with query and key vectors.

I have a few doubts
1. Do we use multiple ffnn after the attention layer? So suppose we have 100 input words for the encoder then 100 ffnn will get trained ? One for each of the word, i checked the source code but they were using only one, so I'm confused how one FFNN can handle multiple embedding specially with batch size.
2. In decoder do we pass multiple input also, just like encoder layer specially in training part?

"query" , "key" , and "value" terms come from the world of databases! So how do individual words in "My name is Ajay" each map to their own query and key and value semantically? that remains a bit foggy. i know you've shown random numbers in the example, but is there any semantic meaning to it? is this the "embeddings" of the LLM?

Kannada Represent!! 🖐

Hi I noticed this has been added to the transformer playlist, but there are 2 unavailable tracks - do i need them in order to get the full end to end grasp?

I watched the video around 3 times but I still don't understand.
Why are these awesome videos so unknown?

How could someone hack my brain wave and convoluted this by evaluate inner voice?
May I know this procedure?
#Thanks

In the demonstration, you generated the q, k & v vectors randomly, but in reality, what will the actual source of those values be?

What is the name of the text book you are using?

Ajay, I don't think the point of capturing the context in terms of words 'after' has a significance in language modelling. In language modelling you are predicting only the next word. Yes, for a task like machine translation, yes. Thus I don't think Bi-directional RNNs have anything better to offer for language modelling than the regular (one-way) RNNs. . Let me know what you think

i still dont understand the difference between Q K V, can someone explain?

@1:41 here you have said that you want the context of the words that will be coming in the future but in masking part of the video you have said that it will be cheating know the context of the words that will be coming in the future

I felt the q, k, v parameter is not explained very well.. similar search analogy would be better to get a intuition of these parameter then explaining as what I can offer, what I actual offer

Bro it's 100% better than your ppt vides

Guru aarama? Nanu kooda Kannada ne!

You forgot to show the most important thing, how to train self-attention with backpropagation? You forgot about backward pass

Good job bro, JESUS IS COMING BACK VERY SOON; WATCH AND PREPARE

Seriously, are u from karnataka ? your accent is so different dude.

Lots of time on the mask, but none on training the attention matrix 🤔

Bro why can't you encounter an example while explaining