Positional Encoding in Transformer Neural Networks Explained

Amazing videos, but dude please remove your face from the thumbnails. It adds zero value and is distracting from choosing the content. Don't follow a herd, better represent something unique in there.

tks!!

i think you desrve it, big thank you to you

:)

shut up @@mello1016 adding his facing makes it less abstract. You can see it is a human behind, and it makes it easier to focus.

Thanks for the super kind words! Definitely more to come. In the middle of making a series on Reinforcement Learning now :)

You are so welcome !

Thanks so much for watching and commenting on them :)

Thanks so much for watching ! Hope you enjoy the rest :)

My pleasure :) Thank you for the support

Thanks so much for watching :)

Haha thanks a lot :) I try

Perfect! Super glad you’re on this journey. The field is very fun :)

Thanks So much

Thank you so much! :D

Nah you are awesome

My pleasure

Thanks so much! Hmm. Maybe hugging face has some good resources too. Aside from this, I’ll be making a playlist on the evolution of language models so some design choices become more intuitive. Hope you’ll stick around for that

Super welcome

Your words are too kind. And good question. So what is fixed in length in this specific architecture is the maximum number of words in a sentence, not the number of words in a sentence. The remain unused words are filled with “padding tokens”. This will be come clearer when you watch the videos of coding out the complete transformer in the playlist “Transformers from Scratch”. We essentially do this so we can pass fixed size vector inputs through every part of the transformer. That said, I have seen more recent implementations where the size is dynamic

Thank you! And yep I have a full time job as a Machine Learning Engineer outside of KRplus :)

Yes! The idea overall is to create meaningful embedding for words that understand context. This is opposed to the tradition CBoW or Skip gram word embeddings that don’t quite get this context.

They way you described sequence length = 50 is correct. It is the maximum number of tokens you can pass into your network at a time (it’s the max number of words/ subwords/characters). D_model is the embedding dimension. Models don’t understand words, but they understand numbers. And so, you transform every token into some set of numbers (called a vector) and the number of numbers in this vector is d_model. Let’s say d_model is 512 and also say we have a sentence “my name is ajay”. The word “my” would be converted into a 512 dimensional vector. As would “name”, “is” and “ajay”. The idea of these vectors/embedding is to get some dense numeric representation of the context of a word (so similar words are represented with vectors that are close to each other and dissimilar words are represented with vectors that are farther from each other)

@@CodeEmporium hm ok good answer, now a doubt if d_model is the dimension that I will put my tokens. Why don't some transformer models accept very long texts? for example: if I have the
string length = 10
d_model = 3
the phrase "my name is Ajay"
would turn 4 vectors
my: [0,0.2,0.6]
name: [0, 0.1, 0.11]
is: [0.5, 0.2, 0.0]
Ajay: [0,0,1] with d_model dimensions each
Why can't I put very large sequences in my model? Why does d_model interfere with this

@@LuizHenrique-qr3lt The max sequence length refers to the maximum number of tokens in a sequence, while d_model represents the dimensionality of the token embeddings. They serve different purposes in the Transformer model. The max sequence length determines the size of the input that can be processed at once, whereas d_model influences the complexity and expressive power of the model. In your example, if the max sequence length is 50 and d_model is 10, the largest sequence would need to be divided into smaller segments or chunks of 10 tokens to fit within the model's input limit.

I have made some math in machine learning videos. Maybe check the playlist “Th e math you should know” on the channel

512 is a hyperparameter. You can actually decide which dimension to use, but it has been proven that higher dimension usually work better, since they are able to capture more linguistic information, e.g. semantics, syntax, etc. BERT for instance uses 768 dimensions and the OpenAI ada embeddings have 1536 dimensions.

Thank you for the compliments

The idea here is we are not going to use a preexisting word2vec for the transformer. Everything in clouding the embedding for every word will be learned during training. An issue with word2vec is they are fixed embeddings and don’t necessarily capture word context very well. This concept was introduced in the paper that introduced ELMo “Deep Contextualized word presentations” (Peter et al., 2018). Would recommend giving this a read if you’re interest

This is a good question. But it intuitively does make sense that the same word In different sentences can have different meanings. Take the word “grounded”. You can represent this as a 512 dimensional vector. But let’s say “grounded” occurs in 2 sentences: (1) The truth is grounded in reality (2) You’re grounded! Go to your room. In these examples, “grounded” has differing meanings and should hence have different vector representations. This is why we need surrounding context to understand word vectors individually. This is probably a lil hard to see with your example since “Ajay” is a proper noun. However, for non-proper nouns, context matters.
I think you should take a look at the paper “Deep Contextualized word Representations” by Matthew Peters (2018). They more formally answer the question you are asking. This is the paper that introduced ELMo embedding. According to this paper, Turns out that using different vectors based on context really improved models on Part of Speech Tagging and Language modeling

You've raised an important point. While it is true that the positional encoding for a word like "Ajay" can vary depending on its position in different sentences.
Let's consider the word "Ajay" in two different sentences and see how the Transformer model handles it:
(1) Sentence 1: "My name is Ajay."
(2) Sentence 2: "Ajay explains very well."
In both sentences, the word "Ajay" has different positions, but the Transformer model can still learn and make sense of it. Here's a simplified example of how it works:
Input Encoding: Each word, including "Ajay," is initially represented by a 512-dimensional vector.
Sentence 1: "Ajay" is represented as [0.1, 0.2, 0.3, ..., 0.4].
Sentence 2: "Ajay" is represented as [0.5, 0.6, 0.7, ..., 0.8].
Positional Encoding: The model incorporates positional encodings to differentiate the positions of words.
Sentence 1: The positional encoding for the 4th position is [0.4, 0.3, 0.2, ..., 0.1].
Sentence 2: The positional encoding for the 1st position is [1.0, 0.9, 0.8, ..., 0.5].
Attention and Context: The Transformer's attention mechanism considers the positional encodings along with the input representations to compute contextualized representations.
Sentence 1: The attention mechanism incorporates the positional encoding and input embedding of "Ajay" at the 4th position to capture its contextual information within the sentence.
Sentence 2: Similarly, the attention mechanism considers the positional encoding and input embedding of "Ajay" at the 1st position in the context of the second sentence.
By attending to different positions and incorporating positional encodings, the model can learn to associate the word "Ajay" with its specific context and meaning in each sentence. Through training on various examples, the model adjusts its weights and learns to generate appropriate representations for words based on their positions, allowing it to make meaningful predictions and capture the contextual relationships between words effectively.

I think I responded to this in another video you asked this question. Hope that helped tho :)

@@CodeEmporium Yeah but you didn't answer it fully

So I haven’t shown the text encoding details just yet. :) since 4 words were encoded into 7 numbers, I assume the “BestTokenizer” is encoding each subword / word piece into some number. Essentially, the tokenizer is taking the sentence, breaking it down into word pieces (7 in this case) and each is being mapped to a unique integer number. Later on, you will see each number being mapped to a larger vector (I explained more details about why these vectors exist in the other comment)

Sequence length = maximum number of characters/words we can pass into the transformer at a time.
Dimension of embedding = size of vector representing each character / word.

@@CodeEmporium thanks a lot.

Why are we using 512 dimensions instead of the 1 hot vector of size equal to the vocabulary size? This is because of the curse of dimensionality. Vocabulary sizes are huge (often in the 10s of thousands). This is a lot for any model, neural network or not, to process. There was a 2001 paper by Yashua Bengio “A Neural Probabilistic Language Model” that describes exactly this issue and why it was introduced. I would recommend giving it a read. Also, my next series will delve into the history of language models so I hope you’ll stay tuned for this. Maybe some of the design choices will become clearer.

Yea that’s correct from a practical standpoint. I dive into this when coding this out in the rest of this playlist “Transformers from scratch “. Hope those videos clear things up!

@@CodeEmporium Thanks for answering. I understand that you have to make a trade-off between simplicity and accuracy. Here, I just wanted to note that little more complexity would have added quite a lot more accuracy.
Your content is excellent!