Tap to unmute

Transformer Neural Networks - EXPLAINED! (Attention is all you need)

μ†ŒμŠ€ μ½”λ“œ
  • κ²Œμ‹œμΌ 2020. 01. 12.
  • Please subscribe to keep me alive: krplus.net/uCodeEmporiu...
    BLOG: medium.com/@dataemporium
    MATH COURSES (7 day free trial)
    πŸ“• Mathematics for Machine Learning: imp.i384100.net/MathML
    πŸ“• Calculus: imp.i384100.net/Calculus
    πŸ“• Statistics for Data Science: imp.i384100.net/AdvancedStati...
    πŸ“• Bayesian Statistics: imp.i384100.net/BayesianStati...
    πŸ“• Linear Algebra: imp.i384100.net/LinearAlgebra
    πŸ“• Probability: imp.i384100.net/Probability
    OTHER RELATED COURSES (7 day free trial)
    πŸ“• ⭐ Deep Learning Specialization: imp.i384100.net/Deep-Learning
    πŸ“• Python for Everybody: imp.i384100.net/python
    πŸ“• MLOps Course: imp.i384100.net/MLOps
    πŸ“• Natural Language Processing (NLP): imp.i384100.net/NLP
    πŸ“• Machine Learning in Production: imp.i384100.net/MLProduction
    πŸ“• Data Science Specialization: imp.i384100.net/DataScience
    πŸ“• Tensorflow: imp.i384100.net/Tensorflow
    [1] The main Paper: arxiv.org/abs/1706.03762
    [2] Tensor2Tensor has some code with a tutorial: www.tensorflow.org/tutorials/...
    [3] Transformer very intuitively explained - Amazing: jalammar.github.io/illustrated...
    [4] Medium Blog on intuitive explanation: medium.com/inside-machine-lea...
    [5] Pretrained word embeddings: nlp.stanford.edu/projects/glove/
    [6] Intuitive explanation of Layer normalization: mlexplained.com/2018/11/30/an...
    [7] Paper that gives even better results than transformers (Pervasive Attention): arxiv.org/abs/1808.03867
    [8] BERT uses transformers to pretrain neural nets for common NLP tasks. : ai.googleblog.com/2018/11/ope...
    [9] Stanford Lecture on RNN: cs231n.stanford.edu/slides/201...
    [10] Colah’s Blog: colah.github.io/posts/2015-08...
    [11] Wiki for timeseries of events: en.wikipedia.org/wiki/Transfo...)

λŒ“κΈ€ • 616

  • @CodeEmporium

    For more details and code on building a translator using a transformer neural network, check out my playlist "Transformers from scratch":

  • @ThisNameWasntTaken
    @ThisNameWasntTaken 3 λ…„ μ „ +698

    what a hugely underrated video. You did such a better job at explaining this on multiple abstraction layers in such a short video than most videos I could find on the topic which were more than twice as long.

  • @PhilbertLin
    @PhilbertLin 3 λ…„ μ „ +205

    Great video! Watched it a few times already so these timestamps will help me out:

  • @Elanus19
    @Elanus19 2 λ…„ μ „ +72

    Incredibly well explained and concise. I can't believe you pulled off such a complete explanation in just 13 minutes!

  • @danbochman
    @danbochman 2 λ…„ μ „ +21


  • @gearoidmurphy4988
    @gearoidmurphy4988 2 λ…„ μ „ +77

    The multi-pass approach to progressively explaining the internals worked well. Thanks for your content!

  • @lingding77
    @lingding77 2 λ…„ μ „ +16

    I love the multi-pass way of explanation so that the viewer can process high level concepts and then build upon that knowledge, great job.

  • @jonathanburrell7055
    @jonathanburrell7055 12 μ‹œκ°„ μ „ +1

    This is awesome!!! Thank you for breaking it down concisely, understandably, and deeply! It’s hard to find explanations that aren’t so simplistic they’re useless, or so involved they don’t save time in achieving understanding. Thank you!!

  • @MrKfirlevi
    @MrKfirlevi 2 λ…„ μ „ +16

    Great video!! I am taking a course in my university and one of the lectures was about RNNs and transformers. Your video of 13 mins explains way better than the 100 mins lecture i attended. Thank you!

  • @Sidnv
    @Sidnv λ…„ μ „ +2

    Really great video. As someone transitioning from pure math into machine learning and AI, I find the language barrier to be the biggest hurdle and you broke down these concepts in a really clear way. I love the multiple layer approach you took to this video, I think it worked really well to first give a big picture overview of the architecture before delving deeper.

  • @mattcoakes5682
    @mattcoakes5682 2 λ…„ μ „ +3

    LOVE the multipass strategy for explaining the architecture. I don't think I've seen this approach used with ML, and it's a shame as this is an incredibly useful strategy for people like me trying to play catch up. I hopped on the ML train a little late, but stuff like this makes me feel not nearly as lost.

  • @paragrk1
    @paragrk1 2 λ…„ μ „ +2

    Went through several videos on 'Attention is all you need' paper before this, all the details you managed to cover in thirteen minutes is amazing. Could not find explanation that is so easy to understand anywhere else. Great job!

  • @motherbear55
    @motherbear55 2 λ…„ μ „ +1

    Great explanation. Could you do another video on positional encoding specifically? It seems to be very important, but I’ve found it the most confusing part of this architecture.

  • @gabrielcournelle3055
    @gabrielcournelle3055 2 λ…„ μ „ +3

    That was an awesome explanation. I have a question about the Add & Norm block. Do you add the embedded vector before or after performing normalization ? Is there even a difference if we do one instead of the other ?

  • @derrxb
    @derrxb 3 λ…„ μ „ +4

    This is one of the best explanations for transformers I've come across online! Awesome job, man! Thanks. I'll totally recommend your channel to some classmates!! :)

  • @codemeai
    @codemeai 2 λ…„ μ „ +2

    Very Underrated. Please keep doing these videos. You have no idea what a great amount of service this is doing to the young research communities who are just learning to read research papers. Instant subscribed.πŸ™Œ

  • @frederikbrammer8006

    This is by far the best explanation of the Transformers architecture that I have ever seen! Thanks a lot!

  • @logicloudy2851
    @logicloudy2851 2 λ…„ μ „ +2

    Thanks, man. This is a really clear and high-level explanation. Really helpful for some guys like me who just stepped into this area.

  • @ryanhewitt9902

    Thank you for making this! As a curious outsider I have been anxious about falling behind in recent years and this was perfect to bring me up to speed - at least enough to follow the conversation.

  • @joshuascholar3220
    @joshuascholar3220 2 λ…„ μ „

    I've worked on none of these things, but listening to this, I think of mechanisms that might improve them. I think real brains do a bit of combinatorial optimization, in understanding ambiguity is resolved by simply trying the alternatives and seeing which works out better, but it's not always in parallel. And in writing there's a bit of trying, failing, trying something different etc. In a machine vocabulary you might think of a simulated annealing algorithm. So when I see multiple encodings being combined with weightings I immediately think "this is not optimal" - it's better to go through combinatorial optimization of encodings for different words to find better and worse fits than to try to resolve multiple encodings without trying in the context of the others.