Introduction

The “Attention Is All You Need” paper introduced the Transformer architecture, which has become the foundation for modern NLP models like BERT and GPT. In this note, we’ll explore the key concepts behind the self-attention mechanism.

Key Concepts

1. Self-Attention

Self-attention allows the model to weigh the importance of different words in a sentence when encoding a particular word.

“The animal didn’t cross the street because it was too tired.”

When the model processes the word “it”, self-attention allows it to associate “it” with “animal”.

2. Multi-Head Attention

Instead of performing a single attention function, the authors found it beneficial to linearly project the queries, keys, and values $h$ times with different, learned linear projections.

$$ \text{Attention}(Q, K, V) = \text{softmax}(\frac{QK^T}{\sqrt{d_k}})V $$

Conclusion

The Transformer architecture demonstrates that recurrence and convolutions are not essential for building high-performance NLP models. Attention mechanisms alone are sufficient.