Attention

The core mechanism that dynamically assigns importance weights (Query, Key, Value) to input elements, enabling the parallel computation and superior long-range context modeling of the Transformer architecture.

Attention is the fundamental innovation powering modern generative AI, replacing the sequential processing of older Recurrent Neural Networks (RNNs). The concept, initially introduced by Bahdanau et al. in 2014, was fully realized in the 2017 paper, 'Attention Is All You Need,' which established the Transformer model. This mechanism operates by computing 'soft' attention weights: a Query vector seeks information, which is scored against all Key vectors, and the resulting scores are applied to the Value vectors. This process allows every element in a sequence to directly reference every other element, regardless of distance. The Self-Attention variant, often implemented as Multi-Head Attention (e.g., 8 parallel heads), is the key to the Transformer's ability to handle massive datasets and form the backbone of Large Language Models (LLMs) like GPT and BERT.

https://arxiv.org/abs/1706.03762