Transformer architecture

The 2017 neural network design that replaced recurrence with self-attention to enable massive parallelization and state-of-the-art sequence modeling.

Introduced by Google researchers in the paper Attention Is All You Need, the Transformer architecture abandoned traditional recurrent (RNN) and convolutional (CNN) layers in favor of a self-attention mechanism. This shift allows the model to weigh the importance of every word in a sentence simultaneously rather than processing them in a fixed order. By leveraging multi-head attention and positional encodings, Transformers handle long-range dependencies with high efficiency. This architecture serves as the foundational backbone for modern large language models (LLMs) like GPT-4 and Claude, scaling effectively across billions of parameters and massive datasets.

https://arxiv.org/abs/1706.03762