Transformer

The Transformer is a neural network architecture that uses a multi-head self-attention mechanism to process sequences in parallel, replacing slower recurrent (RNN) and convolutional (CNN) layers.

The Transformer architecture, introduced in the landmark 2017 paper 'Attention Is All You Need' by Vaswani et al. (Google), revolutionized sequence-to-sequence modeling. It operates entirely on an attention mechanism (multi-head self-attention), eliminating the need for sequential processing via Recurrent Neural Networks (RNNs). This design allows for massive parallelization, drastically reducing training time and enabling the scale-up of models to billions of parameters. It is the foundational technology for all modern Large Language Models (LLMs), including BERT and the Generative Pre-trained Transformer (GPT) series, driving state-of-the-art performance across Natural Language Processing (NLP) and computer vision tasks.

https://arxiv.org/abs/1706.03762