Transformer
The neural network architecture that powers every modern LLM — uses self-attention to process sequences in parallel, replacing the older RNN approach for language modeling.
The transformer architecture, introduced by the 2017 "Attention Is All You Need" paper, is the foundation of every modern LLM. GPT, Claude, Gemini, Llama, Qwen, Mistral, DeepSeek — all are transformer-based. The defining innovation is the [attention mechanism](/glossary/attention-mechanism), which lets the model relate every token in a sequence to every other token in parallel.
For agent builders, you rarely touch the transformer architecture directly. But understanding it explains key practical realities: why context is quadratic in cost, why KV caching matters, why long-context models exist, why MoE models are economically viable.
In 2026 the standard transformer has been extended with mixture-of-experts (MoE), state-space hybrids (Mamba-style architectures), and various efficiency optimizations. The base concept — attention-driven sequence modeling — remains the foundation.
Frequently asked
Are all LLMs transformer-based?+
Effectively yes in 2026. Mamba and other state-space alternatives exist but have not displaced transformers for general-purpose language modeling. Most "non-transformer" claims are transformer-hybrid architectures.
Why is the transformer expensive?+
Attention is quadratic in sequence length — for N tokens, you have N² attention computations. This is why long-context inference is expensive and why optimizations like flash attention, sparse attention, and KV caching matter so much.