Architecture Deltas: 2017 → 2026

Vanilla self-attention has two cost problems. The compute is O(N²·d) per layer, which already hurts at training. But the killer at inference is the KV cache: for a model with L layers and H heads of dimension d, generating one token at context length N requires reading 2 · L · H · N · d floats every single step. A 70B model at 32k context with H=64, d=128 needs ~32 GB of cache per user. Every variant below is, fundamentally, a different way to compress that cache.

Reading the family tree

The first three variants — MHA, MQA, GQA — are the same algorithm with progressively fewer K/V heads. MHA gives every Q-head its own K/V (full expressivity, full cost). MQA goes to the extreme: one shared K/V across all H heads, ~H× smaller cache, but the model loses ~1% on most evals. GQA found the sweet spot: groups of (typically) 8 Q-heads share one K/V each, so cache shrinks ~8× but quality holds. Llama 2 70B shipped GQA at production scale; by 2024 it was the default for any dense LLM above ~30B.

MLA (DeepSeek V2/V3) is a different idea. Instead of cutting the number of K/V heads, it caches a low-rank latent (≈64 dims) and reconstructs full K/V on demand inside the attention op. The cache shrinks to a fraction of even GQA, with no quality drop. The trade-off: a small recompute cost per attention call. For long-context inference this is overwhelmingly worth it.

Sliding window attacks the problem from a different axis: don't cache fewer dimensions per token, just cache fewer tokens. Each query attends only to its last W positions (typically 128–4096), so the cache cost is bounded regardless of how long the context grows. Mistral 7B popularised W=4096; Gemma 3 and GPT-OSS went further to W=128 mixed with full attention every few layers (a "sliding hybrid"). The catch: pure sliding loses information from far away. Attention sinks (a tiny set of always-attended initial tokens) plug that hole — the model relies on those sink tokens to carry global state.

DSA (DeepSeek V3.2's "DeepSeek Sparse Attention") is the first production version of learned sparse attention. A tiny gating network decides, per query, which K positions are worth attending to. The cache stays sparse, but which positions are kept isn't fixed — it's content-dependent.

The most aggressive option is linear attention and its cousins (Gated DeltaNet, KDA). They rewrite softmax(QKᵀ)V using a kernel trick that turns attention into a recurrent update on a fixed-size state — so you cache nothing per token, just d² floats total. The cost: a measurable quality dip when used alone. The fix: hybrids. Qwen3-Next uses 3 linear layers per 1 full-attention layer; Kimi Linear mixes KDA with MLA; Nemotron 3 Nano mixes Mamba (a related state-space variant) with sparse full attention. Hybrids match or beat full transformers at 100k+ context.

The pattern: nobody has won. The 2026 frontier ships five attention variants in different layers of the same model. Long-context inference is now a stack of complementary cache- compression tricks, not one canonical "best attention".

A Mixture-of-Experts replaces one big FFN with N smaller FFNs and a router that selects K of them per token. The win is params ≠ compute: a 671B-parameter model can run at the speed of a 37B-parameter dense model, because only 37B params are actually touched per forward pass. The catch is that you pay training cost for the full 671B, and you have to keep all 671B in HBM at inference. The design space below is what teams actually argue about.

Three arguments still being settled

Argument 1 — Few large experts vs many small ones? Mixtral set the precedent in 2023: 8 large experts, top-2 active per token. The case for few-large is engineering simplicity — fewer all-to-all communication calls between GPUs, easier load balancing, fewer routing pathologies. Llama 4, Grok 2.5, and GPT-OSS all stayed in this basin. DeepSeek V3 took the opposite bet: 256 small experts, top-8 active. The case for many-small is that each expert can specialise in a much narrower input slice, and total parameter count grows much faster than active compute. DeepSeek V3, Qwen3-MoE, Kimi K2, and Mistral Large 3 all ship many-small.

Argument 2 — One shared "always-on" expert, or none? DeepSeek V3 introduced a shared expert that every token passes through, in addition to the 8 routed experts. The intuition: give the model an always-on baseline so the routed experts can each specialise rather than re-learning the average. Empirically it helps stability and final quality by ~0.3–0.5%. Then Qwen3-MoE ran the experiment without one — same parameter count, same active compute, just no shared expert — and matched DeepSeek V3 on most evals. The argument is unresolved. As of 2026, DeepSeek-lineage models (V3, V3.2, Kimi K2, GLM-4.5, Mistral Large 3) keep the shared expert; Qwen-lineage models (Qwen3, Qwen3-Next) drop it.

Argument 3 — Dense layers first, or all-MoE from layer 1? DeepSeek V3 keeps the first 3 layers dense (no MoE) before switching on experts in layers 4 and up. The argument: at the input, the distribution is uniform — there's nothing to specialise on yet — so MoE routing is wasted. Kimi K2 keeps 1 dense layer; most others go all-MoE from layer 0. The cost of the dense layer is small (one extra ~5B param layer for a 671B model is rounding error) but the stability benefit is real.

The pattern: "few-large" is the easier-to-train choice; "many-small + shared" is the higher-quality-per-active-param choice. The split roughly tracks "Western lineage vs Chinese lineage" in 2026, but that's mostly accidental — the technical case for many-small keeps winning.

Normalisation is the most under-appreciated lever in modern LLMs. It doesn't add expressive power; it doesn't reduce parameter count; it doesn't help inference. It only does one thing: it keeps training numerically stable as you stack layers. But that one thing is the difference between a 7B model that converges at 70 layers and a 7B model whose loss spikes to NaN halfway through the run.

The historical arc

The original transformer was post-norm: LayerNorm after the residual add. This is mathematically clean — every block's output is normalised — but it's brittle. Past ~24 layers the gradients into the bottom of the stack collapse, training stalls, and you get the famous "loss plateau then NaN" failure mode.

GPT-2 popularised pre-norm: move the LN before the sub-layer. Now the residual stream is no longer normalised, but the input to each attention/FFN block always is. Training is stable to 100+ layers. Every major dense LLM from 2019 to 2024 used this (with one twist: Llama 1 swapped LayerNorm for RMSNorm — drop the mean-centering, just scale by RMS — same quality, ~20% faster on GPUs).

Then two refinements showed up. Sandwich norm (Gemma 3) adds a second RMSNorm after the sub-layer, before the residual add. The motivation is sliding-window attention: when the window slides, the attention output's distribution can shift sharply, and that shift propagates into the residual stream. Normalising on both sides absorbs it. By 2025 sandwich norm is the "safer at scale" default.

OLMo 2 made the most controversial move: revive post-norm, but pair it with QK-Norm — a tiny LN applied separately to Q and K just before the dot product. The original "post-norm is unstable" failure was driven by attention logits exploding to ±∞; QK-Norm caps that explicitly. With QK-Norm in place, post-norm trains as stably as pre-norm and the residual stream stays cleaner.

The current consensus: pre-norm RMSNorm is the default; add sandwich at scale (esp. with sliding-window); add QK-Norm if you're seeing attention-logit blow-up. The post-norm revival is real but still niche.

Attention is permutation-invariant: shuffle the input tokens and the attention matrix is just shuffled the same way. Without a positional signal, the model sees a bag of words. Every variant below is a different answer to "where do you inject position?"

From "added to embeddings" to "implicit in the mask"

The original transformer used sinusoidal PE: hand-crafted sinusoids of varying frequencies, added to the input embeddings. The clever idea was that any position offset could be expressed as a linear function of the encoding, so the model could in principle generalise to longer sequences. In practice it didn't — quality dropped sharply past the training length.

GPT-2 swapped to learned absolute PE: N trainable position vectors added to embeddings. Simpler, slightly better at training length, completely incapable of extrapolating beyond N.

RoPE (LLaMA 1, originally RoFormer) is the breakthrough. Instead of adding position to the embedding, it rotates Q and K by a position-dependent angle inside the attention op. The dot product Q·K then naturally encodes relative position — the answer "how far apart are these two tokens?" emerges from the rotation difference. RoPE extrapolates much better than absolute PE and became universal by 2024.

Then the long-context arms race demanded longer windows than RoPE was trained for. YaRN (DeepSeek and others) is a smart re-scaling of RoPE's per-dimension frequencies: low-frequency dims (the ones encoding "near vs far") get squeezed, high-frequency dims (the ones encoding fine ordering) are preserved. A 4k-trained model can be retrofitted to 128k context with a few hundred fine-tuning steps.

NoPE (SmolLM3 in production, Stella-style) is the most surprising move: drop positional encoding entirely on some layers. The causal attention mask itself contains positional information ("this token can only see the ones before it"), and a deep model can reconstruct ordinal position from that signal alone. Empirically, mixing NoPE layers with RoPE layers gives slightly better long-context generalisation than RoPE-everywhere.

Partial RoPE (Gemma 4) is the latest twist: apply RoPE only to half the head dimensions; leave the other half un-rotated. The model now has both position-aware and position- agnostic features in parallel — useful for tasks where the answer should be invariant to position (factual recall) and tasks where it shouldn't (code, math).

The pattern: position info has migrated from embeddings (2017) → into attention (2023) → out of the model entirely on selected layers (2025). The trend is to inject less, not more.

The FFN is where most of a transformer's parameters live (~⅔ in a dense Llama-style block). It's also the part that has changed the least in spirit: every variant below is still "project up, do something nonlinear, project down". What changed is the activation function and, eventually, the identity of the FFN itself — MoE turns "one big FFN" into "N small ones plus a router".

Why each step happened

ReLU → GELU. ReLU's hard zero discards every negative input; GELU smooths the transition (it's roughly a Gaussian-weighted ReLU). The smoother gradient improves convergence by ~0.3%. GPT-2 shipped GELU; everyone followed.

GELU → SwiGLU. GLU (gated linear units) split the FFN's first projection into two parallel projections — a "gate" and an "up" — then element-wise multiply them after passing the gate through a sigmoid-like non-linearity. SwiGLU uses SiLU (= Swish) as the gate non-linearity. The cost is one extra weight matrix (3 instead of 2). The benefit is reliably ~1.5% better quality at the same parameter budget. LLaMA 1 made it the dense default; nobody at frontier scale uses anything else.

Dense FFN → MoE FFN. Replace one big SwiGLU with N smaller ones plus a router that picks K. You spend total parameters, but only K are active per token. The first big one was Mixtral 8×7B (8 experts, top-2). DeepSeek V3 went to 256+1 (top-8 + 1 shared). Now MoE is the default for any model above ~50B active. (See the MoE design space for the inside of the router.)

MoE → latent MoE. DeepSeek V3 (in some variants) and Qwen3-Next ship experts that work in a compressed latent space rather than full hidden dim. Each expert is smaller in parameter count but more numerous; total expressivity per active param is higher.

The pattern: activation choice has been settled (SwiGLU). Identity of the FFN is being unsettled — there's no longer one "FFN block" per layer; there's a router and a population.

For nine years, the transformer's quadratic attention cost was the field's main constraint at long context. In 2024–25, three alternatives reached production. None of them dethroned attention on its own — but all three earned a permanent slot inside hybrids.

Three escape routes

xLSTM is the most radical: drop attention entirely. The mLSTM (matrix LSTM) cell maintains a matrix-valued hidden state that is updated left-to-right per token. There's no attention matrix, no Q/K/V, no KV cache. Memory is genuinely constant in context length. xLSTM 7B (2025) was the first scaled model with zero attention layers. The catch: pure recurrent models underperform transformers by 1–3 quality points at the same parameter count.

Mamba and other state-space models (SSMs) are also recurrent, but they keep a structured hidden state and use a "selective scan" — the state-update parameters depend on the input, which lets the model decide what to remember and what to discard. Mamba alone underperforms transformers, but Mamba hybrids — a few attention layers sprinkled into a stack of Mamba layers — match transformer quality at much lower long-context cost. Nemotron 3 Nano (NVIDIA, 2025) was the first major production Mamba hybrid.

Linear attention and its modern descendants (Gated DeltaNet, Kimi's KDA) are the closest to "regular" attention. They keep the Q/K/V projection structure but replace softmax(QKᵀ)V with a kernel approximation φ(Q)(φ(K)ᵀV). The kernel form has an associative property that lets you compute attention recurrently with a fixed-size hidden state of d². Add gating (KDA's per-channel gates, Gated DeltaNet's per-head gates) and the model gets selective forgetting, which matters at long context.

The verdict the field reached in late 2025: hybrids dominate. A single attention type — full or sub-quadratic — alone leaves quality on the table. The frontier 2026 recipe puts 3–12 sub- quadratic layers (linear or SSM) per 1 full attention layer, and pays the quadratic cost only on the layers that need it. This gives you O(N) inference at long context with no quality loss.

The pattern: pure transformer is no longer the terminal architecture. The frontier is "transformer + selective recurrence". Whether you call that a transformer is a vocabulary question.