The Evolving Transformer

This is the series where I build landmark transformer architectures from scratch in PyTorch — one codebase that grows paper by paper. Not slides or pseudocode: runnable code, with the diffs that actually matter between each generation.

Full source for each chapter is on GitHub — abhibisht89/llamasearch-blogs, folder the-evolving-transformer/.

The curated arc: Transformer, GPT-2, CLIP, LLaMA, Mistral, Mixtral, DeepSeek V2/V3, PaliGemma, and Qwen3-Next — 9 stops where the architecture moved in a fundamental way, plus a training capstone on how any of them learn from text.

I hope to release a matching YouTube series — one video per chapter — so you can follow the intuition alongside the code.

Chapters will drop here sequentially as they are ready. Only finished ones are linked below; the timeline is the full roadmap of models we plan to cover.

GPT-2

CLIP

LLaMA

Mistral

Mixtral

DeepSeek

PaliGemma

9Chapters

70+Components

55+Deltas

Original Transformer architecture diagram

STOP 01 — THE BASELINE

Attention Is All You Need

2017 · Vaswani et al.

The encoder-decoder Transformer that started everything. Sinusoidal PE, multi-head attention, LayerNorm, ReLU FFN — all built from scratch in PyTorch.

Encoder + Decoder Sinusoidal PE Multi-Head Attention LayerNorm ReLU FFN

GPT-2 Small architecture

STOP 02 — 4 DELTAS

GPT-2

2019 · Radford et al. (OpenAI)

Decoder-only. Learned positional embedding replaces sinusoidal. Pre-norm (LayerNorm before sub-blocks). GELU replaces ReLU.

Decoder-only Learned PE Pre-norm GELU

CLIP

STOP 03 — MULTIMODAL

CLIP

2021 · Radford et al. (OpenAI)

Dual encoders map images and text into one contrastive space — patch ViT + causal text tower, L2-normalised dot products, symmetric InfoNCE.

ViT patches Causal text tower B×B contrastive Zero-shot labels

LLaMA architecture diagram

STOP 04 — 6 DELTAS

LLaMA

2023 · Touvron et al. (Meta)

RoPE replaces learned PE. RMSNorm replaces LayerNorm. SwiGLU replaces GELU. Grouped Query Attention. No bias, no embedding scaling.

RoPE RMSNorm SwiGLU GQA No bias

Mistral 7B architecture

STOP 05 — 2 DELTAS

Mistral 7B

2023 · Mistral AI

Sliding window attention (W=4096) replaces full causal. Rolling buffer KV cache bounds memory. Otherwise identical to LLaMA.

Sliding window W=4096 Rolling buffer cache

Mixtral 8×7B

STOP 06 — SPARSE MOE

Mixtral 8×7B

2023 · Mistral AI

First sparse MoE at scale. Replace each dense FFN with 8 expert FFNs and a router that picks top-2 per token. ~46.7B stored, ~12.9B active.

MoE (8 experts, top-2) Router gating

DeepSeek V2 and V3

STOP 07 — MLA + DEEPSEEKMOE

DeepSeek V2 & V3

2024 · DeepSeek

V2 introduces MLA and DeepSeekMoE (236B, 21B active). V3 scales to 671B with auxiliary-loss-free routing, MTP, and FP8 training.

MLA (V2) DeepSeekMoE Aux-loss-free (V3) MTP + FP8 (V3)

PaliGemma

STOP 08 — OPEN VLM

PaliGemma

2024 · Google DeepMind

SigLIP vision encoder + LinearProjector + Gemma decoder. Image patch tokens prepended as prefix; generative VLM via next-token prediction.

All patches kept LinearProjector Visual prefix LM Weight tying

Qwen3-Next architecture

STOP 09 — SERIES FINALE

Qwen3-Next 80B-A3B

2025 · Alibaba

80B total / 3B active. Hybrid 3:1 Gated DeltaNet + Gated Attention. High-sparsity MoE + shared expert. MTP. 256K context.

Gated DeltaNet 3:1 Multi-token prediction Partial RoPE Hybrid attention

CAPSTONE — TRAINING

The Training Loop

Generic script · TinyShakespeare

One model-agnostic training loop: dataset, next-token loss, AdamW, validation, generation. GPT-2 ships as the runnable example — swap build_model() for any decoder from the series.

dataset.py train.py Cross-entropy Val + generate