Qwen3-Next 80B-A3B — The Evolving Transformer

Qwen3-Next 80B-A3B · 3:1 DeltaNet:GatedAttn · MoE+shared · 48 layers · 256K

Qwen3-Next-80B-A3B is Qwen’s efficiency-first branch after regular Qwen3 MoE: same 3B active compute per token as Qwen3-30B-A3B, but 80B total capacity (~3.75% active ratio) and a new hybrid attention stack. Qwen’s blog claims base quality comparable to Qwen3-32B at under 10% training cost and 10×+ throughput past 32K context.

The teaching code in qwen3.py implements the core mechanisms at reduced scale: gated delta rule scan, sigmoid-gated GQA with QK-norm and partial RoPE on attention layers only, shared-expert MoE, and multi-token prediction heads.

Delta from PaliGemma (stop 08)

Multimodal prefix LM → text-only hybrid Gated DeltaNet + Gated Attention
Dense/SigLIP+Gemma stack → high-sparsity MoE with shared expert (~3B active)
Single next-token head → multi-token prediction auxiliary heads
Generative VLM transfer → 256K long-context-efficient text LM design

00 · CONFIG

Qwen3-Next-80B-A3B hyperparameters

Production values from the Hugging Face model card and Qwen3-Next blog. Teaching code scales these down for local smoke tests.

dmodel5,120 — hidden size

N48 — decoder layers (3 DeltaNet : 1 GatedAttn)

H / n_kv_heads40 query heads · 8 KV heads (GQA on attention layers)

Total / active~80B total · ~3B activated per token

MoEHigh-sparsity routed experts + shared expert

vocabsize152,064 tokens

max_seq_len262,144 (256K) — YaRN extension toward ~1M

Attention75% Gated DeltaNet · 25% Gated Attention · partial RoPE

MTPMulti-token prediction auxiliary heads

VariantsBase · Instruct · Thinking (Apache 2.0)

Qwen3-30B-A3B vs Qwen3-Next-80B-A3B

Feature	Qwen3-30B-A3B	Qwen3-Next-80B-A3B
Total params	30B	80B
Active params / token	~3B	~3B
Active ratio	~10%	~3.75%
Attention	Standard Transformer GQA	Gated DeltaNet + Gated Attention (3:1)
Context design	Qwen3 standard	Ultra-long-context efficient (256K)
Extra training obj.	—	Multi-token prediction (MTP)
Main upgrade	Sparse Qwen3 MoE	More capacity + hybrid attention

01 · GatedDeltaNet(config)

Linear attention with gated delta rule

Delta vs standard attention + MoE

Full self-attention → Gated DeltaNet (linear attention) for 75% of layers
Standard attention → Gated Attention (sigmoid gate + QK-Norm) for 25%
3:1 ratio: 3 linear layers, then 1 full attention layer
MoE without shared expert → MoE with shared expert (returns from Qwen3)
Single-token prediction → Multi-Token Prediction head
Full RoPE → Partial RoPE

Instead of computing the full S×S attention matrix, DeltaNet maintains a running state that accumulates information. The delta rule updates this state: it first “erases” information along the current key, then writes new information. A 1D convolution on values provides local context, and a gate controls how much of the recurrent state flows to the output.

Below, every line is new relative to a standard softmax self-attention block (marked in the code panel).

qwen3.pyGatedDeltaNet


 1class GatedDeltaNet(nn.Module):
 2    def __init__(self, config):
 3        super().__init__()
 4        self.wq = nn.Linear(config.dmodel,config.dmodel,bias=False)
 5        self.wk = nn.Linear(config.dmodel,config.dmodel,bias=False)
 6        self.wv = nn.Linear(config.dmodel,config.dmodel,bias=False)
 7        self.wb = nn.Linear(config.dmodel,1,bias=True)  # step-size beta
 8        self.wa = nn.Linear(config.dmodel,config.dmodel,bias=True)  # per-channel forget alpha
 9        self.wo = nn.Linear(config.dmodel,config.dmodel,bias=False)
10
11    def forward(self, x):  # sequential scan is causal by construction
12        B,S,D = x.shape
13        Q=self.wq(x); K=F.normalize(self.wk(x),dim=-1); V=self.wv(x)
14        beta=torch.sigmoid(self.wb(x)); alpha=torch.sigmoid(self.wa(x))
15        W=torch.zeros(B,D,D,device=x.device); outs=[]
16        for t in range(S):
17            qt,kt,vt,bt,at = Q[:,t],K[:,t],V[:,t],beta[:,t],alpha[:,t]
18            Wkt=(W@kt.unsqueeze(-1)).squeeze(-1)
19            W=at.unsqueeze(-1)*W +bt.unsqueeze(-1)*(vt-Wkt).unsqueeze(-1)*kt.unsqueeze(-2)
20            outs.append((W@qt.unsqueeze(-1)).squeeze(-1))
21        return self.wo(torch.stack(outs,1))

02 · GatedAttention(config)

Full attention with QK-Norm and output gate

The one-in-four full layers use standard softmax attention with two added controls: a sigmoid gate on the attention output, and QK-Norm — per-head RMSNorm on Q and K before the dot product. Partial RoPE (see factory) is applied by rope_fn outside this module.

qwen3.pyGatedAttention (QK-Norm + sigmoid)

  1class GatedAttention(nn.Module):
  2    """Full attention with sigmoid output gate and QK-Norm."""
  3    def __init__(self, d_model, h, h_kv):
  4        super().__init__()
  5        self.h = h
  6        self.h_kv = h_kv
  7        self.d_k = d_model // h
  8        self.wq = nn.Linear(d_model, h * self.d_k, bias=False)
  9        self.wk = nn.Linear(d_model, h_kv * self.d_k, bias=False)
 10        self.wv = nn.Linear(d_model, h_kv * self.d_k, bias=False)
 11        self.wo = nn.Linear(h * self.d_k, d_model, bias=False)
 12        # QK-Norm: normalize Q and K before dot product
 13        self.q_norm = RMSNorm(self.d_k)
 14        self.k_norm = RMSNorm(self.d_k)
 15        # Output gate
 16        self.gate = nn.Linear(d_model, h * self.d_k)
 17    
 18    def forward(self, x, rope_fn, mask=None):
 19        B, S, _ = x.shape
 20        q = self.wq(x).view(B, S, self.h, self.d_k).transpose(1, 2)
 21        k = self.wk(x).view(B, S, self.h_kv, self.d_k).transpose(1, 2)
 22        v = self.wv(x).view(B, S, self.h_kv, self.d_k).transpose(1, 2)
 23        
 24        # QK-Norm
 25        q = self.q_norm(q)
 26        k = self.k_norm(k)
 27        
 28        # Partial RoPE (applied externally)
 29        q, k = rope_fn(q, k)
 30        
 31        # GQA repeat
 32        rep = self.h // self.h_kv
 33        k = k.repeat_interleave(rep, dim=1)
 34        v = v.repeat_interleave(rep, dim=1)
 35        
 36        scores = (q @ k.transpose(-2, -1)) / math.sqrt(self.d_k)
 37        if mask is not None:
 38            scores = scores.masked_fill(mask == 0, float('-inf'))
 39        attn = torch.softmax(scores, dim=-1)
 40        out = (attn @ v).transpose(1, 2).contiguous().view(B, S, -1)
 41        
 42        # Sigmoid output gate
 43        g = torch.sigmoid(self.gate(x))
 44        return self.wo(g * out)

03 · MultiTokenPrediction(config)

Heads for multiple future steps

Instead of a single next-token head only, the model can train auxiliary heads to predict several future tokens in parallel, improving sample efficiency. Each head is a simple linear map from the last hidden state to the vocabulary (implementation detail may tie weights or share a trunk; here we use independent Linear layers).

qwen3.pyMultiTokenPrediction

  1class MultiTokenPrediction(nn.Module):
  2    """Predict K future tokens simultaneously."""
  3    def __init__(self, d_model, vocab_size, num_predict=4):
  4        super().__init__()
  5        self.heads = nn.ModuleList([
  6            nn.Linear(d_model, vocab_size, bias=False)
  7            for _ in range(num_predict)
  8        ])
  9    
 10    def forward(self, x):
 11        # Returns list of logits, one per future token
 12        return [head(x) for head in self.heads]

04 · MoE — shared + routed

Routed experts with an always-on path

Typical top-k MoE sparsely activates a few feed-forward “experts” per token. Qwen3-Next also keeps a shared expert (or MLP) that every token passes through, then adds the routed result — a pattern reintroduced from earlier Qwen3 designs to stabilize training and add capacity on every position.

qwen3.pyMoE (shared expert)

  1class MoEWithSharedExpert(nn.Module):
  2    """Routed top-k SwiGLU experts plus one shared FFN for every position."""
  3    def __init__(self, d_model, d_ff, num_experts, top_k):
  4        super().__init__()
  5        self.top_k = top_k
  6        self.router = nn.Linear(d_model, num_experts, bias=False)
  7        self.experts = nn.ModuleList([
  8            SwiGLUFeedForward(d_model, d_ff, dropout=0.0) for _ in range(num_experts)
  9        ])
 10        self.shared = SwiGLUFeedForward(d_model, d_ff, dropout=0.0)
 11    
 12    def forward(self, x):
 13        w = torch.softmax(self.router(x), dim=-1)  # (B, S, E)
 14        w_top, e_idx = torch.topk(w, self.top_k, dim=-1)  # (B, S, K)
 15        w_top = w_top / w_top.sum(dim=-1, keepdim=True)
 16        B, S, d_m = x.shape
 17        stacked = torch.stack([expert(x) for expert in self.experts], dim=2)  # (B, S, E, d_m)
 18        routed = x.new_zeros(B, S, d_m)
 19        for k in range(self.top_k):
 20            idx = e_idx[:, :, k].long().view(B, S, 1, 1).expand(B, S, 1, d_m)
 21            y_k = torch.gather(stacked, 2, idx).squeeze(2)
 22            routed = routed + w_top[:, :, k, None] * y_k
 23        return self.shared(x) + routed

05 · Block(config, layer_idx)

3:1 — DeltaNet vs GatedAttention by layer index

Every fourth block (layer_idx % 4 == 3) is a full GatedAttention layer; the other three use GatedDeltaNet. The FFN is always the MoE-with-shared path.

qwen3.pyBlock(config, layer_idx)

  1class HybridDecoderBlock(nn.Module):
  2    def __init__(self, d_model, h, h_kv, d_ff, layer_idx, num_experts=64, top_k=8):
  3        super().__init__()
  4        use_full_attn = (layer_idx % 4 == 3)  # every 4th layer
  5        if use_full_attn:
  6            self.mixer = GatedAttention(d_model, h, h_kv)
  7        else:
  8            self.mixer = GatedDeltaNet(d_model)
  9        self.ffn = MoEWithSharedExpert(d_model, d_ff, num_experts, top_k)
 10        self.norm1 = RMSNorm(d_model)
 11        self.norm2 = RMSNorm(d_model)
 12        self.use_full_attn = use_full_attn
 13    
 14    def forward(self, x, rope_fn=None, mask=None):
 15        if self.use_full_attn:
 16            x = x + self.mixer(self.norm1(x), rope_fn, mask)
 17        else:
 18            x = x + self.mixer(self.norm1(x))
 19        x = x + self.ffn(self.norm2(x))
 20        return x

06 · Qwen3(config)

Factory: embeddings, partial RoPE, blocks, LM head, MTP

Construction mirrors other decoder-only LMs, but wires in MultiTokenPrediction and RotaryPositionalEmbedding with a partial_ratio for Qwen3-Next’s partial RoPE (illustrative; match your checkpoint config).

qwen3.pyQwen3

  1def build_qwen3_next(vocab_size, max_seq_len=131072, d_model=5120, N=64, h=40, h_kv=8, d_ff=13824, num_experts=64, top_k=8, mtp_tokens=4):
  2    tok_emb = nn.Embedding(vocab_size, d_model)
  3    rope = RotaryPositionalEmbedding(d_model // h, max_seq_len, partial_ratio=0.5)
  4    
  5    blocks = nn.ModuleList([
  6        HybridDecoderBlock(d_model, h, h_kv, d_ff, i, num_experts, top_k)
  7        for i in range(N)
  8    ])
  9    
 10    norm = RMSNorm(d_model)
 11    head = nn.Linear(d_model, vocab_size, bias=False)
 12    mtp = MultiTokenPrediction(d_model, vocab_size, mtp_tokens)
 13    return tok_emb, rope, blocks, norm, head, mtp

SUMMARY

Standard attention + MoE vs Qwen3-Next

High-level map of the architectural shifts — hybrid linear/full attention, shared expert, MTP, and partial RoPE.

      
        ComponentStandard attention + MoEQwen3-Next

        Self-attentionFull softmax, every layer, O(S²)75% Gated DeltaNet O(S) + 25% Gated Attention
Layer patternAll identical blocks3:1 (linear : full) every 4 layers
Full-attn extras—QK-Norm, sigmoid output gate, partial RoPE
FFN / MoERouted experts onlyRouted experts + shared expert
Output headsSingle LM head (next token)Primary LM head + multi-token prediction heads
Position (RoPE)Full RoPE on all head dims (typical)Partial RoPE (e.g. half the dims; illustrative)

Component	Standard attention + MoE	Qwen3-Next
Self-attention	Full softmax, every layer, O(S²)	75% Gated DeltaNet O(S) + 25% Gated Attention
Layer pattern	All identical blocks	3:1 (linear : full) every 4 layers
Full-attn extras	—	QK-Norm, sigmoid output gate, partial RoPE
FFN / MoE	Routed experts only	Routed experts + shared expert
Output heads	Single LM head (next token)	Primary LM head + multi-token prediction heads
Position (RoPE)	Full RoPE on all head dims (typical)	Partial RoPE (e.g. half the dims; illustrative)

09 · REFERENCES

Papers & technical sources

Primary reports and references for this chapter. Read these for full equations, training details, and official hyperparameters.

Qwen3-Next: Towards Ultimate Training & Inference EfficiencyOfficial blog — hybrid attention, high-sparsity MoE, MTP, Base/Instruct/Thinking.
Qwen3 Technical ReportQwen3 dense and MoE family (30B-A3B, 235B-A22B).
Qwen3-Next-80B-A3B-Instruct (Hugging Face)Model card and config for the public checkpoint.
Qwen3-Next-80B-A3B-Thinking (Hugging Face)Reasoning-focused variant; GSPO post-training notes.