GLM-5.2

Tuning MTP speculative decoding to the limit

Sources: GLM-5.2: Built for Long-Horizon Tasks — Z.ai, 2026; IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse — Bai et al., 2026; Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling — Li et al., 2026

Section §18 ended with a zoo of draft strategies: separate small models, EAGLE, Medusa heads. The frontier open models went one step further and moved the draft model inside the model. Multi-Token PredictionMulti-Token PredictionMulti-Token Prediction (MTP) — a training objective where the model predicts several future tokens at each position (not just the next one), densifying the learning signal and enabling faster speculative decoding later.See in glossary → layers, trained during pre-training to predict several future tokens, double as a built-in draft model at serving time. GLM-5.2 is the clearest published example of what happens when a lab treats that MTP layer as a first-class serving component and tunes everything around it: draft cost, acceptance rate, and the loss it was trained with. This chapter walks through the full stack.

The draft model moved inside

An MTP layer is a small extra transformer block sitting on top of the trunk. During training it predicts token $t{+}2$ from the hidden state that predicted $t{+}1$, densifying the training signal. At serving time you run it speculatively: the MTP layer drafts a chain of future tokens (GLM-5.2 uses 7 draft steps, with parameters shared across steps), and the target model verifies the whole chain in one forward pass, exactly the §18 protocol. Compared with a separate draft model, the MTP layer shares the trunk’s embeddings and hidden states, so its drafts start out well-aligned with the target distribution: think Medusa, but trained into the model from the start.

GLM-5.2’s team states the two objectives plainly: (1) make the MTP layer as cheap as possible as a draft model, and (2) make its drafts get accepted as often as possible. Every optimization below serves one of the two.

IndexShare and KVShare: a cheaper, more honest draft

GLM-5.2’s attention is sparse (DSADeepSeek Sparse AttentionDeepSeek Sparse Attention (DSA): an attention variant where a lightweight indexer scores every past token with cheap dot products, a top-k selection keeps the most relevant ones, and full attention runs over only that subset.See in glossary →): a lightweight indexer scores past tokens and full attention runs over the top-k. IndexShareIndexShareSharing one sparse-attention indexer across a block of consecutive layers (or across speculative-decoding draft steps): the first computes which past tokens matter and the rest reuse its top-k selection, eliminating most indexer compute.See in glossary →, covered from the training side in the pre-training explainer, shares one indexer across every four backbone layers. The MTP layer gets the same treatment across time: the indexer runs only on the first draft step, and every following step reuses its top-k indices. The draft’s per-step cost drops accordingly (objective 1).

The surprise is that this same reuse also fixes an acceptance problem (objective 2). In multi-step drafting, GLM-5.1’s second draft step attended to a KV cacheKV cacheThe stored keys and values from all past tokens, so attention at step t only needs to compute Q for the new token.See in glossary → that was a mixture: keys and values for earlier positions came from the target model, but the newest entry came from the MTP layer’s own hidden state. During training, with teacher forcing, the MTP layer only ever saw target-model hidden states. So at inference it attended to a kind of history it had never seen in training: a classic train/inference mismatch, paid for in rejected drafts.

With IndexShare (plus the shared KV cache, KVShare), a later draft step reuses the first step’s indices and therefore attends only to positions whose KV came from the target model. What the draft sees at inference is now exactly what it saw in training. Training reuses the first draft step’s KV cache and indices the same way, closing the loop.

Rejection sampling and a loss that optimizes acceptance directly

The third and fourth improvements come from the Bebop paper (arXiv 2606.12370), which studied why MTP acceptance rates degrade, especially during RL training, and traced the problem to fluctuations in the model’s output entropy. Two fixes carry over to serving:

• Probabilistic rejection sampling instead of greedy drafting. Sample the draft tokens from the draft distribution and accept/reject them against the target distribution (the exact §18 protocol, which preserves the target distribution). Greedy drafts are brittle when the target distribution is flat; probabilistic drafting degrades gracefully.
• An end-to-end total-variation (TV) loss. Per-token cross-entropy trains the MTP layer to be a good predictor, but what serving actually pays for is the multi-step acceptance rate of the whole drafted chain. The TV loss optimizes that quantity directly, end to end across draft steps.

The ablation, measured as acceptance length on coding workloads (7 draft steps, GLM-5.1 backbone and data):

Serving a million tokens

The MTP work targets decode speed; GLM-5.2’s other serving headache is its headline feature. Extending context from 200K to 1M tokens shifts the bottleneck away from per-token FLOPs (which the sparse-attention work already cut) toward everything §12/§13 warned about: KV-cache capacity, kernels whose cost grows with context length, and CPU-side overhead in the serving engine. A 1M-token request’s KV cache is enormous even when its attention is cheap.

Z.ai reports optimizing along three fronts: finer-grained KV memory management and parallelization (building on LayerSplit) to reclaim usable cache space for ultra-long requests; coordinating long-context kernels with the KV-cache transfer pipeline so cache movement doesn’t stall prefill or decode; and CPU-side scheduling and runtime work to remove bubbles from the GPU pipeline. The result compounds with context length: normalized engine throughput vs GLM-5.1 goes from 1.03× at 32K to 6.97× at 1M, a regime where GLM-5.1 simply runs out of context.

The recap and pointers to further reading are next.