DeepSeek-V4

The next generation of efficient MoE

Paper: DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence — DeepSeek-AI, 2026

DeepSeek-V4 (DeepSeek-AI, 2026) is the most architecturally dense report of the 2026 frontier, and the natural successor to chapter 18. Where V3 was about training a frontier model cheaply, V4 is about a single hard target: million-token context at affordable cost. Every new piece serves that goal.

The lineup

The V4 series is, unsurprisingly, Mixture-of-ExpertsMixture of ExpertsMixture of Experts (MoE) — a layer with many parallel sub-networks ("experts") where a router sends each token to only a few. The model has a huge total parameter count but activates only a fraction per token, so compute stays modest.See in glossary →: DeepSeek-V4-Pro with 1.6 trillion total parameters (49B activeactive parametersIn a Mixture-of-Experts model, the subset of parameters actually used to process a given token. DeepSeek-V3 has 671B total but only 37B active per token, so compute tracks the smaller number.See in glossary →) and DeepSeek-V4-Flash at 284B (13B active), both supporting a one-million-token context windowcontext lengthThe maximum number of tokens the model can attend to at once (also called the context window or sequence length). Pre-training picks a context length; later stages often extend it.See in glossary →. Pro was pre-trained on 33 trillion tokens and Flash on 32T, roughly double DeepSeek-V3’s 14.8T, a reminder that token counts keep climbing.

Hybrid attention for million-token context

The KV cache is the enemy of long context, and V4 attacks it with a hybrid attention architecture, two mechanisms interleaved in a fixed pattern through the stack:

• Compressed Sparse AttentionCompressed Sparse AttentionCompressed Sparse Attention (CSA) — a DeepSeek-V4 attention variant that attends to a compressed, sparsely-selected subset of past tokens to make million-token context affordable.See in glossary → (CSA): pool every 4 past tokens into one compressed entry, then let a lightweight lightning indexer score those entries with cheap dot products and keep only the top 1,024 for real attention. Compression stacked on the sparse top-k selectionDeepSeek Sparse AttentionDeepSeek Sparse Attention (DSA): an attention variant where a lightweight indexer scores every past token with cheap dot products, a top-k selection keeps the most relevant ones, and full attention runs over only that subset.See in glossary → DeepSeek introduced in V3.2.
• Heavily Compressed Attention (HCA): compress far harder, 128 tokens per entry, and then attend densely over all of them. No selection is needed; the compression alone has already made the history 128× shorter.

So CSA layers do precise retrieval over a lightly compressed history, while HCA layers keep a cheap global gist of everything. Two small patches restore what compression loses: every query also attends to the most recent 128 raw tokens through a sliding window (a token cannot see inside its own not-yet-complete compressed block), and learnable attention sinksattention sinksLearnable per-head "park here" slots added to the attention softmax: when no past token is actually relevant, a head can place its attention mass on the sink instead of being forced to smear probability over unrelated tokens.See in glossary → let a head place its attention mass on nothing at all when no past entry is relevant.

The two halves of hybrid attention

CSA (top) compresses KV entries and selects the top-k with a lightning indexer; HCA (bottom) compresses far harder and attends densely over everything.

This is the lineage of V3’s Multi-head Latent AttentionMulti-head Latent AttentionMulti-head Latent Attention (MLA) — DeepSeek's attention variant that compresses the keys and values into a small shared low-rank latent vector, drastically shrinking the KV cache while keeping multi-head expressivity.See in glossary → and V3.2’s sparse attention, pushed to the extreme that million-token context demands. The payoff is stark: at one-million-token context, V4-Pro needs only ~27% of the per-token inference FLOPs and ~10% of the KV cache of DeepSeek-V3.2, itself already a sparse-attention model, and the smaller Flash gets to 10% and 7%. Long context stops being a memory catastrophe and becomes routine.

Two more upgrades: mHC and Muon

V4 also revisits two pieces we’d long taken as fixed.

1. Manifold-Constrained Hyper-Connections (mHC). Every deep transformer is held together by residual connectionsresidual connectionoutput = x + f(x). Lets gradients flow through deep stacks and means each block adds a refinement rather than rewriting.See in glossary → (chapter 9): each layer computes an update and adds it straight back to its own input, $x + \text{Sublayer}(x)$. That one fixed wiring (carry the full previous signal forward, add the new contribution on top) is what lets gradients travel down a hundred layers without fading away.

But “always add the whole signal back” is a hard-coded choice, and it forces a compromise: wire layers together too strongly and their representations blur into each other; too weakly and the gradient signal dies. Hyper-connectionshyper-connectionsA learnable replacement for the residual connection: instead of always adding a layer's output straight back to its input, the model keeps several parallel copies of the signal and learns how much of each to mix between layers. DeepSeek-V4's Manifold-Constrained variant (mHC) keeps those learned weights in a safe range so training stays stable.See in glossary → stop hard-coding that choice. They keep a few parallel copies of the running signal and let the model learn how much of each to feed into the next layer, and how much of that layer’s output to mix back in, in effect learning the wiring between layers instead of fixing it in advance.

The risk is that handing the network that many free knobs can make training wobble; the paper reports that unconstrained hyper-connections frequently go numerically unstable once many layers stack. The manifold-constrained part is the safety rail: V4 keeps those learned mixing weights confined to a well-behaved set (a manifoldmanifoldA smooth space that, up close, looks like ordinary flat space — e.g. the surface of a sphere is a 2D manifold. "Constraining weights to a manifold" means restricting them to a well-behaved subset (such as matrices of bounded size or fixed norm) instead of letting them take any value.See in glossary →). Concretely, each layer-to-layer mixing matrix is kept doubly stochastic: all entries non-negative, every row and column summing to one, so the mixing can redistribute signal between the parallel streams but never amplify it, no matter how many layers you stack. The result is a smarter, learnable stand-in for the plain residual that’s still safe to train at scale. After years of treating the residual as untouchable, even that primitive is now being upgraded.

2. The MuonMuonA newer optimizer (Momentum Orthogonalized by Newton-Schulz) that orthogonalizes each weight-matrix update instead of scaling it per-element like Adam. Used at scale by Kimi K2.5 via the MuonClip variant.See in glossary → optimizer. Like Kimi, V4 uses Muon for most parameter matrices, while retaining AdamWAdamWAdam with decoupled weight decay. It applies weight decay directly to the parameters instead of folding it into the gradient, which often regularizes more cleanly.See in glossary → for the embedding module, prediction head, RMSNorm weights, and the small mHC gating parameters. Two practical details stand out. Muon’s updates are rescaled to a fixed magnitude chosen so that AdamW’s learning-rate settings carry over unchanged, which means years of accumulated tuning wisdom did not have to be rediscovered. And unlike Kimi’s MuonClipMuonClipA stabilized variant of the Muon optimizer (used by the Kimi models) that clips/rescales attention query-key logits to prevent the loss spikes that can derail very large training runs.See in glossary → (chapter 24), V4 needs no logit-clipping mechanism at all: applying RMSNorm directly to attention queries and KV entries keeps attention logits bounded by construction. Two independent frontier labs adopting Muon at this scale is a signal that the optimizer landscape may be shifting.

Managing the run: curriculum and stability

The paper is unusually candid about how the training run itself was managed, and two details connect back to earlier chapters.

Sparsity is a curriculum, not a starting point. Sequence length grows in stages, 4K to 16K to 64K and finally to one million tokens, and attention starts out fully dense for the first trillion tokens. Only at the 64K stage does V4 switch the sparse machinery on, after a short stage that warms up just the lightning indexer. The model first learns what attention should retrieve before being asked to retrieve it through a compressed, sparse view. The data side evolves too: web filtering explicitly targets batched, auto-generated and templated content, a defense against the model-collapse risk of a web increasingly written by models.

Loss spikes got a diagnosis and two fixes. DeepSeek traced its loss spikes to activation outliers in the MoE layers, amplified by the router itself: an outlier shifts routing scores, which reshuffles experts, which produces more outliers. Rolling back and skipping data did not stop recurrences. The fix is Anticipatory RoutingAnticipatory RoutingA DeepSeek-V4 stability mechanism: when a loss spike is detected, the MoE router temporarily picks experts using slightly older parameters while the rest of the model trains with current ones, breaking the outlier→routing-shift→outlier feedback loop, then reverts.See in glossary →: when a spike is detected, the router temporarily computes its expert choices with slightly older parameters while the rest of the model trains with current ones, breaking the feedback loop for the duration of the episode at about 20% overhead, then switching back. Alongside it, the feed-forward activations are simply clamped to a fixed range throughout training. The authors admit the deeper mechanism behind spikes “remains an open question”: even at the frontier, some of this is still engineering ahead of theory.

One frontier report remains, and it widens the lens as far as it goes: to a model pre-trained on every modality at once.