DeepSeek-V4

The next generation of efficient MoE

Paper: DeepSeek-V4 Technical Report — DeepSeek-AI, 2026

DeepSeek-V4 (DeepSeek-AI, 2026) is the most architecturally dense report of the 2026 frontier, and the natural successor to chapter 18. Where V3 was about training a frontier model cheaply, V4 is about a single hard target: million-token context at affordable cost. Every new piece serves that goal.

The lineup

The V4 series is, unsurprisingly, Mixture-of-ExpertsMixture of ExpertsMixture of Experts (MoE) — a layer with many parallel sub-networks ("experts") where a router sends each token to only a few. The model has a huge total parameter count but activates only a fraction per token, so compute stays modest.See in glossary →: DeepSeek-V4-Pro with 1.6 trillion total parameters (49B activeactive parametersIn a Mixture-of-Experts model, the subset of parameters actually used to process a given token. DeepSeek-V3 has 671B total but only 37B active per token, so compute tracks the smaller number.See in glossary →) and DeepSeek-V4-Flash at 284B (13B active), both supporting a one-million-token context windowcontext lengthThe maximum number of tokens the model can attend to at once (also called the context window or sequence length). Pre-training picks a context length; later stages often extend it.See in glossary →. Both were pre-trained on more than 32 trillion tokens, roughly double DeepSeek-V3’s 14.8T, a reminder that token counts keep climbing.

Hybrid attention for million-token context

The KV cache is the enemy of long context, and V4 attacks it with a hybrid attention architecture, different attention mechanisms in different layers:

• Compressed Sparse AttentionCompressed Sparse AttentionCompressed Sparse Attention (CSA) — a DeepSeek-V4 attention variant that attends to a compressed, sparsely-selected subset of past tokens to make million-token context affordable.See in glossary → (CSA): attend to a compressed, sparsely-selected subset of past tokens rather than all of them.
• Heavily Compressed Attention (HCA): an even more aggressively compressed variant for layers that can tolerate it.

This is the lineage of V3’s Multi-head Latent AttentionMulti-head Latent AttentionMulti-head Latent Attention (MLA) — DeepSeek's attention variant that compresses the keys and values into a small shared low-rank latent vector, drastically shrinking the KV cache while keeping multi-head expressivity.See in glossary →, pushed to the extreme that million-token context demands. The payoff is stark: at one-million-token context, V4-Pro reportedly needs only ~27% of the per-token inference FLOPs and ~10% of the KV cache of the previous DeepSeek generation. Long context stops being a memory catastrophe and becomes routine.

Two more upgrades: mHC and Muon

V4 also revisits two pieces we’d long taken as fixed.

1. Manifold-Constrained Hyper-Connections (mHC). Every deep transformer is held together by residual connectionsresidual connectionoutput = x + f(x). Lets gradients flow through deep stacks and means each block adds a refinement rather than rewriting.See in glossary → (chapter 9): each layer computes an update and adds it straight back to its own input, $x + \text{Sublayer}(x)$. That one fixed wiring (carry the full previous signal forward, add the new contribution on top) is what lets gradients travel down a hundred layers without fading away.

But “always add the whole signal back” is a hard-coded choice, and it forces a compromise: wire layers together too strongly and their representations blur into each other; too weakly and the gradient signal dies. Hyper-connectionshyper-connectionsA learnable replacement for the residual connection: instead of always adding a layer's output straight back to its input, the model keeps several parallel copies of the signal and learns how much of each to mix between layers. DeepSeek-V4's Manifold-Constrained variant (mHC) keeps those learned weights in a safe range so training stays stable.See in glossary → stop hard-coding that choice. They keep a few parallel copies of the running signal and let the model learn how much of each to feed into the next layer, and how much of that layer’s output to mix back in, in effect learning the wiring between layers instead of fixing it in advance.

The risk is that handing the network that many free knobs can make training wobble. The manifold-constrained part is the safety rail: V4 keeps those learned mixing weights confined to a well-behaved set (a manifoldmanifoldA smooth space that, up close, looks like ordinary flat space — e.g. the surface of a sphere is a 2D manifold. "Constraining weights to a manifold" means restricting them to a well-behaved subset (such as matrices of bounded size or fixed norm) instead of letting them take any value.See in glossary →: loosely, their sizes and proportions stay bounded), so the new freedom can never let the signal blow up or collapse. The result is a smarter, learnable stand-in for the plain residual that’s still safe to train at scale. After years of treating the residual as untouchable, even that primitive is now being upgraded.

2. The MuonMuonA newer optimizer (Momentum Orthogonalized by Newton-Schulz) that orthogonalizes each weight-matrix update instead of scaling it per-element like Adam. Used at scale by Kimi K2.5 via the MuonClip variant.See in glossary → optimizer. Like Kimi, V4 uses Muon for most parameter matrices, while retaining AdamWAdamWAdam with decoupled weight decay. It applies weight decay directly to the parameters instead of folding it into the gradient, which often regularizes more cleanly.See in glossary → for the embedding module, prediction head, and RMSNorm weights. The report attributes this mixed optimizer setup to faster convergence and improved training stability. Two independent frontier labs adopting Muon at this scale is a signal that the optimizer landscape may be shifting.

One frontier report remains, and it widens the lens as far as it goes: to a model pre-trained on every modality at once.