DeepSeek-V3

MoE, MLA, MTP, and FP8 at scale

Paper: DeepSeek-V3 Technical Report — DeepSeek-AI, 2024

If Llama 3 is the modern dense baseline, DeepSeek-V3 (DeepSeek-AI, 2024) is the modern efficiency masterclass. It trained a model with 671 billion total parameters to frontier quality for under $6 million of compute, a result that reset expectations for what a training run costs. Almost every part of that came from rethinking architecture and numerics together. This is the densest chapter; take it slowly.

Mixture-of-Experts: huge capacity, small active compute

DeepSeek-V3 is a Mixture-of-ExpertsMixture of ExpertsMixture of Experts (MoE) — a layer with many parallel sub-networks ("experts") where a router sends each token to only a few. The model has a huge total parameter count but activates only a fraction per token, so compute stays modest.See in glossary → model. Of its 671B parameters, only 37B are activeactive parametersIn a Mixture-of-Experts model, the subset of parameters actually used to process a given token. DeepSeek-V3 has 671B total but only 37B active per token, so compute tracks the smaller number.See in glossary → per token: a router sends each token to a small subset of experts, so compute tracks the 37B, not the 671B. You get the knowledge capacity of a giant model at the FLOP cost of a medium one.

DeepSeek’s specific design, DeepSeekMoE, has two ingredients:

• Fine-grained experts. Many small experts instead of a few big ones, giving the router more precise specialization.
• Shared expertshared expertIn DeepSeekMoE, an expert that every token always passes through (alongside a few routed experts), used to capture common knowledge so the routed experts can specialize.See in glossary →. One expert processes every token, absorbing common knowledge so the routed experts are free to specialize.

Play with the router above and you’ll feel the core tension of MoE: as you add experts, total capacity grows but each expert sees fewer tokens, and the load across experts gets uneven. An overloaded expert (and the GPU holding it under expert parallelismexpert parallelismPlacing different experts of a Mixture-of-Experts layer on different GPUs, so each device holds only some experts and tokens are routed across the network to reach them.See in glossary →) becomes a bottleneck.

Multi-head Latent Attention: a tiny KV cache

DeepSeek’s second architectural lever is Multi-head Latent AttentionMulti-head Latent AttentionMulti-head Latent Attention (MLA) — DeepSeek's attention variant that compresses the keys and values into a small shared low-rank latent vector, drastically shrinking the KV cache while keeping multi-head expressivity.See in glossary → (MLA). Standard attention caches a key and value vector for every head at every position: the KV cache that dominates memory at long context. MLA instead compresses the keys and values into a single small low-rank latent vector per token, from which the per-head keys and values are reconstructed on the fly. The cache stores the latent, not all the heads.

The widget compares the three attention designs we’ve now met. GQAGQAGrouped-Query Attention — multiple query heads share one K/V head, shrinking the KV cache by 4–8× with minimal quality loss.See in glossary → (Llama, Qwen, Gemma) shrinks the KV cache by sharing key/value heads; MLA shrinks it much further by storing one compressed latent. A smaller KV cache means longer context and bigger batches fit in memory, which helps both training throughput and inference cost.

Multi-Token Prediction

The third change is to the objective itself. Ordinary pre-training asks the model a single question at every position: given everything so far, what is the very next token? That is next-token predictionnext-token predictionThe pre-training objective for GPT-style models: given the tokens so far, predict a probability distribution over the next token. Also called causal or autoregressive language modeling.See in glossary → with a cross-entropycross-entropy lossA classification loss that penalizes the model according to the negative log-probability it assigned to the correct answer.See in glossary → loss, and it supplies exactly one unit of supervision per position. DeepSeek-V3 asks for more. With Multi-Token PredictionMulti-Token PredictionMulti-Token Prediction (MTP) — a training objective where the model predicts several future tokens at each position (not just the next one), densifying the learning signal and enabling faster speculative decoding later.See in glossary → (MTP), each position is also trained to predict the token two ahead, three ahead, and so on for a few steps, so one forward pass yields several training signals instead of one.

The detail that makes DeepSeek’s version work is how those further tokens are predicted. The naive approach bolts several independent heads onto the final layer and lets them all guess in parallel (roughly what Medusa does, purely for inference speed). DeepSeek instead chains small MTP modules in sequence: the module for prediction depth $k$ takes the hidden state from depth $k\!-\!1$ together with the embedding of the actual next token, runs it through its own transformer block, and predicts the token one step further out. Because each module is fed the real tokens that precede its target, the full causal dependency is preserved at every depth: the model is predicting a coherent continuation, not making $k$ blind parallel guesses. The per-depth losses are averaged, scaled by a small weight $\lambda$, and added to the main next-token loss.

Why does this help? During training it densifies the signal: being asked to predict several tokens ahead forces the representations to encode where the sequence is going, not just the immediate next symbol. The result is better sample efficiency and a bit of built-in planning. DeepSeek-V3 keeps it cheap, using a single extra depth (predicting two tokens total), and reports a consistent quality gain across most benchmarks for that one addition.

FP8 training, validated at scale

Finally, the numerics. DeepSeek-V3 is the first model to demonstrate FP8FP88-bit Floating Point (typically E4M3 or E5M2 layouts). The newest training precision, used on H100/Blackwell GPUs to roughly double throughput; needs careful scaling to stay numerically stable.See in glossary → mixed-precisionmixed-precision trainingDoing the heavy matrix multiplies in a low-precision format (BF16/FP8) for speed while keeping a high-precision (FP32) copy of the weights and accumulating sensitive sums in FP32 for stability.See in glossary → training on an extremely large run. Recall from the precision chapter that FP8’s range is tiny; DeepSeek’s answer is fine-grained scaling (separate scaling factors for small tiles/blocks of each tensor) plus keeping the most sensitive accumulations in higher precision. The payoff is roughly double the throughput of BF16 and half the memory, which is a big part of how the run came in so cheap.

DeepSeek-V3 is the template for efficient frontier pre-training: sparse compute (MoE), a tiny KV cache (MLA), a denser objective (MTP), and aggressive numerics (FP8). We’ll see its ideas echo through the 2026 frontier. But first, two more open models (Qwen and Gemma) each with a different emphasis.