Kimi K3

KDA, attention across depth, and stable extreme-sparse MoE

Paper: Kimi K3: Open Frontier Intelligence — Kimi Team, 2026

The previous frontier chapters each rebuilt one part of the pre-training stack. Kimi K3 (Kimi Team, July 2026) rebuilds three at once. It is a 2.8-trillion-parameter Mixture-of-ExpertsMixture of ExpertsMixture of Experts (MoE) — a layer with many parallel sub-networks ("experts") where a router sends each token to only a few. The model has a huge total parameter count but activates only a fraction per token, so compute stays modest.See in glossary →, with 104 billion parameters active per token, native vision, and a one-million-token context window. Its organizing idea is more useful than the parameter count: scale the paths along which information moves, across the sequence, through the network’s depth, and across its expert width.

Across the sequence: mostly recurrent, periodically global

K3 replaces an all-attention stack with a repeating hybrid: three Kimi Delta AttentionKimi Delta AttentionKimi Delta Attention (KDA) — a gated delta-rule linear-attention mechanism that compresses prior tokens into a fixed-size recurrent state with channel-wise forgetting.See in glossary → (KDA) layers followed by one gated MLAMulti-head Latent AttentionMulti-head Latent Attention (MLA) — DeepSeek's attention variant that compresses the keys and values into a small shared low-rank latent vector, drastically shrinking the KV cache while keeping multi-head expressivity.See in glossary → layer. Across the 93-layer stack that gives 69 KDA layers and 24 MLA layers, with one extra rule: the final layer is always a global MLA layer, so the representation feeding the prediction head can look at the entire context. (“Gated” is literal: each attention output passes through a learned, input-dependent sigmoid gate before rejoining the stream, letting the model dial each channel’s contribution per token.) KDA is a form of linear attentionlinear attentionAn attention-family mechanism that summarizes earlier tokens into a recurrent state instead of comparing every query with every prior key. Its sequence cost can grow linearly, but the summary is compressed.See in glossary →. Instead of keeping a key and value for every prior token, it compresses the past into a fixed-size recurrent staterecurrent stateA fixed-size memory updated as each token arrives. Unlike a KV cache, it does not grow with sequence length, but later updates overwrite and mix information into the same state.See in glossary →. Each new token reads that state, selectively forgets parts of it, and writes new information into it.

That fixed state is the appeal: its size does not grow with context length. The tradeoff is compression. A recurrent state cannot preserve every past token exactly, so the periodic MLA layer supplies unrestricted global lookup. K3 spends three quarters of its attention layers on cheap recurrent mixing and one quarter on full, compressed-KV attention.

KDA’s details are designed backwards from hardware. Its channel-wise decay determines how quickly each part of the state forgets. Earlier KDA formulations allowed the decay to become arbitrarily small; inside the parallel training algorithm, dividing by a product of those tiny values could overflow. K3 lower-bounds the decay so the worst 16-token product remains representable in BF16BF16Brain Floating-point 16-bit: 1 sign + 8 exponent + 7 mantissa bits. Keeps FP32's wide exponent range (so it rarely overflows) at the cost of precision — the workhorse format for modern pre-training.See in glossary →. That numerical bound eliminates a special slow path and lets every tile use dense Tensor Core matrix multiplication. The co-design runs in the other direction too: attention outputs are kept in FP32 during training to cancel a small biased rounding error in the standard FlashAttention recipe, and the kernel was rebuilt so the larger output tile hides behind existing memory traffic.

The global MLA layers use no explicit positional encoding. KDA’s ordered recurrence already supplies position and recency information, so the model does not need to retune a RoPERoPERotary Position Embeddings — rotates Q/K vectors by an angle proportional to position. Standard in modern LLMs.See in glossary → base when the context grows. The team reports direct extrapolation to one million tokens without position interpolation.

Across depth: attention replaces the residual pile

A standard residual connectionresidual connectionoutput = x + f(x). Lets gradients flow through deep stacks and means each block adds a refinement rather than rewriting.See in glossary → carries one running vector through the stack. By layer 90, that vector is a sum in which every earlier update has already been blended together. K3’s Attention ResidualsAttention ResidualsA learned replacement for uniform residual accumulation in which each layer attends over representations from earlier layers or blocks, selectively retrieving information across network depth.See in glossary → (AttnRes) treat depth the way the Transformer treated time: let the next layer retrieve earlier representations with learned attention weights.

The full version could preserve every layer output, but that would increase activation memory and pipeline communication. Block AttnRes instead sums outputs within blocks of twelve layers and attends across the resulting eight block-level representations, with the token embedding always included as a ninth source. It recovers most of the benefit while reducing the saved state from one vector per layer to one per block.

This is a different answer to the same question DeepSeek-V4’s hyper-connections raised in chapter 26. A residual stream need not be fixed plumbing. V4 learns how parallel streams mix; K3 lets each layer look back and choose which depth-level representation it needs.

Across width: extreme sparsity in a smaller latent space

K3 routes each token to 16 of 896 experts. A conventional MoE would send the model’s full 7,168-dimensional token vector to every selected expert, making communication and weight traffic grow with the number of routes. LatentMoELatentMoEA Mixture-of-Experts design that projects the routed path into a narrower latent space, letting a token visit more specialized experts without sending its full-width representation to each one.See in glossary → projects the routed path down to half width first. Two shared experts retain the full-width path for common transformations; the 896 specialists work in the cheaper latent space.

Extreme sparsity exposed two stability problems: routed activations could explode, and nearly 900 experts were difficult to balance. The explosion risk is structural. The routed path is a down-projection, a gated two-branch feed-forward, and an up-projection: nearly four consecutive matrix multiplications whose product can grow fast at 2.8T scale. Stable LatentMoE answers with three small mechanisms:

• An extra RMSNorm stabilizes the aggregate before projecting it back to full width.
• SiTU-GLUSiTU-GLUSigmoid Tanh Unit GLU — a gated feed-forward activation that resembles SwiGLU near the origin but soft-caps both multiplicative branches to control activation outliers.See in glossary → soft-caps both multiplicative branches of the feed-forward activation (the gate at 4, the up branch at 25, so their product never exceeds 100). Near zero it behaves like SwiGLU; at large values it cannot grow without bound.
• Quantile BalancingQuantile BalancingAn auxiliary-loss-free MoE balancing rule that derives each expert routing bias from the router-score quantile corresponding to that expert’s target token load.See in glossary → sets each expert’s routing bias from the score quantile that would give it its target token load. A distributed histogram estimates that global quantile without gathering millions of router scores.

The last mechanism upgrades DeepSeek-V3’s auxiliary-loss-free balancing. A fixed bias step has to choose between slow correction and oscillation. Quantile Balancing estimates the correction the current score distribution actually requires, then applies it to the next batch. The routing bias changes which experts are selected but not their mixture weights, so balancing does not compete with the language-modeling loss, and at inference the bias is simply frozen.

Training the whole system

K3 continues the Kimi line’s use of MuonMuonA newer optimizer (Momentum Orthogonalized by Newton-Schulz) that orthogonalizes each weight-matrix update instead of scaling it per-element like Adam. Used at scale by Kimi K2.5 via the MuonClip variant.See in glossary → (with K2’s weight-clipping safeguard retained), but orthogonalizes the query, key, and value updates one attention head at a time. A single full-matrix operation lets large-gradient heads dominate its scale; Per-Head Muon gives each head its own normalization problem and reportedly improves large-scale stability. The learning-rate schedule is a plain cosine decay with a short warmup, and the paper defends that choice with data: in their scaling study, cosine consistently beat the newer warmup-stable-decay schedule once each schedule had its own hyperparameters tuned, a useful caution against comparing schedules under shared settings.

Vision is native, as in K2.5, but the initialization changes. K3 trains its 401-million-parameter MoonViT-V2 encoder from scratch under the same next-token objective rather than attaching a contrastively pre-trained vision tower. The from-scratch tower had smaller, less spiky gradients and matched the initialized baseline on vision evaluations. At this scale, a separate contrastive apprenticeship was unnecessary.

The text corpus spans four domains (web text, code, mathematics, knowledge), and the knowledge and math portions are rephrased with diverse prompting and then verified against their sources for fidelity, the same controlled use of synthetic rewriting the K2 line established. The vision corpus adds a distinctive ingredient: programmatic multimodal data, code snippets paired with the visuals they render (SVG, web pages, 3D assets, CAD), where the correspondence between text and image is exact by construction.

The one-million-token window is a curriculum, not the starting point. Context grows from 8K to 64K in pre-training, then 256K to 1M during cooldown. Natural long documents and videos are cleaned and upsampled, but length alone is not enough: synthetic multimodal sequences scatter dependent sub-tasks throughout the full window so the answer genuinely requires distant information.

The paper reports that the combined architecture, data, and training recipe shifts the fitted scaling curve by roughly 2.5× relative to Kimi K2: the same held-out loss at about 40% of the compute. It is a collective result, not an ablation credit assigned to one component. That is the right final lesson for this explainer. Frontier pre-training is no longer one large trick. It is a numerical range, a routing rule, a residual path, a data curriculum, a kernel, and an optimizer designed together so a model this sparse and this long can train at all.

One frontier paper remains, and it turns the lens back on this explainer’s own toolkit: the scaling laws of chapters 14 and 16 get a correction of their own.