Section 22

Serving recurrent attention

Hybrid caches, state replay, and million-token scheduling

Listen to this chapter

Paper: Kimi K3: Open Frontier Intelligence — Kimi Team, 2026

Everything so far assumed that attention remembers a prefix by caching keys and values for its tokens. Kimi K3 makes serving more complicated—and, at very long context, more interesting. Three quarters of its attention layers use Kimi Delta AttentionKimi Delta AttentionKimi Delta Attention (KDA) — a gated delta-rule linear-attention mechanism that compresses prior tokens into a fixed-size recurrent state with channel-wise forgetting.See in glossary → (KDA), which compresses the past into a fixed-size recurrent staterecurrent stateA fixed-size memory updated as each token arrives. Unlike a KV cache, it does not grow with sequence length, but later updates overwrite and mix information into the same state.See in glossary →. The remaining quarter use MLA and retain a sequence-growing KV cacheKV cacheThe stored keys and values from all past tokens, so attention at step t only needs to compute Q for the new token.See in glossary →.

A K3 request therefore carries two different memories of the same prefix. One grows per token and is naturally paged. The other stays fixed in size but is mutated on every step. A prefix is reusable only where both memories describe exactly the same tokens.

One pool, two cache types

The serving engine packs KDA states into the same physical page pool as MLA keys and values. Both page types use one allocator, reference-counting scheme, eviction policy, and transfer path. That does not make the contents interchangeable; it makes their lifecycle machinery shared.

Their useful granularities are different. An MLA cache can be hashed in small blocks—512 tokens in the paper’s example—so a request can reuse a prefix ending at any completed hash block. Saving KDA’s relatively large recurrent state every 512 tokens would consume too much memory. If hashing were forced to use KDA’s coarse checkpoint interval, however, short prefixes and partially filled pages would almost never hit.

The solution is to decouple physical allocation, prefix hashing, and recurrent-state checkpoints:

  • Large physical pages allocate memory efficiently.
  • Small logical hash blocks identify fine-grained matching MLA prefixes.
  • KDA checkpoints exist at only a sparse subset of those hash boundaries, especially conversation turns.
One reusable prefix, two kinds of state
A cache hit is valid only where the growing MLA cache and a KDA checkpoint describe the same prefix.
MLA KV cache — grows token by token
0–512
512–1024
1024–1536
1536–2048
2048–2560
2560–3072
KDA state — fixed size, checkpointed sparsely
Longest joint hit: token 2,560. Restore a private KDA state, reuse five MLA hash blocks, and resume prefill there.
Match: hash the fine MLA blocks.
Validate: require every KDA cache group.
Resume: copy the immutable checkpoint before updating.

Lookup first finds the longest matching MLA prefix, then walks back to the longest boundary with a checkpoint in every KDA cache group. It restores that immutable checkpoint into the request’s private running state and copy-on-writes the partially filled MLA page. Prefill resumes at the joint boundary without recomputing the earlier prefix.

Concurrent scheduling adds consistency rules. All cache groups pin their hit pages before any private allocation can trigger eviction. Newly copied blocks do not become matchable until the GPU copy has completed. Evicting one KDA group’s checkpoint invalidates its siblings atomically. The invariant is simple even when the implementation is not: a visible cache entry always corresponds to the token boundary it claims.

Speculative decoding without state snapshots

Mutable recurrent state creates another problem for speculative decodingspeculative decodingA small draft model proposes K tokens; the big target model verifies them all in one pass. Net effect: more tokens per target-model step.See in glossary →. Suppose a draft proposes seven tokens and KDA advances its state through all seven before the target rejects token four. A KV cache can discard the rejected token entries. A single recurrent state has already mixed them in; it cannot be sliced backward.

The obvious fix—save the full state after every draft position—multiplies state memory traffic, which is already the decode bottleneck. K3 instead caches the projected inputs of the draft tokens. These are much smaller than the recurrent state. After verification, a fused kernel replays only the accepted projections on-chip, reconstructs the correct state, adds the target model’s bonus token, and continues into the next draft window.

This is a useful general pattern for state-space and recurrent models: when a mutable state is expensive to copy but cheap to recompute from compact inputs, log the inputs and replay the accepted prefix.

Kernels follow the serving phase

KDA uses different kernels for different regimes. Training and prefill expose parallel work across chunks; decoding exposes only a small token step and is dominated by updating the state. Block Attention Residuals similarly use sequence-sharded activations during prefill but overlap their inter-block read on a side stream during decode. Stable LatentMoE fuses its projection with routing and uses token-centric expert kernels when small batches make weight streaming the bottleneck.

Schedule sessions, not average requests

At one million tokens, a cache miss can cost orders of magnitude more than a hit. The paper’s representative coding request has a 400K-token cached prefix and only 4K new tokens. Routing that next turn to a cluster without its cache turns a small incremental prefill into a 404K-token one.

K3 uses cache-aware affinity: a session returns to the cluster holding its prefix. Consistent hashing assigns a secondary cluster as well, so a primary failure spreads the expensive re-prefill work across the fleet rather than concentrating it on one backup.

Request cost also spans roughly three orders of magnitude, from under 2K to one million tokens. A queue sized by request count or “average request” cannot protect latency when a burst of ultra-long prompts arrives. Budget-based admission controladmission controlA serving policy that decides which requests may enter execution given available capacity. Budgeting request classes separately prevents very costly traffic from starving cheaper requests.See in glossary → gives different request classes separate resource budgets, preventing the long-context class from consuming all capacity and destroying time-to-first-token for short traffic.

The inference story ends where the explainer began: with memory. KDA replaces a linearly growing cache with fixed recurrent state, but that state still has to be checkpointed, restored, replayed, transferred, and scheduled correctly. Eliminating one memory curve does not eliminate systems work. It changes the shape of the state the serving engine must manage.

The final chapter recaps the full path from tokenization to fleet-level scheduling.