Reasoning effort is trainable

Budget-controlled RL, nine teachers, and persistent rollouts

Paper: Kimi K3: Open Frontier Intelligence — Kimi Team, 2026

Inference scaling is usually described as a serving-time choice: let the model think longer when a problem is hard. Kimi K3’s post-training recipe makes a stronger claim. Reasoning effort can itself be a training target. The team trains policies for different domains and budgets, then consolidates them into one model that can operate at low, high, or maximum effort.

That target changes the shape of the RL system. A maximum-effort coding agent may make thousands of tool calls and accumulate millions of context tokens. Its rollout cannot be treated like a long chat completion that must finish before the next update. Training has to preserve the policy’s state, the environment’s state, and the intended effort level across many optimizer iterations.

Three domains times three effort levels

K3 divides RL into three broad domains:

• General tasks: knowledge, reasoning, vision, search, and faithfulness.
• General agents: deep research, long-form writing, and assistant workflows.
• Coding agents: software engineering, kernels, and web development.

For each domain, it trains low-, high-, and max-effort experts: nine policies in total.

The effort control is learned with a per-problem budget. The cold-start model first estimates a baseline token budget $b_0(x)$ for problem $x$. A rollout loses its task reward if its total output exceeds a multiple $\tau b_0(x)$. For ordinary reasoning, that total counts thinking tokens; for agents, it includes reasoning and tool-call arguments across the trajectory.

Training begins with a relatively generous multiplier to produce the max-effort expert, while still preventing unlimited overthinking. The multiplier is then annealed downward for high and low effort. This is more targeted than one global response-length penalty: a task that naturally needs 20,000 tokens gets a larger allowance than one that normally needs 500.

For non-verifiable work, an agentic generative reward modelgenerative reward modelA reward model that is itself a language model: it writes out a critique or reasons step by step before scoring a response, rather than emitting a single opaque scalar. More accurate and interpretable than a bare scalar head.See in glossary → compares candidates after writing a rubric and scoring each candidate against it. A parallel verbosity rule makes overlong outputs automatically lose the comparison. The same budget idea therefore controls both hidden reasoning and the visible temptation to win a judge by saying more.

Partial rollouts: stop waiting for the longest agent

Synchronous RL normally generates a batch of rollouts, waits for all of them, then updates the policy. Long-horizon agents create an extreme stragglerstragglerThe slowest item in a synchronized batch. Every other worker may sit idle while the system waits for this last rollout or computation to finish.See in glossary → problem: one trajectory can continue for hours after most of the batch has finished.

K3 uses partial rolloutspartial rolloutA trajectory that is paused before completion, carried across policy updates with its model and environment state intact, and resumed in a later RL iteration.See in glossary →. Generation pauses once a chosen fraction of active trajectories completes. Finished groups go to policy optimization; unfinished trajectories enter a queue and resume at the start of a later iteration. One logical episode can therefore span multiple policy updates.

That creates stale data: the trajectory began under an older policy than the one currently being optimized. K3 relies on per-token regularization that constrains each update to a local neighborhood, allowing the learner to tolerate these unusually off-policy fragments. The important systems idea is that a rollout is now a resumable process, not a disposable string.

The environment must resume too. K3’s microVM sandboxes can checkpoint only dirtied memory pages, pause without consuming CPU or memory, fork an exact state for side-effect-free judging, and retain snapshots for recovery. A personal-assistant task can evolve across simulated days and thousands of tool calls without being reconstructed after every RL iteration.

Train across harnesses, not for one harness

An agent can overfit to a tool schema or context-management convention just as a language model can overfit to a prompt format. K3 represents a harness as configurable modules—tools, prompts, memory, compaction, skills, and subagents—and varies their combinations across RL tasks. The intended invariant is the loop underneath the scaffold: reason, act, observe, verify, and adapt.

The training tasks reinforce that loop with external state and verifiers. Examples include professional workflows in mock SaaS applications, visual reasoning where Python crops and transforms an image, kernel optimization rewarded for correctness and speed, and autonomous execution tasks whose hidden verifier checks the final environment rather than the agent’s claim that it finished.

Nine teachers become one policy

Shipping nine separate models would defeat the point. K3 consolidates them with multi-teacher on-policy distillationon-policy distillationDistillation where the student generates its own rollouts and a teacher grades every token of them. Combines the dense per-token signal of distillation with the on-policy benefit of RL — much cheaper than RL for transferring a teacher's reasoning.See in glossary →. The student samples a domain and effort level, generates its own rollout, and receives a dense per-token log-ratio reward from the matching teacher. Clipping extreme log ratios stabilizes the signal.

This is chapter 31’s method applied along two explicit axes. The teachers are not only domain specialists; they represent distinct computational budgets. The released student learns all nine behaviors and an interface for selecting among the effort levels.

Post-training is also deployment-aware. The expert weights use MXFP4 and their activations MXFP8 throughout SFT and RL, while sensitive non-expert modules stay at higher precision. Rollout and training use the same quantization scheme, preventing the policy from being optimized in one numerical environment and sampled in another.

The frontier pipeline now looks less like one monolithic RL run and more like a small organization: specialist policies work at different budgets, persistent environments keep their projects alive, and a final student absorbs their behavior. The recap reassembles that pipeline from the first instruction demonstrations to these million-token trajectories.