On-policy distillation

A dense teacher signal on the student's own rollouts

Papers: On-Policy Distillation of Language Models (GKD) — Agarwal et al., 2023 · On-Policy Distillation — Thinking Machines Lab, 2025 · Qwen3 Technical Report — Qwen Team, 2025 · RL vs. Distillation — Kim et al., 2025

Two of the most useful tools in this explainer pull in opposite directions. Distillationknowledge distillationTraining a smaller "student" model to match the full output probability distribution of a larger "teacher" model, rather than just the one-hot next token. Richer targets let the student learn more per token.See in glossary → (train a small student to copy a strong teacher) gives a dense signal: the teacher has an opinion about every token. Reinforcement learning gives an on-policy signal: the model learns on the exact outputs it actually produces. On-policy distillation is the idea of getting both at once, and it has become one of the cheaper ways to teach a small model to reason.

Two flavors of distillation

Everywhere distillation has appeared so far, it has been off-policy: the student trains on text the teacher produced. Alpaca fine-tuned on GPT-generated answers; DeepSeek-R1’s distilled models are plain supervised fine-tuning on a fixed set of R1’s reasoning traces. The student copies a frozen pile of teacher outputs.

There is a subtle problem with that, and it’s the same distribution mismatch we keep running into. The student is trained on the teacher’s trajectories, but at inference it has to walk on its own. The moment it makes a mistake the teacher never would, it’s in a state it was never trained on, and small errors compound across a long answer. Worse, copying the teacher’s tokens teaches its style and confidence without guaranteeing the student can actually reach the right answer from the places it tends to wander into.

On-policy distillationon-policy distillationDistillation where the student generates its own rollouts and a teacher grades every token of them. Combines the dense per-token signal of distillation with the on-policy benefit of RL — much cheaper than RL for transferring a teacher's reasoning.See in glossary → fixes the mismatch the same way RL does: let the student generate its own rollouts, then have the teacher grade them. The student is now trained exactly on the states it visits at inference time, but instead of one reward at the end, the teacher hands back a judgment on every token.

The mechanism: a dense per-token teacher

Concretely, for each token the student samples, you ask the teacher: how likely were you to say that, here? The training signal is the per-token gap between the student’s and teacher’s distributions (a reverse KL divergencereverse KLThe KL divergence measured with the student's distribution in front, D(student ‖ teacher). It is mode-seeking — it pushes the student to concentrate on what the teacher actually does rather than spread mass thinly. Used as the per-token signal in on-policy distillation.See in glossary →) used as a dense reward and optimized with the very policy-gradientpolicy gradientA family of RL methods that directly adjust the policy’s parameters in the direction that increases expected reward, using the score-function (REINFORCE) estimator.See in glossary → machinery from the RL section. The student is pushed, token by token, toward the teacher’s distribution on its own trajectories.

The reverse-KL objective has a convenient property: it is mode-seeking (it pushes the student to concentrate on what the teacher would actually do, rather than smear probability everywhere) and, because the teacher scores every token, it is hard to game. There is no sparse proxy to exploit, unlike a learned reward.

Why it matters: reasoning on the cheap

The payoff is cost. Reasoning RL is powerful but expensive: it spends enormous compute generating and grading rollouts for a thin end-of-sequence reward. On-policy distillation, when you have a capable teacher, reaches similar reasoning quality for far less. Qwen3’s report describes exactly this: an off-policy distillation warm-up followed by an on-policy phase, reaching strong math-reasoning scores at a small fraction of the compute a full RL run would need.

What distillation can and can’t teach

On-policy distillation also sharpens a debate this explainer has been circling: does optimization add capability, or just polish? A careful comparison (Kim et al., 2025) found that distillation lifts genuine capability (solving problems the student couldn’t before) only when it transfers new knowledge the student lacked. Distilling a teacher’s reasoning style alone behaves much like RLVRRLVRReinforcement Learning from Verifiable Rewards — use an automatic checker (unit tests, an answer key, a math grader) as the reward instead of a learned reward model. No reward hacking of a neural proxy.See in glossary →: it raises accuracy on problems the student could already sometimes solve, without expanding the set of solvable problems. The teacher has to actually know something the student doesn’t for distillation to push the frontier outward.

That sets up the next question directly. If imitating a stronger teacher is bounded by what the teacher knows, can reinforcement learning — with no teacher at all — push a model past its own starting boundary? That’s where we turn next.