RL scaling laws
How far does RL compute take reasoning?
Papers: ProRL: Prolonged RL Expands Reasoning Boundaries — Liu et al. (NVIDIA), 2025 · The Art of Scaling RL Compute for LLMs — Khatri et al., 2025 · RL vs. Distillation — Kim et al., 2025
Pre-training has scaling laws: spend more compute, in the right proportions, and loss falls along a predictable curve. That predictability is what lets labs commit hundreds of millions of dollars to a single run. Reinforcement learning has had no such map. RL has been famously finicky, and it was an open question whether pouring more compute into it kept paying off or just wasted GPUs. Two strands of 2025 work begin to answer that, and they bear directly on a question we’ve been circling: does RL make a model better, or just more consistent?
Does RL expand the boundary, or only sharpen it?
There’s a deflationary story about RL from verifiable rewardsRLVRReinforcement Learning from Verifiable Rewards — use an automatic checker (unit tests, an answer key, a math grader) as the reward instead of a learned reward model. No reward hacking of a neural proxy.See in glossary →, and it has real evidence behind it. Measure a model with pass@k: the chance that at least one of sampled answers is correct. RL reliably lifts pass@1 (the model’s single best guess gets better), but several studies found it often doesn’t lift pass@k at large . The interpretation: RL isn’t teaching new solutions, it’s just concentrating probability onto solutions the base model could already stumble onto occasionally. It sharpens; it doesn’t expand. By that account, the reasoning was latent in the base model all along, and RL only made it reliable.
The counter-evidence: prolonged RL
That deflationary picture may be an artifact of stopping too early. ProRL (NVIDIA, 2025) argues that with enough training — thousands of RL steps, plus stability machinery to keep the policy from collapsing (entropy control, periodic reference resets, the kind of fixes from the refinements chapter) — RL does expand the boundary. Their prolonged-RL models beat the base model across a wide range of pass@k, including on tasks where the base model scored essentially zero no matter how many samples you drew. If the base model can never produce a correct answer, RL can’t merely be reweighting existing samples: it has found something genuinely new.
The honest reading is that both can be true: short RL runs mostly sharpen, and the “RL only reweights” result is real for that regime; but sustained, stable RL on the right problems can push past the base model’s reach. How long and how stably you train turns out to matter as much as the algorithm.
RL, made predictable
The deeper contribution of 2025 is treating RL itself as a scaling problem. The Art of Scaling RL Compute (Khatri et al., 2025) ran a systematic ~400,000-GPU-hour study and found that RL performance follows a clean sigmoidal curve in compute: a predictable rise toward an asymptotic ceiling. Crucially, the curve is extrapolatable (you can fit it on small runs and forecast where a big one lands), and the design choices we’ve met (the loss form, advantage normalization, batch construction) mostly move the efficiency and the ceiling rather than the shape. They distill the best choices into a recipe, ScaleRL, whose scaling matches the predictability long taken for granted in pre-training.
RL scaling lawsRL scaling lawsEmpirical curves predicting how RL post-training performance grows with compute — the RL analogue of pre-training scaling laws. Recent work fits a sigmoidal curve that can be extrapolated from small runs.See in glossary → are young and far less settled than their pre-training cousins: the curves depend heavily on the task, the verifier, and stability tricks, and the field is actively arguing about how far the boundary really moves. But the trajectory is clear: RL is graduating from a temperamental finishing step into a compute lever you can plan around.