Generative reward models

Rewards beyond verifiable domains

Papers: RM-R1: Reward Modeling as Reasoning — Chen et al., 2025 · OpenRubrics: Synthetic Rubric Generation for Reward Modeling — Liu et al., 2025

The reasoning era was powered by a lucky break: in math and code you can check the answer, so RL from verifiable rewardsRLVRReinforcement Learning from Verifiable Rewards — use an automatic checker (unit tests, an answer key, a math grader) as the reward instead of a learned reward model. No reward hacking of a neural proxy.See in glossary → sidesteps a learned reward model entirely. But most things we want from an assistant aren’t checkable. Was this medical explanation responsible? Is this essay well-argued? Did this agent’s plan make sense? There’s no unit test for “good advice.” This chapter is about the reward signal for everything outside the verifiable island, and how it’s being rebuilt to be more capable and less gameable.

The squeeze: between hackable scalars and rare verifiers

We’ve met the two ends. A learned reward modelreward model (RM)A model trained from human preference data to output a scalar score for how good a response is. Stands in for a human judge so RL can query reward millions of times.See in glossary → (the Bradley–Terry scalar) works anywhere, but it’s a black box that emits one opaque number, and optimizing it hard invites reward hacking, since the policy can find inputs where the number is wrong. A verifierverifierAn automatic, often rule-based checker that returns whether a response is correct (e.g. runs unit tests, compares to a known answer). Provides the reward in RLVR.See in glossary → is trustworthy and unhackable, but it only exists where correctness is mechanically decidable. The frontier of reward modeling is the search for something with the verifier’s robustness and the scalar RM’s generality.

Reward models that reason

The first move is to stop asking a reward model for a snap scalar and let it think first. A generative reward modelgenerative reward modelA reward model that is itself a language model: it writes out a critique or reasons step by step before scoring a response, rather than emitting a single opaque scalar. More accurate and interpretable than a bare scalar head.See in glossary → is itself a language model: given a prompt and a response, it produces a written critique (or compares two responses step by step) and only then emits a judgment. RM-R1 (2025) trains exactly this, treating reward modeling as a reasoning task: the model generates its own evaluation criteria and works through them before scoring, and the reasoning makes it both more accurate and far more interpretable than a bare scalar head. This is the same lesson as the rest of the reasoning era (letting a model use tokens as scratch space) applied to the judge instead of the policy.

Rubrics: writing the criteria down

The second move makes the standard explicit. Instead of hoping a reward model has internalized “what good looks like,” a rubric-based rewardrubric-based rewardScoring a response against an explicit written checklist of criteria instead of a black-box score. More transparent and harder to game — the policy must satisfy named requirements — and the criteria can be generated automatically.See in glossary → scores a response against a written checklist of criteria (is it correct, does it address the constraint, is it appropriately hedged?), much like the constitution in Constitutional AI, but per-task and fine-grained. OpenRubrics (2025) tackles the obvious bottleneck (writing good rubrics by hand doesn’t scale) by generating them: it derives criteria by contrasting preferred against rejected responses, then scores against those. Explicit criteria are harder to game than a black-box score (the policy has to actually satisfy named requirements) and easier to debug when the reward is wrong.

The catch is the old catch

None of this dissolves the core problem, it relocates it, exactly as AI feedback did. A generative judge is still a model with biases; a generated rubric still encodes someone’s values and can miss what matters. A more capable, more transparent judge raises the ceiling on where we can apply reward-based optimization and makes hacking harder to hide. But the reliability of the whole pipeline still rests on the judge being trustworthy enough to supervise, which remains the unfinished business of scalable oversightscalable oversightThe challenge of supervising models on tasks too hard or numerous for humans to label directly — addressed by AI feedback, critiques, and verifiers.See in glossary →. The reward signal keeps getting better; it never gets free.