Section 15

Value & advantage

The critic, and 'better than expected' as a signal

The baseline from the last chapter graded every response against one number: the batch’s typical reward. But “typical” depends on where you’re standing. A model asked what is two plus two almost always scores high; asked to prove a theorem, it might struggle.

Judging both against the same average punishes every honest attempt at the hard prompts and celebrates every lazy answer to the easy ones.

So a far better baseline is state-dependent: the reward we expect from this particular starting point. This chapter builds that idea into the three quantities at the heart of modern policy optimization (value, Q, and advantage), and they are all, at bottom, one intuition: surprise.

The value function: what do I usually earn from this state?

The expected reward from a given starting point has a name, the value function:

V(s) = \mathbb{E}_{y \sim \pi_\theta}\big[\, r \mid \text{state } s \,\big]

Read aloud: $V$ of a state $s$ (this prompt, these tokens so far) is the reward the current policy expects to earn from here on out, averaged over everything it might do next. A strong prompt where the model usually does well has a high $V$ ; a hard one where it might struggle has a low $V$ .

A concrete picture: the mountain biker

Watch a mountain biker. At the top of a smooth hill they’ve ridden down a hundred times successfully and only two times they fell off, the expected outcome is excellent: they ride it clean nearly every time, so the $V$ of that state is high. The state is the starting point, not the ride downhill itself. The expected outcome is the average of all the downhill attempts they’ve ridden so far.

At the start of a backflip they’ve never landed, the honest forecast is a crash more often than not: $V$ is low. The state is the starting point, not the backflip itself. The expected outcome is the average of all the backflips they’ve landed successfully so far (in this example, zero).

Notice what $V$ is not. It isn’t a grade of the rider’s overall skill, and it isn’t the outcome of any single attempt. It’s the average result of this rider starting from this spot. And it updates as the situation unfolds: halfway through the flip, when it looks like they’re about to crash again, $V$ has already collapsed, even though the crash hasn’t happened yet. A language model mid-response works the same way: a proof that just took a wrong turn is a low- $V$ state long before the final answer is graded.

We don’t know $V$ , so we learn it with a critic. The critic can be a separate model, or a value head sharing some parameters with the policy, trained by regression to predict returns: plain supervised learning riding alongside the RL. The policy (the “actor”) proposes; the critic estimates expected return. This actor–critic pattern is common in PPO, though implementations differ.

Advantage: the surprise score

Now grade a specific action against that expectation. Let $Q(s,a)$ be the expected reward of taking action $a$ in state $s$ and then continuing as usual. The advantage is the gap:

A(s, a) = Q(s, a) - V(s)

In words: how much better is this specific action than what I typically manage from this state? A positive advantage is a pleasant surprise: do more of that. A negative advantage is a disappointment: do less. Zero means “exactly as expected,” and expected behavior needs no correction.

Play with what that means in practice:

Advantage explorer

Pick a prompt, then a response. The advantage A = Q − V scores the response against what this prompt usually earns, not against zero.

“What is 2 + 2?” — the policy almost always nails this

expected V(s)

Sampled response — click one

Surprise meter

Pick a response above.

Try the Q = 6 response on the Easy prompt and then on the Hard prompt. Same score, opposite lessons: 6 − 8 = −2 (worse than this prompt deserves) versus 6 − 2 = +4 (a genuine win). That is everything the advantage adds over the raw reward: it grades on a curve set by the state.

The point to carry away from the widget: the same score is good news on a hard prompt and bad news on an easy one. Advantage grades on a curve set by the state, which is exactly the adaptive baseline we were missing.

Three quantities, one sentence each

Value $V(s)$ : the reward the policy typically earns starting from state $s$ , averaged over whatever it might do next. For an LLM, $s$ is the prompt plus the tokens written so far, and $V(s)$ is the score the finished response is expected to get from here, before choosing the next token.

Q $Q(s,a)$ : the reward it typically earns if it takes this specific action $a$ from state $s$ and then continues as usual. For an LLM, that’s committing to one particular next token (say, starting the correct step of a proof rather than a wrong turn) and then finishing the response normally.

Advantage $A = Q - V$ : how much better or worse this action is than the state’s typical outcome; positive is a pleasant surprise, negative a disappointment. For an LLM, this is per-token credit assignment: out of a whole response that got one score, it identifies which individual token choices helped and which hurt, and that per-token signal is exactly what PPO trains on.

The modern policy gradient

Swap the advantage in for the raw reward and you get the update used, in one form or another, by essentially every RL method later in this explainer:

\nabla_\theta J = \mathbb{E}\big[\, A(s, a) \, \nabla_\theta \log \pi_\theta(a \mid s) \,\big]

This is the exact same shape as the policy gradient from chapter 13 (average, over sampled actions, of each action’s “make-it-more-likely” direction) with one swap: the scalar weight is now the advantage $A(s,a)$ instead of the raw reward. That single change is the whole point. An action that beat the state’s average ( $A > 0$ ) gets pushed up; one that fell short ( $A < 0$ ) gets pushed down; and an action that was merely typical ( $A \approx 0$ ) is left almost untouched instead of being yanked around by the reward’s size.

For a terminal reward with a learned baseline, this is just $A \approx r(y) - V(s)$ : reward minus the critic’s prediction. The advantage is the cleaned-up, centered learning signal: the variance-inflating level of the reward subtracted away, leaving only the informative part.

One problem left

We’ve defined the learning signal we want, but defining it isn’t computing it. In practice the advantage must be estimated from a handful of noisy rollouts and an imperfect critic, and how you blend those two flawed sources is a genuine dial with “trust what happened” at one end and “trust the critic” at the other. That dial is Generalized Advantage Estimation, the next chapter.