Section 16

Generalized advantage estimation

One dial between noisy truth and smooth guess

The advantage is defined; now we have to measure it, and we only have two flawed instruments. We can watch what actually happened, the real rewards of the rollout. That’s honest but jittery: a single noisy outcome standing in for an expectation. Or we can ask the critic what it predicts. That’s calm, since the critic averages over everything it has seen, but it’s calm even when it’s wrong.

Weather gives the right instinct. To judge today, you can look out the window (true, but one noisy day) or read the seasonal forecast (smooth, but systematically off if the forecaster is bad). The best estimate blends both, and how far you lean toward the window is a dial, not a fixed answer. This chapter is that dial.

The unit of surprise: the TD error

The blend is built from a small, local quantity. At each step, compare what the critic predicted before the step with what the world just revealed, namely the reward received plus the (discounted) value of wherever you landed:

\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)

Read aloud: the temporal-difference error at step $t$ is the reward you just got, plus $\gamma$ times the value of where you ended up, minus the value of where you were. It’s a one-step surprise: positive when the step went better than the critic expected, negative when it went worse. ( $\gamma$ is the usual discount on the future, kept close to $1$ .)

Walk through a rollout and watch each surprise get computed, and then folded into an advantage estimate:

One surprise at a time

Step through a 6-step rollout. Each step yields one TD error δ_t, and GAE folds it into the advantage estimate with weight (γλ)^t (γ = 0.99, λ = 0.95 fixed).

steps seen 0 / 6

Press the button: the policy takes step t = 0, earns a reward, and lands somewhere new.

Surprises so far, and their weights in Â₀

t0 · r=0

×1.00

t1 · r=1

×0.94

t2 · r=0

×0.88

t3 · r=0

×0.83

t4 · r=2

×0.78

t5 · r=-1

×0.74

Running advantage estimate for the first step, Â₀ = Σ (γλ)^t · δ_t

—

Each new surprise counts a little less — by step t = 5 its weight is already ×0.74 — so distant, noisier steps can't yank the estimate around. That geometric fade is the entire GAE trick. The next widget puts γ and λ on dials so you can bend this fade yourself.

GAE: sum the surprises, with fading trust

Generalized Advantage Estimation (GAE, Schulman 2016) estimates the advantage at step $t$ by summing that step’s surprise and the surprises after it, each weighted a bit less than the one before, a geometric fade controlled by a knob $\lambda$ between $0$ and $1$ . Nearby surprises count fully; distant, noisier ones fade out. The two endpoints of the knob recover exactly our two flawed instruments:

$\lambda = 0$ : keep only the immediate surprise $\delta_t$ , leaning entirely on the critic. Low variance, but it inherits the critic’s bias.
$\lambda = 1$ : never fade, summing every surprise, which works out to the full observed return minus the baseline. Unbiased (at this end the critic only enters as a subtracted baseline, so its errors cancel in expectation), but as noisy as the rollout itself.
$\lambda \approx 0.95$ , the usual RLHF setting: keep most of the variance reduction while paying only a little bias.

The formula, and the trust-horizon reading

Written out, $\hat{A}_t^{\text{GAE}(\gamma,\lambda)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l \, \delta_{t+l}$ , with each later surprise multiplied by one more factor of $\gamma\lambda$ . Think of $(\gamma\lambda)^l$ as a trust horizon. Small $\lambda$ trusts the critic’s value estimate and only looks a step or two into the actual rewards before deferring to it: calm but slightly wrong if the critic is off. Large $\lambda$ insists on watching the real rewards play out many steps before trusting the critic: accurate in expectation but jittery. GAE lets you slide continuously between “believe the critic” and “believe what happened,” and tune for the noise level of your reward signal.

Try it: the GAE dial

Below, the same rollout as the step-through above, but with $\gamma$ and $\lambda$ on dials. Crank $\lambda$ toward 1 and the advantage estimates get spikier (high variance); pull it toward 0 and they smooth out toward the critic’s one-step view (low variance, more bias). This single picture is the heart of why PPO is stable.

GAE bias / variance dial

A fixed 6-step rollout. Set γ and λ; the bars are the GAE advantage Â_t at each step. λ slides between one-step temporal-difference (TD) and Monte-Carlo.

Discount γ = 0.99GAE λ = 0.95· λ → 1: near Monte-Carlo — low bias, high variance (trusts the noisy returns)

0 · TD1 · Monte-Carlo

Each step's temporal-difference (TD) error is δ_t = r_t + γ·V(s_{t+1}) − V(s_t). GAE sums future δ's with geometric weights (γλ)^k. At λ = 0 only the immediate δ survives — the estimate leans entirely on the (biased) value function but barely fluctuates. At λ = 1 the weights never decay, so you sum the whole noisy return: unbiased but high-variance. PPO's usual λ ≈ 0.95 keeps most of the variance reduction while adding only a little bias.

Where this is heading

Baselines, a critic, and GAE give us a low-variance advantage signal $\hat{A}_t$ . But we still haven’t fixed the other failure mode of naive policy gradients: taking a step so large that it wrecks the policy in a single update. That is the problem trust regions and clipping solve, and it brings us to the algorithm at the center of RLHF: PPO, in the next chapter.