GRPO refinements

DAPO, Dr.GRPO, VAPO, RLOO, REINFORCE++

Papers: DAPO: An Open-Source LLM Reinforcement Learning System at Scale — Yu et al., 2025 · Understanding R1-Zero-Like Training (Dr. GRPO) — Liu et al., 2025 · VAPO — Yue et al., 2025 · RLOO — Ahmadian et al., 2024 · REINFORCE++ — Hu et al., 2025

GRPOGRPOGroup Relative Policy Optimization (Shao, 2024) — drop PPO’s critic; sample a group of responses per prompt and use their mean reward as the baseline, giving a group-relative advantage. Memory-cheap RL that powered DeepSeek-R1.See in glossary → became a widely used starting point for open reasoning RL. Reproduction efforts reported issues including entropy collapse, response-length growth, reward noise, and sensitivity to implementation choices. The first half of 2025 produced a flurry of proposed refinements; their results are informative but are not a single settled diagnosis of GRPO.

DAPO: GRPO with four sharp edges filed off

The most complete public fix is DAPODAPOA fully open GRPO refinement (ByteDance/Tsinghua, 2025) combining Clip-Higher, dynamic sampling, token-level loss, and overlong-reward shaping to stabilize large-scale reasoning RL.See in glossary → (Yu et al., 2025), from ByteDance and Tsinghua. And crucially, it shipped as a fully open system: code, data, and the exact recipe, where DeepSeek had described R1 but not handed over the training stack. DAPO is GRPO plus four targeted tricks, each aimed at a specific failure mode the authors hit when scaling long chain-of-thoughtlong chain-of-thoughtExtended internal reasoning — thousands of tokens of self-correction, backtracking, and exploration — that reasoning-RL elicits and that test-time scaling rewards.See in glossary → RL.

• Clip-Higher. Standard PPOPPOProximal Policy Optimization (Schulman, 2017) — nudges the policy toward higher reward in small, clipped steps with a KL leash to a reference model. The RLHF workhorse: stable, simple, widely used.See in glossary →/GRPO uses a symmetric clipped surrogateclipped surrogate objectivePPO’s loss: maximize the probability-ratio-weighted advantage, but clip the ratio to [1−ε, 1+ε] so a single update can’t move the policy too far.See in glossary → with the same $\epsilon$ on both sides. The problem: the lower clip silently caps how much probability mass a rare but good token can gain, so exploration dies and entropy collapses. DAPO decouples the bounds (a larger upper clip $\epsilon_{\text{high}}$), letting promising low-probability tokens grow:

• Dynamic Sampling. When every completion in a group is right (or every one is wrong), the group’s reward variance is zero, so the group-relative advantagegroup-relative advantageGRPO’s advantage estimate: a response’s reward minus the mean reward of its group of siblings (often divided by their standard deviation), replacing a learned value function.See in glossary → is zero for all of them: that prompt contributes no gradient. DAPO filters these out and keeps sampling until the batch is full of prompts that actually carry signal, which both speeds convergence and stops the effective batch from quietly shrinking.
• Token-Level loss. GRPO normalizes the loss per sample, which means a 4,000-token answer and a 40-token answer get the same total weight, so individual tokens in long answers are under-counted, and long-form reasoning is poorly shaped. DAPO averages over tokens across the whole batch, giving every token equal pull regardless of which response it lived in.
• Overlong Reward Shaping. Truncated answers (cut off at the length limit) would otherwise be scored as wrong, teaching the model that being long is bad rather than that rambling is bad. DAPO adds a soft, length-aware penalty near the limit instead of a hard truncation cliff: a form of length-rewardlength / format rewardAuxiliary reward terms that shape output length or enforce a required format (e.g. putting reasoning in tags, the answer in a box) — used to keep reasoning-RL outputs usable.See in glossary → shaping that controls verbosity without punishing legitimately long reasoning.

With these components, the DAPO paper reports 50 points on AIME 2024 with Qwen2.5-32B using 50% of the training steps of its cited prior baseline. This is a paper-specific experimental comparison, not a universal sample-efficiency result.

Dr. GRPO: the bias hiding in the denominator

Dr. GRPODr.GRPOA corrected GRPO that removes length and standard-deviation normalization biases, so the gradient is unbiased and long wrong answers aren’t implicitly favored.See in glossary → (Liu et al., 2025) argues that response-length and within-group standard-deviation normalization introduce optimization biases. In its analysis, per-response length normalization weakens negative updates on longer incorrect responses, while per-group standardization reweights prompts by reward dispersion. Dr. GRPO removes those normalizers to recover its stated unbiased estimator. This is a specific theoretical and empirical proposal; the choice of normalization remains an active implementation question.

VAPO: bringing the critic back

GRPO’s headline selling point was killing the criticcriticA model trained to predict the value function. PPO uses an actor (the policy) and a critic; GRPO drops the critic and uses a group average instead.See in glossary →: no value functionvalue functionThe expected return from a given state under the current policy. A learned value function (the critic) provides a baseline that reduces the variance of policy-gradient updates.See in glossary → to train, half the memory. VAPOVAPOValue-Augmented PPO (2025) — brings a well-trained critic back for long chain-of-thought RL, building on DAPO’s tricks to beat critic-free methods on reasoning.See in glossary → (Yue et al., 2025) argues that for long chains of thought, that was a false economy. With rewards arriving only at the end of a 10,000-token answer, the group-mean baselinebaselineA reference value subtracted from the reward to reduce gradient variance without adding bias. Can be a learned critic, a group mean (GRPO), or a leave-one-out average (RLOO).See in glossary → is a blunt instrument; a trained value function gives a far lower-variance, per-token advantageadvantageHow much better an action was than the baseline expectation: A = reward − value. Positive advantage pushes an action’s probability up, negative pushes it down.See in glossary → via GAEGAEGeneralized Advantage Estimation — a way to trade bias against variance in advantage estimates using a decay parameter λ. The standard advantage signal inside PPO.See in glossary →, which matters most exactly when the horizon is long. VAPO is value-augmented PPO built on DAPO’s tricks, and it edged out DAPO on the same AIME benchmark: evidence that the critic-free era is a pendulum, not a one-way door.

RLOO and REINFORCE++: maybe PPO was overkill

A parallel thread asks whether we needed PPO’s clipping and critics at all. RLOORLOOREINFORCE Leave-One-Out — use the average reward of the other samples in a group as each sample’s baseline. A simple, critic-free policy-gradient method for LLMs.See in glossary → (REINFORCE Leave-One-Out; Ahmadian et al., 2024) points out that for LLMs (where a full response is one action and rewards are deterministic given the prompt), plain REINFORCEREINFORCEThe basic Monte-Carlo policy-gradient estimator (Williams, 1992): scale the gradient of each action’s log-probability by the reward (or advantage) it earned. Everything else builds on it.See in glossary → with a good baseline is enough. Its baseline is elegantly simple: for each of the $k$ sampled completions, use the mean reward of the other $k-1$ as that sample’s baseline. That leave-one-out estimate is unbiased and needs no critic: essentially GRPO’s group baseline, derived independently from first principles.

REINFORCE++REINFORCE++A critic-free baseline that adds PPO-style stabilizers (token-level KL, clipping) to plain REINFORCE, aiming for robustness without a value network.See in glossary → (Hu et al., 2025) is another critic-free proposal. Its central change is global, rather than prompt-local, advantage normalization; its group-sampling variant combines a group baseline with global normalization and a separate KL treatment. It is presented as a more stable alternative to local-normalization methods, with evidence limited to its reported experiments.

Try it

The knobs below are DAPO’s four tricks on a toy long-CoT training run. Toggle Clip-Higher, Dynamic Sampling, Token-Level loss, and Overlong Shaping on and off, and watch the effect on the reward curve, the entropy, and the average response length. The default GRPO baseline (everything off) shows the classic failure: reward stalls while entropy collapses and length balloons. Turn the tricks on one at a time to see which symptom each one treats.

A few things to notice as you experiment:

• Clip-Higher is the entropy medicine: turn it off and watch the entropy line nose-dive as exploration dies.
• Overlong Shaping and Token-Level loss are the length medicine: without them, average response length drifts up without a matching reward gain, the visual signature of length hacking.
• Dynamic Sampling mostly buys speed: the reward curve climbs faster because no step is wasted on zero-variance groups.

No single trick is magic; the point of the widget is that they’re complementary, each closing one specific leak.

Where this leaves us

The algorithm zoo has gotten crowded (DAPO, Dr. GRPO, VAPO, RLOO, REINFORCE++), but they’re variations on a now-stable theme: a group baseline (or a revived critic) for the advantage, asymmetric clipping and KL for stability, and length-aware shaping to keep answers honest. With the optimizer mostly settled, the action moves to assembling whole pipelines at scale: how the open ecosystem stitches SFT, preference optimization, and verifiable-reward RL into a single reproducible recipe. That’s the next chapter.