Section 27

GRPO & DeepSeek-R1

Group-relative advantage and critic-free RL

Papers: DeepSeekMath (GRPO) — Shao et al., 2024 · DeepSeek-R1 — DeepSeek-AI, 2025

This chapter brings together an algorithm for critic-free policy optimization (GRPO) and a prominent open reasoning report, DeepSeek-R1. GRPO can be paired with learned or rule-based rewards; DeepSeek-R1-Zero used rule-based rewards on selected reasoning tasks. Neither report provides a complete reproduction of OpenAI o1’s undisclosed pipeline.

GRPO: throw away the critic

Recall PPO from chapter 17. To compute the advantage (“was this action better or worse than expected?”), PPO trains a second network alongside the policy: a value function that estimates expected return at each token. That critic is roughly the same size as the policy, must be trained in lockstep, and doubles the memory and a big chunk of the compute of the whole RL run. For a giant LLM, the critic is a brutal tax.

GRPO (Group Relative Policy Optimization), introduced by Shao et al. (2024) in DeepSeekMath, asks: what if we just delete the critic? The critic only ever existed to provide a baseline: a reference point to subtract from each reward so we know whether a sample was above or below average. GRPO gets that baseline a completely different way: sampling.

For each prompt, GRPO samples a group of $G$ completions from the current policy. Each completion $i$ receives a reward $r_i$ from a reward model or rule-based verifier. The group mean supplies a baseline. For outcome supervision, the group-relative advantage is the centered reward normalized by the group’s spread:

A_i = \frac{r_i - \operatorname{mean}(r_1, \ldots, r_G)}{\operatorname{std}(r_1, \ldots, r_G)}

That’s the whole trick. A completion that scored above the group average gets a positive advantage and is reinforced; one below average gets a negative advantage and is suppressed. The group is its own baseline. No value network, ever.

The GRPO objective

GRPO keeps PPO’s stabilizers. It’s PPO with the advantage swapped out. For each token in completion $i$ , let $\rho_{i,t}$ be the importance ratio (new-policy probability over old-policy probability for that token). The objective is the familiar clipped surrogate, using the group-relative advantage $A_i$ , plus a KL penalty to a frozen reference model:

\mathcal{J}_{\text{GRPO}} = \mathbb{E}\!\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t} \min\!\big(\rho_{i,t}\,A_i,\; \operatorname{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\,A_i\big)\right] - \beta\, D_{\text{KL}}\!\left(\pi_\theta \,\|\, \pi_{\text{ref}}\right)

The $\min$ /clip is exactly PPO’s trust-region trick: don’t let the policy move too far in one step. The KL term is the leash to the reference model that keeps the policy from drifting into degenerate text. The only structural change from PPO is that $A_i$ comes from the group mean instead of a critic, and that single change removes an entire network from the training loop.

Try it

Below, sample a group of completions, see each one’s verifier reward, watch the group mean become the baseline, and see the resulting normalized advantages: positive for above-average samples, negative for below. Toggle the critic on and off to feel what GRPO removes.

GRPO: group-relative advantage

Sample a group of completions for one prompt. The baseline is the group's own mean reward — no critic needed.

Prompt

Prove that the sum of the first n odd numbers is n².

Group size N = 6

completion · reward → advantagebaseline 0.67 (mean)

induction, base + step ✓

A = -1.41

visual L-shapes argument ✓

A = -1.41

algebraic sum, off-by-one ✗

A = +0.71

restates claim, no proof ✗

A = +0.71

telescoping sum ✓

A = +0.71

induction, missing base case ✗

A = +0.71

Group mean (baseline)

0.667

Group std

0.471

Critic network

none

GRPO drops PPO's critic. Instead of training a separate value network to predict a baseline, it samples a group of completions for the same prompt and uses the group's own mean reward as the baseline; each completion's advantage is (reward − mean) / std. Completions above the group average get pushed up (teal), those below get pushed down (rose). No value network means roughly half the model memory of PPO — the trick behind DeepSeek-R1. Flip to PPO to see the same update driven by a learned, imperfect critic estimate that costs an entire extra network to train.

DeepSeek-R1-Zero: reasoning from nothing

In January 2025, DeepSeek-AI reported DeepSeek-R1-Zero. It took DeepSeek-V3-Base, a pre-trained base model, and applied RL without an SFT cold start. For math, coding, and logic, it used rule-based accuracy rewards plus a format reward requiring reasoning tags; it did not use a learned outcome or process reward model for those reasoning tasks.

The report observed chains of thought growing longer during training, plus reflective and alternative-solution behaviors. These are empirical behaviors reported by DeepSeek, not proof that the model acquired a human-like general reasoning process. On AIME 2024, the report shows R1-Zero rising from 15.6% to 77.9% pass@1; self-consistency decoding reached 86.7% in the reported evaluation. These are report-specific benchmark results, not a general guarantee from pure RL.

The aha moment

The DeepSeek team documented a now-legendary training artifact: mid-trajectory, R1-Zero would literally write something like “Wait, wait. That’s an aha moment. Let me re-evaluate this step by step.”, and then go back and fix its own reasoning. The model wasn’t told to second-guess itself; it discovered that pausing to reconsider earns reward. This is the aha moment: emergent self-correction, arising spontaneously from reinforcement learning against a correctness signal. It is, as close as anything we have, a glimpse of a model learning how to think rather than being shown what to say, and it is the most quoted result of the entire reasoning era.

DeepSeek-R1: cold-start for readability

R1-Zero proved the principle, but it had warts. Pure-RL reasoning traces were powerful yet unreadable: mixed languages, chaotic formatting, chains that worked but no human would want to read. So the full DeepSeek-R1 added a small amount of supervised cold-start data: a curated set of clean, well-formatted long chain-of-thought examples used to fine-tune the base model before the GRPO stage. Cold-start gave the model a readable, well-behaved starting point; GRPO then drove its reasoning ability up from there, followed by a final alignment pass for helpfulness and safety.

The report compared DeepSeek-R1 competitively with o1 on several reasoning benchmarks, while results varied by benchmark. DeepSeek released weights and a detailed technical report, but the o1 training recipe remains undisclosed, so R1 should not be described as a full reproduction of it.

The whole arc, reassembled

Stand back and look at the relationship across this section. STaR used correctness as a filter. PRM/ORM work sharpened the credit-assignment question. o1 highlighted test-time compute, while DeepSeek-R1-Zero demonstrated that a base model plus group-relative RL and rule-based rewards can develop longer, more reflective traces on the reported tasks. Verifiers are useful only insofar as their specifications are reliable, and these results do not establish a universal route to reasoning.

This is the climax of the reinforcement-learning story this explainer has been building since the preference era. We began by imitating good answers (SFT), moved to learning human preferences (RLHF), and arrive here: a model improving its own thinking against the bedrock of verifiable truth, inventing strategies no one taught it. The remaining chapters refine the algorithm (DAPO, Dr.GRPO, and friends), scale the recipe across the open ecosystem, and push it into agentic, tool-using, multi-turn settings. But the conceptual summit is this one: the moment reasoning stopped being something we wrote into a model, and became something a model could learn for itself.