Recap

The pipeline reassembled, and further reading

We started with a base modelbase modelA model straight out of pre-training — a powerful text continuator that has not yet been taught to follow instructions, hold a conversation, or refuse harmful requests.See in glossary → trained to continue text and ended at systems that can search, run code, and sometimes allocate long reasoning traces before answering. This final chapter steps back to find the through-line, the levers that explain the differences, and a reading order for the papers.

The through-line

Here is a useful through-line: post-trainingpost-trainingEverything done to a model after pre-training to turn a raw next-token predictor into a useful assistant: supervised fine-tuning, RLHF, and RL from verifiable rewards.See in glossary → changes the data, feedback, and objectives used to shape a model beyond broad next-token likelihoodlikelihoodThe probability a model assigns to observed data. Supervised fine-tuning maximizes the likelihood of human-written target responses given their prompts.See in glossary → training.

Pre-training produces a model that predicts its training distribution. Post-training can add demonstrations, comparative judgments, and automatically checked outcomes. These signals have different coverage and failure modes:

• Imitation (SFTsupervised fine-tuning (SFT)Training a pre-trained model on curated (prompt, response) pairs with the ordinary next-token objective, so it imitates demonstrated assistant behavior. The first stage of post-training.See in glossary →) upgrades the signal from “any text on the internet” to “good demonstrations of helpful behavior.”
• Preferences (RLHFRLHFReinforcement Learning from Human Feedback — train a reward model on human preference comparisons, then optimize the policy against that reward with RL (typically PPO), with a KL leash to a reference.See in glossary →, DPODPODirect Preference Optimization (Rafailov, 2023) — a closed-form supervised loss that optimizes the RLHF objective directly from preference pairs, with no separate reward model and no RL loop.See in glossary →) add comparative supervision: “this answer is better than that one” under a labeling policy.
• Verifiable rewards (RLVRRLVRReinforcement Learning from Verifiable Rewards — use an automatic checker (unit tests, an answer key, a math grader) as the reward instead of a learned reward model. No reward hacking of a neural proxy.See in glossary →, reasoning RL) use automatically checked outcomes where a verifier adequately specifies the target task.

The objective got better and better; the model’s raw next-token machinery never changed. That’s the constant. Everything else is engineering around the signal.

The recurring levers

Almost every paper in this explainer is a particular setting of four dials:

• The signal. What tells the model it did well? It marched from demonstrations (SFT) → preferencespreference dataData where humans (or an AI) compare two or more model responses to the same prompt and mark which is better. The training signal for reward models and DPO.See in glossary → (reward models) → verifiable rewards (RLVR). Each step removes a bottleneck: human demonstrations, then human comparisons, then human judgment altogether on checkable tasks.
• The optimizer. How is the signal turned into a weight update? REINFORCEREINFORCEThe basic Monte-Carlo policy-gradient estimator (Williams, 1992): scale the gradient of each action’s log-probability by the reward (or advantage) it earned. Everything else builds on it.See in glossary → → PPOPPOProximal Policy Optimization (Schulman, 2017) — nudges the policy toward higher reward in small, clipped steps with a KL leash to a reference model. The RLHF workhorse: stable, simple, widely used.See in glossary → → DPO → GRPOGRPOGroup Relative Policy Optimization (Shao, 2024) — drop PPO’s critic; sample a group of responses per prompt and use their mean reward as the baseline, giving a group-relative advantage. Memory-cheap RL that powered DeepSeek-R1.See in glossary → and its refinements. The arc here is mostly about variance and stability: better baselinesbaselineA reference value subtracted from the reward to reduce gradient variance without adding bias. Can be a learned critic, a group mean (GRPO), or a leave-one-out average (RLOO).See in glossary →, trust regions, dropping the critic, then sometimes bringing it back.
• The leash. What regularizes the update? A KL penaltyKL penaltyA term added to the RLHF reward that subtracts β times the KL divergence from the reference policy, keeping the optimized model from drifting too far while chasing reward.See in glossary → to a reference modelreference modelA frozen copy of the policy (usually the SFT model) that RLHF and DPO stay close to via a KL penalty, preventing the optimized policy from drifting into degenerate text.See in glossary → is common in PPO-style RLHF and central to DPO’s derivation, but is not present in every modern recipe.
• The eternal enemy. Reward hackingreward hackingWhen a policy finds ways to score high on the reward model without actually being better — exploiting quirks of an imperfect proxy. A central danger of RL post-training.See in glossary →: Goodhart’s law made concrete. Optimize any proxy hard enough and it stops tracking what you wanted. Reward ensembles, KL leashes, verifiable rewards, and careful eval are all, at bottom, defenses against this one failure mode.

The arc, in one paragraph each

• The instruction-tuning era. A base model isn’t an assistant; instruction tuning taught it to follow instructions, and SFT on good demonstrations (increasingly synthetic and self-generated) gave us the first usable assistants. Signal: demonstrations.
• The RLHF era. Preference-based pipelines collect comparisons, often fit a reward model, and can optimize it with PPO under a KL penalty. RLAIF and Constitutional AI used AI feedback for some judgments, while retaining human choices and, in the original CAI work, human helpfulness data. Signal: preferences.
• The DPO / offline era. Running an online RL loop is fiddly, so DPO collapsed RLHF into a single supervised loss with an implicit rewardimplicit rewardIn DPO, the reward is never trained explicitly; it is implied by the log-ratio between the policy and the reference. Optimizing the DPO loss is equivalent to RLHF under that implied reward.See in glossary →, and a whole zoo of variants (IPO, KTO, ORPO, SimPO) plus rejection-sampling methods followed. Signal: preferences, optimized offline.
• The reasoning / RLVR era. For some math and code tasks, a verifierverifierAn automatic, often rule-based checker that returns whether a response is correct (e.g. runs unit tests, compares to a known answer). Provides the reward in RLVR.See in glossary → can replace a learned reward model if its tests correctly specify the task. Bootstrapped reasoning, process rewards, test-time compute (o1), and GRPO / DeepSeek-R1 made this a major direction in reasoning research. Signal: mechanically checked outcomes where available.
• The agentic frontier. The unit of optimization moved from the response to the whole trajectorytrajectoryThe sequence of states and actions in a rollout. For text generation, the tokens generated one after another, each conditioned on those before it.See in glossary →: agentic and tool-use RL, where the model acts, observes, and acts again, and credit must be assigned across many turns. Signal: outcomes of multi-step interaction.
• The moving edge. Beyond the core recipe, a few fast-moving threads: on-policy distillation (reach reasoning-RL quality far more cheaply with a dense teacher signal on the student’s own rollouts), RL scaling laws (RL becoming a predictable, plannable compute lever, and the live debate over whether it expands a model’s boundary or only sharpens it), and generative and rubric-based reward models (carrying a verifier’s robustness into tasks with no answer key), and GLM-5.2’s long-horizon production recipe (critic-based PPO over compaction-split trajectories, online anti-hacking guards, and parallel on-policy distillation). Signal: still being invented.

The pipeline, reassembled

Here is the whole thing as one map. Every node is a stage we built up across the explainer. Click through to revisit any of them. Read left to right and you’re reading the standard 2026 recipe: a base model, made into an assistant by SFT, polished by preference optimization, and sharpened by verifiable-reward RL.

Further reading: the papers, in order

The instruction-tuning and SFT roots:

• Wei et al., Finetuned Language Models Are Zero-Shot Learners (FLAN, 2021) — arxiv 2109.01652.
• Wang et al., Self-Instruct (2022) — arxiv 2212.10560. Bootstrapped instruction data.

The RLHF lineage:

• Christiano et al., Deep RL from Human Preferences (2017) — arxiv 1706.03741. The founding RLHF paper.
• Ziegler et al., Fine-Tuning LMs from Human Preferences (2019) — arxiv 1909.08593. The KL-to-reference penalty.
• Ouyang et al., Training LMs to Follow Instructions with Human Feedback (InstructGPT, 2022) — arxiv 2203.02155. The SFT → RM → PPO recipe.
• Bai et al., Constitutional AI (2022) — arxiv 2212.08073. RLAIF and scalable oversight.

The optimization algorithms:

• Schulman et al., Proximal Policy Optimization (2017) — arxiv 1707.06347. The RLHF workhorse.
• Rafailov et al., Direct Preference Optimization (DPO, 2023) — arxiv 2305.18290. RLHF as one supervised step.
• Shao et al., DeepSeekMath (GRPO, 2024) — arxiv 2402.03300. The critic-free group baseline.
• Yu et al., DAPO (2025) — arxiv 2503.14476. The open GRPO refinement.

The reasoning / RLVR era:

• Lightman et al., Let’s Verify Step by Step (2023) — arxiv 2305.20050. Process reward models.
• OpenAI, o1 (2024). RL-trained long chain-of-thought; test-time compute.
• DeepSeek-AI, DeepSeek-R1 (2025) — arxiv 2501.12948. Pure-RL reasoning and the “aha moment.”
• Lambert et al., Tülu 3 (2024) — arxiv 2411.15124. The open post-training reference manual.

The agentic frontier:

• Zhang et al., The Landscape of Agentic RL for LLMs: A Survey (2025) — arxiv 2509.02547.

The moving edge:

• Agarwal et al., On-Policy Distillation of Language Models (GKD, 2023) — arxiv 2306.13649. The student samples; the teacher grades every token.
• Liu et al., ProRL (2025) — arxiv 2505.24864. Prolonged RL expands the reasoning boundary.
• Khatri et al., The Art of Scaling RL Compute (2025) — arxiv 2510.13786. Predictable RL compute-scaling curves.
• Chen et al., RM-R1: Reward Modeling as Reasoning (2025) — arxiv 2505.02387. Reward models that reason before they score.
• Z.ai, GLM-5.2: Built for Long-Horizon Tasks (2026) — blog post. Critic-based PPO for long horizons, anti-hacking, and parallel OPD in production.