Rejection-sampling alignment

RAFT, RRHF, and best-of-N fine-tuning

Paper: RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment — Dong et al., 2023

Here is a simple post-training pattern: ask the model to generate several answers, keep high-scoring ones, and fine-tune on them. Repeat if useful. It avoids the policy-gradient loop of PPOPPOProximal Policy Optimization (Schulman, 2017) — nudges the policy toward higher reward in small, clipped steps with a KL leash to a reference model. The RLHF workhorse: stable, simple, widely used.See in glossary → and is a component of several reported post-training pipelines, but it is not present in every frontier model.

Generate, filter, fine-tune

The recipe goes by a few names but the loop is always the same:

1. For each prompt, sample $N$ candidate answers from the current policy.
2. Score each candidate with some reward signal: a reward modelreward model (RM)A model trained from human preference data to output a scalar score for how good a response is. Stands in for a human judge so RL can query reward millions of times.See in glossary →, a human rating, or, as we’ll see, a verifierverifierAn automatic, often rule-based checker that returns whether a response is correct (e.g. runs unit tests, compares to a known answer). Provides the reward in RLVR.See in glossary →.
3. Keep the best (the top-scoring one, or all that clear a threshold) and fine-tune the model on those winners with a plain supervised cross-entropycross-entropy lossA classification loss that penalizes the model according to the negative log-probability it assigned to the correct answer.See in glossary → loss.
4. Iterate: the improved model generates better candidates next round.

This is rejection samplingrejection samplingGenerate several candidate responses, keep only the best-scoring one(s) by some reward or verifier, and fine-tune on those. A simple, stable, RL-free way to improve a model.See in glossary →: a classical statistical idea (propose samples, accept the good ones, reject the rest) repurposed for alignment. Keeping the single best of $N$ candidates is exactly best-of-Nbest-of-NSampling N responses and selecting the highest-reward one. Used both at inference time and as the data-generation step in rejection-sampling fine-tuning.See in glossary → sampling; the twist here is that instead of just returning the best answer at inference time, you train on it, baking the best-of-N behavior back into the weights.

RAFTRAFTReward-rAnked Fine-Tuning — iteratively sample, rank by reward, and fine-tune on the top responses. Offline, RL-free preference alignment.See in glossary → (Reward-rAnked FineTuning, Dong et al. 2023) is the clean formulation of this loop: sample a batch, rank by reward, fine-tune on the top-ranked, repeat. RRHF (Yuan et al. 2023) is a close cousin from the same year: it scores a set of candidates and uses a ranking loss to make the model prefer the higher-reward ones, blending the rejection-sampling idea with a contrast between candidates.

Why it’s so appealing

Rejection-sampling alignment is offlineoffline RLOptimizing from a fixed dataset of responses and preferences without generating new rollouts during training. DPO and rejection-sampling methods are offline.See in glossary → in the same sense DPO is (each round trains on a frozen batch of pre-scored samples) but it’s even more bare-bones, and the simplicity buys real advantages:

• Dead simple. It’s just sampling plus standard supervised fine-tuning. Any team that can run generation and an SFT job can do it; there’s no new optimizer, no special loss to debug.
• Operationally simple. There’s no importance samplingimportance samplingReweighting samples from one distribution to estimate expectations under another, via the probability ratio π_new/π_old. The ratio PPO clips comes from here.See in glossary →, clipped surrogateclipped surrogate objectivePPO’s loss: maximize the probability-ratio-weighted advantage, but clip the ratio to [1−ε, 1+ε] so a single update can’t move the policy too far.See in glossary →, or criticcriticA model trained to predict the value function. PPO uses an actor (the policy) and a critic; GRPO drops the critic and uses a group average instead.See in glossary →. Ordinary SFT can still overfit, degrade capabilities, or amplify a flawed scorer, so simplicity is not a guarantee of safety or improvement.
• Trivially parallel and reusable. Generation is embarrassingly parallel, the filtered data is just text you can inspect, mix, dedup, and reuse, and you can throw a stronger reward model at it later without changing the training code.

The contrast with the methods around it is sharp. Against PPO, rejection sampling drops the entire RL apparatus (no clipping, no critic, no KLKL penaltyA term added to the RLHF reward that subtracts β times the KL divergence from the reference policy, keeping the optimized model from drifting too far while chasing reward.See in glossary →-penalized policy gradient) at the cost of being off-policy and only as good as your best-of-N samples. Against DPO, the difference is what signal it consumes: DPO needs pairwise preferences and learns from both the chosen and the rejected; rejection sampling needs only a score, and in its basic form learns from the positives only (or best-vs-rest), never explicitly pushing down the losers. Less information per example, but a much easier signal to obtain.

The bridge to reasoning

Now change one thing about the recipe. In everything above, the “reward” was a learned reward model: soft, fuzzy, and potentially exploitable. But suppose the task is math or code, where you can check the answer. The filter can be a verifierverifierAn automatic, often rule-based checker that returns whether a response is correct (e.g. runs unit tests, compares to a known answer). Provides the reward in RLVR.See in glossary → that returns a mechanically checked correct/incorrect signal. Its reliability still depends on the specification, answer extraction, and execution environment.

With a verifier as the filter, “sample N, keep the ones that got the right answer, fine-tune, repeat” becomes a method for bootstrapping reasoning. The model generates many chains of thoughtchain-of-thought (CoT)Having a model write out intermediate reasoning steps before its final answer. Improves accuracy on multi-step problems and is the substrate reasoning RL optimizes.See in glossary →, you keep only the ones that reached the verified answer, and you fine-tune on those correct reasoning traces. The model learns to reason better by training on its own successful reasoning. This is exactly the idea behind STaRSTaRSelf-Taught Reasoner (Zelikman, 2022) — generate chain-of-thought rationales, keep those that reach the correct answer, fine-tune on them, and repeat. Bootstraps reasoning from a model’s own correct attempts.See in glossary → (Self-Taught Reasoner), the subject of the next chapter: rejection-sampling fine-tuning with a verifier in the loop.

And it doesn’t stop there. Keep that same verifier but, instead of fine-tuning on filtered samples once per round, optimize against it online with a policy-gradient method, and you’ve arrived at RL from verifiable rewardsRLVRReinforcement Learning from Verifiable Rewards — use an automatic checker (unit tests, an answer key, a math grader) as the reward instead of a learned reward model. No reward hacking of a neural proxy.See in glossary →, the engine behind DeepSeek-R1 and the modern reasoning models, which we reach in chapter 26. Rejection sampling is the gateway: the same generate-and-filter skeleton, with the reward swapped for a verifier and the fine-tune swapped for an RL update.

That’s the thread for the rest of this explainer. We came into Section 5 trying to avoid an exploitable reward-model loop, and we leave it having found the simplest possible alternative, and discovered that, with the reward replaced by a verifier, that same simple idea is the on-ramp to reasoning. Section 6 takes the on-ramp.