What is post-training?

Turning a base model into an assistant

Sources: Training language models to follow instructions with human feedback — Ouyang et al., 2022; Direct Preference Optimization — Rafailov et al., 2023; DeepSeek-R1 — DeepSeek-AI, 2025

Ask a freshly pre-trained language model “What is the capital of France?” and you might get this back:

What is the capital of France? What is the largest city in France? What is the official language of France? List three famous French painters.

It didn’t answer. It did something stranger and, on reflection, completely logical: it continued the document. Somewhere in its training data lived a quiz, a worksheet, a list of trivia questions, and the single most likely thing to follow one question is another question. The model did exactly what it was built to do. It is a brilliant autocomplete. It is also a terrible assistant.

This explainer is about closing that gap. A chat product combines a post-trained model with product-level components such as tool runtimes and safety systems. Post-training is the phase that teaches the model to answer questions, follow instructions, and prefer particular response styles rather than merely continue text. That phase is post-trainingpost-trainingEverything done to a model after pre-training to turn a raw next-token predictor into a useful assistant: supervised fine-tuning, RLHF, and RL from verifiable rewards.See in glossary →, and it is the whole subject of this explainer.

Two models, one set of weights

Pre-training, the subject of the sibling explainer, produces a base modelbase modelA model straight out of pre-training — a powerful text continuator that has not yet been taught to follow instructions, hold a conversation, or refuse harmful requests.See in glossary →: a network trained on large curated collections of text and other data to do one thing, predict the next tokennext-token predictionThe pre-training objective for GPT-style models: given the tokens so far, predict a probability distribution over the next token. Also called causal or autoregressive language modeling.See in glossary →. That objective encourages it to absorb grammar, facts, code, and reasoning patterns, because all of those help it guess what comes next. The result is a model with broad learned capabilities but no training signal that specifically says how to respond to a user.

The reason is subtle but important. The base model’s training distribution is the broad mix pre-training draws on: web pages, books, code, and other primary sources. Very little of that text is a helpful assistant responding to a user. It is articles, forum flame wars, half-finished code, published prose, and, yes, trivia worksheets. When you prompt the base model, it doesn’t ask “how would a helpful assistant respond?” It asks “what is the most likely continuation of this text?” And that continuation is often unhelpful, repetitive, or actively wrong, because plenty of unhelpful, repetitive, wrong text exists.

Post-training often reshapes how existing capabilities are expressed, though it can also teach new facts, skills, and tool-use conventions from its own data. The capital of France may already be represented in a base model; what can be missing is the disposition to answer directly, in a helpful tone, while declining requests it should not honor. Full fine-tuning updates the same parameters set during pre-training, while parameter-efficient methods can train adaptersadapterA small set of extra parameters trained on top of a frozen base model, so fine-tuning updates only the adapter rather than the full network. Adapters cut memory and storage cost and can be swapped in and out per task.See in glossary → instead. Both usually use far less data than pre-training, but their cost and ease of iteration depend on the goals and method.

The boundary between the two phases isn’t perfectly sharp. Many modern pipelines insert a mid-trainingmid-trainingA phase between the main pre-training run and post-training, used to inject specialized data or capabilities (e.g. long context, code-from-execution) while still training the base model on a next-token-style objective.See in glossary → phase in between: continued next-token training on a curated, capability-dense mixture (math, code, reasoning traces, long-context data) that still looks like pre-training in mechanism, covered in the pre-training explainer. This explainer picks up after that, with a base model in hand and the question of how to turn it into an assistant.

A note on cost

It’s tempting to call post-training “the cheap part” and pre-training “the expensive part.” Resist it. Pre-training is a massive one-time compute bill: a frontier run can cost tens of millions of dollars in GPU time. But modern post-training is far from trivial: it involves collecting human preference labels at scale, training auxiliary reward models, and running reinforcement-learning loops that repeatedly sample from the model. The reasoning-focused RL pipelines we’ll meet later can burn enormous amounts of inference compute generating and grading rollouts.

The honest framing is that the two phases trade in different currencies. Pre-training spends raw FLOPs to build knowledge. Post-training spends a mix of human labor, careful data curation, and a different shape of compute to build behavior. Neither is categorically “the expensive one.” It depends on the model, the goals, and the year.

The post-training stack

Here is the whole arc at a glance. A base model goes through some subset of these stages, in roughly this order:

1. Supervised fine-tuning (SFT) / instruction tuning. Show the model high-quality examples of instructions paired with good responses, and continue next-token training on those examples, usually computing loss on the assistant-response tokens. This teaches the model the format of being an assistant: that a user turn should be followed by a helpful answer, not another question. We cover it in Section 2.

2. Preference optimization. SFT learns from demonstrations, while preference data provides a relative signal about which of two generated sample responses is better. This is the heart of RLHFRLHFReinforcement Learning from Human Feedback — train a reward model on human preference comparisons, then optimize the policy against that reward with RL (typically PPO), with a KL leash to a reference.See in glossary → (reinforcement learning from human feedback), including some techniques we will cover to be able to achieve this at scale, without requiring a massive amount of human labeled data.

3. Reinforcement learning from verifiable rewards (RLVR) / reasoning RL. For tasks where correctness can be checked automatically (math with a known answer or code that must pass tests), we can use a verifier rather than a learned reward model. Optimizing such signals with specialized reinforcement-learning algorithms has been an important ingredient in recent reasoning-model pipelines, alongside data, prompting, and other training choices.

The map below makes this navigable. Each node is a stage; click through to see how they connect.

Three eras, briefly

It helps to see how this stack assembled itself historically, because each layer was a response to the limits of the last.

• 2021–2022: instruction tuning and RLHF. FLAN and T0 showed that fine-tuning on instructions phrased in natural language can improve zero-shot task generalization. InstructGPT established the influential three-step recipe (SFT, then a reward model, then reinforcement-learning optimization against it) that later informed many chat-model pipelines. Alignment became a training problem, not just a prompting trick.

• 2023: offline and direct methods. RLHF’s RL loop can be finicky and expensive to run. A direct, offline method showed that, under a particular preference-modeling formulation, a loss on preference pairs can optimize a policy without an explicit reward model or on-policy rollouts. A wave of variants followed, broadening the use of offline preference optimization.

• 2024–2026: verifiable rewards and reasoning. OpenAI’s o1 highlighted the impact of reinforcement learning on reasoning, though its full training recipe is not public. DeepSeek-R1-Zero demonstrated reasoning behavior emerging from pure RL, while DeepSeek-R1 combined cold-start data and multi-stage RL. Verifiable rewards are now an important ingredient in reasoning and agentic RL research.

What this explainer covers

We build the stack from the bottom up. After this on-ramp, the next three chapters lay the conceptual foundation: the probability-and-optimization toolkit every method reuses, the three quantities that toolkit assembles into (likelihood, KL divergence, entropy), and the alignment problem that motivates moving beyond imitation in the first place.

From there: instruction tuning and SFT (Section 2), the RLHF preference era and reward models (Section 3), the reinforcement-learning fundamentals that make RLHF work (Section 4), the offline and direct-preference methods (Section 5), the RLVR and reasoning era including DeepSeek-R1 (Section 6), and finally the modern algorithms and agentic frontier (Section 7).

Throughout, we care about both halves of the craft: the machine-learning theory (why optimizing a preference signal is different from imitating a demonstration) and the practice, the data, the instabilities, the failure modes that make post-training as much engineering as science. Let’s start where pre-training left off: with the probability-and-optimization vocabulary every later method is built from.