From next-token to behavior

Likelihood, KL divergence, and entropy

Sources: Training language models to follow instructions with human feedback — Ouyang et al., 2022; Direct Preference Optimization — Rafailov et al., 2023

With the toolkit from the previous chapter in hand — distributions, sampling, expected rewards, policies, and gradients — we can now name the three specific quantities those methods actually optimize. Several methods in this explainer — supervised fine-tuning, RLHF, direct-preference methods, and reasoning-RL pipelines — recur around three quantities: a likelihood, a KL divergence, and an entropy. They sound abstract, but each has a sharp, intuitive job. Likelihood is how we say “make the model put more probability on this text.” KL divergence is a regularizer that can keep a model from wandering too far from where it started. Entropy is one tool for resisting collapse into a single answer. Get comfortable with these three now and many later objectives become rearrangements of pieces you already understand.

We’ll keep things intuitive, but the formulas matter — they’re the actual objects the optimizers manipulate.

Recap: the next-token objective

Pre-training trained the model to predict the next token. Concretely, the model defines a probability distribution over the next token given everything before it. Write the model as a policy policy In RL, the thing that chooses actions — here, the language model itself, viewed as a distribution over next tokens given the context. RL post-training optimizes the policy. See in glossary → $\pi_\theta$ with parameters $\theta$; for a sequence of tokens $y = (y_1, y_2, \ldots, y_T)$ following a context $x$, the model assigns the sequence a probability by multiplying its per-token predictions together:

Each factor is a softmax softmax Function that turns any vector into a probability distribution (positive, sums to 1) by exponentiating and normalizing. See in glossary → over the vocabulary, read off the final-layer logits logits The raw, pre-softmax scores the model produces — one per vocabulary token, per position. Bigger logit = the model finds that token more likely; the actual value can be any real number, positive or negative. Applying softmax across the vocabulary turns logits into a probability distribution that sums to 1. Sampling then picks one token from that distribution. See in glossary → . Training minimized the cross-entropy cross-entropy loss The standard LM loss: the negative log-probability the model assigned to the actual next token, averaged over all positions. Zero would mean perfect confidence in every correct token. See in glossary → between this distribution and the actual next token in the corpus. That’s the entire pre-training story, and it’s the foundation everything here builds on. (If the perplexity perplexity The exponential of the cross-entropy loss — roughly "how many equally-likely tokens is the model choosing between?" Lower is better; a perplexity of 1 means perfect prediction. See in glossary → and cross-entropy machinery is hazy, the pre-training explainer covers it in depth.)

Likelihood, and why SFT is “just more pre-training”

The likelihood likelihood The probability a model assigns to observed data. Supervised fine-tuning maximizes the likelihood of human-written target responses given their prompts. See in glossary → of a piece of text under the model is simply the probability the model assigns to it: $\pi_\theta(y \mid x)$, the product above. Because multiplying many small probabilities underflows to zero, we always work in logs, where the product becomes a sum:

Let’s go through the notation and explain what it means. The $\pi$ is the model itself, viewed as a policy policy In RL, the thing that chooses actions — here, the language model itself, viewed as a distribution over next tokens given the context. RL post-training optimizes the policy. See in glossary → — a function that maps a context to a probability distribution over what comes next. Its subscript $\theta$ is the model’s parameters (its billions of weights); writing $\pi_\theta$ emphasizes that the distribution is the thing training adjusts, since changing $\theta$ changes every probability. Inside the parentheses, $x$ is the prompt (the instruction), $y$ is the response, and the vertical bar $\mid$ means “given” — so $\pi_\theta(y \mid x)$ is the probability of the response $y$ conditioned on the prompt $x$. On the right, the sum walks token by token: $y_t$ is the $t$-th token of the response, $T$ is its length, and $y_{<t}$ is everything generated before position $t$ — so each factor is the probability of the next token given the prompt and all tokens so far. Multiplying those per-token probabilities gives the whole-sequence probability; taking logs turns that product into the sum above.

This single expression is the engine of supervised fine-tuning supervised fine-tuning (SFT) Training a pre-trained model on curated (prompt, response) pairs with the ordinary next-token objective, so it imitates demonstrated assistant behavior. The first stage of post-training. See in glossary → . SFT takes a curated dataset of good (instruction, response) pairs and adjusts $\theta$ to maximize the log-likelihood of the target responses. We’re telling the model: “make text that looks like this — a helpful assistant answering — more probable.”

The takeaway is that SFT uses the same basic autoregressive cross-entropy machinery as pre-training. The data and loss masking usually differ: instead of broad pre-training data, we feed carefully chosen demonstrations and commonly apply loss only to assistant-response tokens. Optimizer settings and schedules can differ too. Instruction tuning instruction tuning Fine-tuning on many tasks phrased as natural-language instructions so the model learns to follow instructions in general — including ones it never saw in training. See in glossary → is this idea applied to instruction-following data. We unpack it properly in Chapter 6; for now, the key idea is that “maximize likelihood of good responses” is the SFT objective.

KL divergence: the leash

Maximizing likelihood is fine when you have demonstrations to imitate. But the reinforcement-learning methods later in this explainer do something riskier: they let the model generate its own text and push it toward higher reward. Left unconstrained, that optimization can drag the model far from sensible English — it can discover degenerate, high-reward gibberish, or simply forget how to write. We need a way to say “improve, but don’t drift too far from the model you started as.”

That measure is the Kullback–Leibler (KL) divergence KL divergence Kullback–Leibler divergence — a measure of how far one probability distribution is from another. Used in post-training as a "leash" that keeps a model close to a reference policy. See in glossary → . Given two distributions $p$ and $q$ over the same set of outcomes, it is defined as:

Let’s go through the notation and explain what each piece means. The $D$ stands for divergence — the kind of quantity this is, much as $d(x, y)$ denotes a distance. The subscript $\mathrm{KL}$ names which divergence (Kullback–Leibler; there are others), and $(p \,\|\, q)$ are the two distributions being compared, with the double bar $\|$ separating them and reminding us that order matters. “Divergence” is deliberately weaker than “distance”: a true distance must be symmetric and obey the triangle inequality triangle inequality The rule that a direct distance is never longer than a detour: d(a, c) ≤ d(a, b) + d(b, c). It is one of the properties a true distance metric must satisfy — and one that KL divergence does not, which is why KL is a divergence rather than a distance. See in glossary → , whereas a divergence need only be non-negative and zero exactly when $p = q$. KL meets those two conditions but not symmetry — which is precisely why it’s written as a divergence $D$ rather than a distance $d$.

Read it as an expected log-ratio: for outcomes that $p$ considers likely, how much do $p$ and $q$ disagree? If $p$ and $q$ are identical, every ratio is $1$, every log is $0$, and the divergence is $0$. The more $p$ puts mass where $q$ does not, the larger it grows.

The units depend on the logarithm’s base. With the natural log $\ln$ (base $e$), as we use throughout here, the result is measured in nats — the natural unit of information. Using $\log_2$ instead would report the same quantity in bits; the choice is just a constant rescaling. The interactive below reports KL in nats.

In post-training, the second distribution is almost always a frozen copy of the model before RL began — the reference model reference model A frozen copy of the policy (usually the SFT model) that RLHF and DPO stay close to via a KL penalty, preventing the optimized policy from drifting into degenerate text. See in glossary → , $\pi_{\mathrm{ref}}$ (typically the SFT checkpoint). We measure $D_{\mathrm{KL}}(\pi_\theta \,\|\, \pi_{\mathrm{ref}})$ and add it to the objective as a penalty. The reward pulls the policy toward better behavior; the KL term pulls it back toward the reference. The balance between those two forces is the central knob of RLHF, and we’ll see the exact KL penalty KL penalty A term added to the RLHF reward that subtracts β times the KL divergence from the reference policy, keeping the optimized model from drifting too far while chasing reward. See in glossary → when we get to RLHF in practice. For now, hold onto the image: KL is the leash, and the reference model is the post the leash is tied to.

Entropy: keeping the distribution alive

The last tool is entropy entropy A measure of how spread-out (uncertain) a probability distribution is. In RL post-training, keeping entropy up preserves exploration and prevents premature collapse onto one answer. See in glossary → , a measure of how spread-out a distribution is. For the model’s next-token distribution $\pi_\theta(\cdot \mid x)$ over the vocabulary, it is:

High entropy means the model is genuinely uncertain — probability spread across many plausible next tokens. Low entropy means it’s nearly committed to one. (Note the shape: entropy is just the expected negative log-probability the model assigns to its own samples.)

Why do we care during post-training? Because reward optimization can reduce entropy. As the policy learns that a particular phrasing scores well, it may pile probability onto it, and the distribution sharpens. Pushed too hard, this becomes mode collapse: the model converges on one rigid template and produces it for everything, losing diversity and often the ability to explore better answers. (Note the term: mode collapse is a distribution collapsing onto a single mode mode A peak of a probability distribution — an outcome (or region) of locally maximal probability. A distribution can have several modes; "mode collapse" is when a model piles nearly all its probability onto just one of them. See in glossary → — distinct from model collapse, the separate phenomenon of quality degrading when a model is trained over generations on its own synthetic output.)

There’s a real tension here, and it runs through the whole field. We want the model to become more confident about good behavior while not collapsing into a single mode. Entropy bonuses, sampling choices, KL regularization, and data diversity can all influence that balance.

The toolkit, assembled

Three quantities, three jobs:

• Likelihood ($\log \pi_\theta(y \mid x)$) — the target SFT pushes up on good text. The “imitate this” signal.
• KL divergence ($D_{\mathrm{KL}}(\pi_\theta \| \pi_{\mathrm{ref}})$) — the leash that keeps an optimizing policy anchored to a trusted reference. The “don’t drift” signal.
• Entropy ($H(\pi_\theta)$) — the spread we protect to keep the model exploring and diverse. The “don’t collapse” signal.

These quantities recur throughout the explainer, sometimes explicitly and sometimes implicitly. When you meet the RLHF policy-optimization loss or the direct-preference loss for the first time and it looks intimidating, come back here: many of the same pieces are present, though each method makes additional assumptions and design choices. Next, we turn to why we need anything beyond likelihood at all — the alignment problem, and the limits of pure imitation.