How a model learns

Gradient descent and backpropagation

We have an objective (minimize the cross-entropy loss) and a model with billions of parametersparametersThe numbers (weights) inside a model that get adjusted during training. A “7B model” has 7 billion of them.See in glossary → to adjust. This chapter is about the engine that does the adjusting: gradient descentgradient descentThe core training algorithm: repeatedly nudge each parameter a small step in the direction that lowers the loss, as told by the gradient.See in glossary →, powered by backpropagationbackpropagationThe algorithm that computes the loss gradient for every parameter efficiently by applying the chain rule backward through the network, reusing intermediate results from the forward pass.See in glossary →. Together they are the reason a pile of random numbers turns into something that can write code.

The loss is a landscape; training walks downhill

Fix the training data and think of the loss $\mathcal{L}(\theta)$ as a function of the parameters $\theta$ alone. With billions of parameters this loss landscapeloss landscapeThe (extremely high-dimensional) surface of loss as a function of the parameters. Training is a walk downhill on this surface toward a low-loss region.See in glossary → lives in billions of dimensions, but the three-dimensional picture carries over: two parameters spread across the ground, height is the loss, and we want to reach a low valley.

The tool for going downhill is the gradientgradientThe vector of partial derivatives of the loss with respect to every parameter — it points in the direction of steepest loss increase, so we step the opposite way to reduce the loss.See in glossary →, $\nabla_\theta \mathcal{L}$ (the symbol $\nabla$ is read “nabla” or “del”): the vector of partial derivatives of the loss with respect to every parameter. It points in the direction of steepest increase, so to decrease the loss we step the opposite way:

The scalar $\eta$ (the Greek letter eta) is the learning ratelearning rateThe size of each parameter step. Too high and training can diverge; too low and it crawls.See in glossary →: how big a step to take. That single update rule, repeated, is gradient descent.

Play with the learning rate above and the central tension of all training appears immediately. Too small and progress is glacial. Too large and the ball overshoots the valley, zig-zags, or flies off entirely. And because the surface is steeper in one direction than the other, no single learning rate is ideal for both axes at once. A real network has billions of axes with wildly different curvatures. Holding that thought; it is precisely what the optimizer in the next chapter exists to solve.

Stochastic gradient descent: don’t read the whole library each step

The true loss is an average over the entire corpus. Computing its exact gradient would mean a forward pass over trillions of tokens for a single update. Absurd. Instead we estimate the gradient from a mini-batchmini-batchThe chunk of training examples processed together in one step. Gradients are averaged over the mini-batch, trading off gradient noise against memory and compute.See in glossary → of sequences sampled from the data. The estimate is noisy, but it is unbiased and millions of times cheaper, and the noise even helps escape bad regions. This is stochastic gradient descentSGDStochastic Gradient Descent — gradient descent using a noisy gradient estimated from one mini-batch at a time rather than the whole dataset.See in glossary →, and one such update is a training steptraining stepOne iteration of the loop: forward pass on a batch, backward pass to get gradients, optimizer update. A large model is trained for hundreds of thousands of steps.See in glossary →.

A frontier run is hundreds of thousands to millions of these steps. Teams often avoid repeatedly sampling the same general-web data when fresh data is available, but curated sources can be deliberately upsampled and data can recur across training stages. The key idea is that, when plentiful fresh data exists, repeated exposure usually has diminishing returns. (We will quantify this tradeoff with scaling laws.)

Backpropagation: getting a billion derivatives for the price of one pass

The catch is step 3. We need $\partial \mathcal{L} / \partial \theta_i$ for every parameter: billions of derivatives. Computing each one independently would be hopeless. BackpropagationbackpropagationThe algorithm that computes the loss gradient for every parameter efficiently by applying the chain rule backward through the network, reusing intermediate results from the forward pass.See in glossary → computes them all in a single backward sweep, and it is the algorithm that makes deep learning feasible.

The idea is the chain rulechain ruleThe calculus rule for differentiating composed functions. Backpropagation is just the chain rule applied layer by layer, from the loss back to the inputs.See in glossary →. A neural network is a long composition of simple operations (matrix multiplies, normalizations, nonlinearities). In the forward passforward passRunning inputs through the network to produce outputs (logits) and the loss, caching intermediate activations that backpropagation will need.See in glossary → we run inputs through to produce the loss, caching the intermediate activationsactivationsThe values produced by a layer after applying its activation function. During training, intermediate activations are often kept for the backward pass.See in glossary →. In the backward passbackward passThe second half of a training step: backpropagation walks from the loss back through the network, computing each parameter's gradient.See in glossary → we walk the operations in reverse, and at each one multiply the gradient flowing back by that operation’s local derivative. Each layer needs only its cached inputs and the gradient arriving from the layer above, so the whole gradient costs about the same as two forward passes, regardless of how many parameters there are.

Two consequences of backprop shape everything downstream:

• Memory. The backward pass needs the forward activations, so they must be kept in memory until used. At long context lengthcontext lengthThe maximum number of tokens the model can attend to at once (also called the context window or sequence length). Pre-training picks a context length; later stages often extend it.See in glossary → these activations balloon, which is why gradient checkpointinggradient checkpointingActivation recomputation — saving memory by discarding most activations in the forward pass and recomputing them during the backward pass, trading extra compute for far less memory.See in glossary → (recomputing activations instead of storing them) becomes essential at scale.
• Numerics. Gradients are computed through many multiplications and can become very small or very large. Keeping them representable is the job of the precision machinery: loss scaling, BF16, careful normalization.

You rarely write backprop by hand; frameworks like PyTorch build a computational graph during the forward pass and differentiate it automatically. But knowing that the gradient is cheap to compute but expensive to store explains an enormous amount about how large models are actually engineered.

With gradients in hand, the remaining question is how to use them well, which is the optimizer.