GPT-3

175B parameters and in-context learning

Paper: Language Models are Few-Shot Learners (GPT-3) — Brown et al., 2020

GPT-3 (Brown et al., 2020, Language Models are Few-Shot Learners) is the model that took the scaling hypothesisscaling hypothesisThe idea — crystallized around GPT-2 — that simply scaling up model size, data, and compute keeps improving capabilities, without needing fundamentally new architectures.See in glossary → and Kaplan’s “go big” recipe to their conclusion: 175 billion parameters, more than ten times any previous dense language model. But its lasting contribution wasn’t the size. It was a capability that fell out of scale, one the loss curves never advertised.

The same objective, two orders of magnitude bigger

Architecturally, GPT-3 is barely changed from GPT-2: a decoder-onlydecoderThe half of a transformer that generates a sequence one token at a time using masked (causal) self-attention. GPT-style language models are decoder-only.See in glossary → transformer trained on next-token predictionnext-token predictionThe pre-training objective for GPT-style models: given the tokens so far, predict a probability distribution over the next token. Also called causal or autoregressive language modeling.See in glossary →. It just scaled everything. The flagship is 96 layers, hidden size 12,288, 96 attention heads, a 2,048-token context windowcontext lengthThe maximum number of tokens the model can attend to at once (also called the context window or sequence length). Pre-training picks a context length; later stages often extend it.See in glossary →, and ~175B parameters (with alternating dense and locally-banded sparse attention to keep attention costs manageable). Following Kaplan, it was trained on roughly 300B tokens: a lot, but as we’ll see next chapter, too few for its size.

The data was a deliberate mixture: filtered Common CrawlCommon CrawlA free, monthly public crawl of the web — petabytes of raw HTML. It is the raw feedstock for most large pre-training corpora after heavy filtering.See in glossary → as the bulk, plus higher-quality sources (a WebText-style corpus, two book collections, and Wikipedia) up-weighted in the mixturedata mixtureThe recipe specifying what fraction of training tokens comes from each source (web, code, books, math, multilingual). Tuning the mixture is one of the highest-leverage data decisions.See in glossary → so the model saw them more often than their raw size would imply. That up-weighting of quality is a direct ancestor of every modern data-mixing strategy.

The surprise: in-context learning

Here’s what scale unlocked. GPT-3 can perform a brand-new task at inference time, with no gradient updates, simply from a description and a few examples placed in its prompt:

Translate English to French:
sea otter => loutre de mer
cheese => fromage
plush giraffe => ???

The model just continues the pattern. This is in-context learningin-context learningA model performing a new task purely from examples or instructions placed in its prompt, with no gradient updates. GPT-3 showed this emerges from pure next-token pre-training at scale.See in glossary →, and GPT-3 measured it across three regimes: zero-shotzero-shotPerforming a task from instructions alone, with no examples given. GPT-2 showed a pre-trained LM can do many tasks zero-shot, just by being prompted.See in glossary → (instructions only), one-shot (one example), and few-shotfew-shotGiving the model a handful of worked examples in the prompt before the real query, so it infers the task from them. Contrast with zero-shot (instructions only) and one-shot (a single example).See in glossary → (a handful). Crucially, the larger the model, the steeper its in-context learning curves: bigger models extract far more from the same few examples.

Emergence

In-context learning is the headline example of emergent abilitiesemergent abilitiesCapabilities that are absent in smaller models but appear, sometimes abruptly, once a model is large enough — e.g. multi-step arithmetic or in-context learning of novel tasks.See in glossary →: capabilities largely absent in smaller models that appear, sometimes sharply, past a size threshold. GPT-3 reported many: arithmetic, word unscrambling, using novel words, all weak-to-absent at small scale and suddenly workable at 175B. (How “sharp” emergence really is became a subject of later debate, but the qualitative point — new abilities show up with scale — reshaped the field’s ambitions.)

GPT-3 followed Kaplan faithfully: a giant model trained on comparatively modest data. It worked spectacularly, and yet it left a vast amount of performance on the table. Showing exactly how much, and why, is the achievement of Chinchilla.