Section 11

BERT

Masked language modeling and bidirectionality

Listen to this chapter

Paper: BERT: Pre-training of Deep Bidirectional Transformers — Devlin et al., 2019

GPT-1 bet on the causal, generative objective. A few months later, Google’s BERT (Devlin et al., 2019) made the opposite bet, and on language understanding benchmarks, it won decisively. BERT matters to us not because it’s the line that became modern LLMs (it isn’t), but because it crisply illustrates the single most important fork in pre-training: what objective do you train on?

The objection BERT answered

A causal language modelcausal language modelA model that predicts each token using only earlier tokens (never future ones). "Causal" because information flows strictly left to right. The GPT family are causal LMs (Language Models).See in glossary → reads strictly left to right. That’s necessary for generation (you can’t condition on words you haven’t written yet) but it’s a handicap for understanding. To classify the sentiment of a sentence, you’d love to use the whole sentence, both directions, at every word.

BERT’s insight: if you’re not trying to generate, you don’t need the causal constraint. Drop it, use the encoderencoderThe half of a transformer that reads an input sequence with full (bidirectional) attention, producing a contextual representation of it. BERT is encoder-only.See in glossary → with full bidirectionalbidirectionalAble to use context from both the left and the right of a token. BERT is bidirectional; a causal language model is left-to-right only.See in glossary → attention so every token sees every other token. But now you need a different objective, because a bidirectional model trained on next-token prediction would trivially cheat (each token could see itself through the layers).

Masked language modeling

The replacement is the masked language modelmasked language modelMasked Language Model (MLM) — a pre-training objective (used by BERT) that hides a fraction of tokens and trains the model to fill them in using context from both sides. Contrast with next-token prediction.See in glossary → (MLM) objective, a denoisingdenoising objectiveAny pre-training objective that corrupts the input (masking, deleting, or shuffling tokens) and trains the model to restore the original. Masked LM and span corruption are both denoising objectives.See in glossary → task. Hide a fraction of the tokens and train the model to reconstruct them from the surrounding context on both sides:

  • 15% of tokens are selected for prediction.
  • Of those, 80% are replaced with a special [MASK] token, 10% with a random token, and 10% are left unchanged. This mix mitigates the train/inference mismatch: [MASK] never appears at fine-tuning time, so exposing the model to ordinary and randomly replaced tokens during pre-training prevents it from relying exclusively on [MASK].
  • The model predicts the originals with the same cross-entropycross-entropy lossA classification loss that penalizes the model according to the negative log-probability it assigned to the correct answer.See in glossary → loss we already know.

BERT added a second objective, Next Sentence Predictionnext sentence predictionNext Sentence Prediction (NSP) — a secondary BERT objective: given two sentences, predict whether the second actually follows the first. Later work found it largely unnecessary.See in glossary → (NSP) — given two sentences, predict whether the second really follows the first — to help with sentence-pair tasks. (Later work, notably RoBERTa, found NSP largely unnecessary, a nice reminder that not every component of a famous model turns out to matter.)

Three pre-training objectives, one sentence
What does each objective actually ask the model to predict?
Thequickbrownfoxjumpsoverthelazydog
Left-to-right only. Blue = visible left context, teal = the token being predicted, dim = the future, hidden by the causal mask. Every position is a training example, all computed in parallel. This is the objective that became the LLM.

The widget above makes the contrast tangible. Toggle between the objectives and notice the fundamental trade:

The specifics

  • Two sizes: BERT-Base (12 layers, hidden 768, 12 heads, 110M parameters, deliberately matched to GPT-1) and BERT-Large (24 layers, hidden 1024, 16 heads, 340M parameters). Feed-forward inner size is 4H4H throughout.
  • WordPieceWordPieceA subword tokenization algorithm (used by BERT) closely related to Byte Pair Encoding, building a vocabulary of word pieces from frequent character sequences.See in glossary → tokenization with a 30,000-token vocabulary.
  • Pre-training data: BooksCorpusBooksCorpusA dataset of around 7,000 unpublished books (~800M words) used to pre-train GPT-1. Long contiguous passages made it good for learning long-range structure.See in glossary → (800M words) plus English Wikipedia (2,500M words), about 3.3 billion words. Like GPT-1, the authors stressed using document-level text to preserve long contiguous passages.