Qwen2.5

18 trillion tokens and data quality

Paper: Qwen2.5 Technical Report — Qwen Team, 2024

Qwen2.5 (Qwen Team, Alibaba, 2024) doesn’t introduce a flashy new architecture: it’s a conventional dense transformer in the modern style (pre-normpre-normPlacing the normalization layer before each sub-layer (inside the residual branch) rather than after it. Pre-norm transformers are far more stable to train at depth, and became standard after GPT-2.See in glossary →, GQAGQAGrouped-Query Attention — multiple query heads share one K/V head, shrinking the KV cache by 4–8× with minimal quality loss.See in glossary →, RoPERoPERotary Position Embeddings — rotates Q/K vectors by an angle proportional to position. Standard in modern LLMs.See in glossary →, SwiGLU). Its lesson is about data scale and data discipline, and it’s a clean illustration of the single most reliable lever in pre-training: more, better-filtered tokens.

From 7 trillion to 18 trillion tokens

The defining change from Qwen2 to Qwen2.5 is the corpus: pre-training data grew from 7 trillion to 18 trillion tokens, among the largest disclosed for an open model. Crucially, the growth wasn’t just “more web text.” The expansion concentrated on the categories that most improve capability: knowledge, code, and mathematics, with heavy quality filteringquality filteringDiscarding low-value text (spam, boilerplate, gibberish) using heuristics and trained classifiers, keeping the corpus closer to the kind of text you want the model to learn.See in glossary → to keep the added tokens valuable rather than merely numerous.

Staged training and mixture transitions

Qwen2.5’s pre-training is staged: the data mixture changes over the course of training rather than staying fixed. This is the curriculum idea from chapter 8 in practice: early training on broad data to build general ability, later stages shifting toward higher-quality and more specialized mixtures, much like Llama 3’s annealingannealingA final pre-training phase that upsamples small amounts of the highest-quality data (math, code, curated text) while the learning rate decays to its floor. Reliably boosts quality and can be used to gauge a dataset's value.See in glossary → finish. The model is fed a changing diet tuned to what it most needs at each phase.

The family, and the MoE hint

Qwen2.5 ships an unusually wide range of open dense sizes — 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B — a deliberate choice to serve everything from on-device to data-center deployment, each size over-trainedover-trainingDeliberately training a model on far more tokens than the compute-optimal ~20 per parameter. It costs more training compute for a slightly better, much smaller model that is cheaper to run at inference.See in glossary → on the full corpus for inference efficiency in the Llama style. Alibaba also fielded proprietary Mixture-of-ExpertsMixture of ExpertsMixture of Experts (MoE) — a layer with many parallel sub-networks ("experts") where a router sends each token to only a few. The model has a huge total parameter count but activates only a fraction per token, so compute stays modest.See in glossary → variants (Qwen2.5-Turbo and -Plus), and Turbo pushed context lengthcontext lengthThe maximum number of tokens the model can attend to at once (also called the context window or sequence length). Pre-training picks a context length; later stages often extend it.See in glossary → to as much as a million tokens, foreshadowing the long-context and MoE directions that dominate the 2026 chapters.

Qwen scaled data. The next model, Gemma 2, does something cleverer with the targets the model trains against, and brings two architectural efficiency tricks worth knowing.