Section 25

Inference scaling & o1

Test-time compute and the reasoning model

System: Learning to Reason with LLMs (OpenAI o1) — OpenAI, September 2024

In September 2024, OpenAI announced o1, a model that spends additional internal reasoning tokens before answering. OpenAI reported substantial gains on hard math, code, and science evaluations and showed scaling curves with more train-time and test-time compute. This made more compute at inference time a prominent practical scaling axis alongside model size, data, and training compute.

A new kind of model

o1 was a high-profile reasoning modelreasoning modelA model trained (usually with RL) to produce long internal chains of thought before answering, trading inference compute for accuracy on hard problems. o1 and DeepSeek-R1 are examples.See in glossary →: a model designed to spend a variable, sometimes large, amount of computation on a problem before committing to an answer. OpenAI exposes only a summary rather than the raw internal chain of thought, so public descriptions of the exact trace length and mechanism are necessarily limited.

OpenAI described behaviors such as trying alternative approaches, checking work, and correcting mistakes. It reports training o1 with large-scale reinforcement learning, but does not disclose the full objective, data mixture, reward design, or raw internal traces. Public evidence therefore supports the observed product behavior and the use of RL, not a precise causal account of which training signal produced each strategy.

Test-time compute as a scaling axis

Here is the result that reorganized the field. Plot o1’s accuracy against the amount of test-time computetest-time computeCompute spent at inference — longer chains of thought, more samples — to improve answer quality, as opposed to compute spent during training.See in glossary → it’s allowed to spend (the length of its reasoning trace) and you get a clean, rising curve. Let it think longer, and it gets more right. This is inference scalinginference scalingThe empirical finding that accuracy improves predictably as you spend more test-time compute (longer reasoning, more samples) — a second scaling axis beyond model and data size.See in glossary → (also called test-time scaling): performance that improves predictably with compute spent at inference, not at training.

To feel why this is a big deal, recall the pre-training scaling laws from the sibling pre-training explainer. For years, the recipe for a better model was: more parameters, more data, more training compute. Loss fell predictably as you scaled those up. That axis is expensive and slow: you train one giant model, once, and you’re stuck with whatever capability it has. o1 revealed an orthogonal axis. Take a fixed trained model and simply let it think longer on the hard problems, and accuracy climbs along its own curve. You can now dial capability up per query, at inference, by spending more tokens.

This connects straight back to the deepest idea of chapter 23: tokens are compute. Self-consistency scaled test-time compute in parallel: sample NN independent chains and vote. o1 scales it serially: one very long chain that builds on itself, backtracks, and self-corrects. Serial scaling is more powerful for genuinely hard reasoning, because later steps can use the conclusions (and the discovered dead-ends) of earlier ones, which independent parallel samples cannot.

Try it

Below, slide the test-time-compute budget and watch accuracy climb. Compare the serial long-CoT curve against parallel self-consistency, and notice the diminishing returns: every doubling of compute buys a bit less than the last.

Test-time scaling
Accuracy vs test-time compute. Spend more on sampling and reasoning length — accuracy rises, then saturates.
25%50%75%100%test-time compute (log scale) →
Reasoning-trainedBase model
Reasoning-trained accuracy
45.6%
Base model accuracy
43.7%
budget ≈ 4 × 512 = 2,048 units of test-time compute
Accuracy improves predictably with test-time compute — a second scaling axis beyond model size and training data. But the base model saturates early: more sampling and longer chains stop helping once it has exhausted what it knows. The reasoning-trained model has a higher ceiling and keeps climbing for longer, because it was trained to actually use the extra thinking budget rather than just repeat itself.

The cost, and the honesty about it

There is no free lunch. Extra reasoning tokens, samples, tool calls, and verification all consume inference resources, so test-time scaling creates a recurring per-query cost. OpenAI has not publicly disclosed a universal o1 compute multiplier. The economics of generating tokens (KV-cache growth, latency per token, throughput) are covered in the LLM & vLLM Inference explainer.

The catch: o1 was a closed box

o1 was a landmark, but it was also a black box. OpenAI hid the reasoning traces, published no training details, and released no weights. The community knew the destination (RL-trained long chain-of-thought, scaling with test-time compute) but not the route. How do you train a base model to reason like this? What reward? What algorithm? Does it need the careful step-level supervision of chapter 24, or can a simple correctness signal suffice? Could anyone reproduce it in the open?

Within months, someone did. The next two chapters tell that story. Chapter 26 gives us the reward (RL from verifiable rewards, the automatic-correctness signal) and chapter 27 gives us the algorithm and the open model that put it all together: GRPO and DeepSeek-R1. o1 proved the destination existed. R1 drew the map.