Section 01

What is an LLM?

And what does "inference" mean?

Sources: The Llama 3 Herd of Models — Grattafiori et al., 2024; Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023

When you type something into ChatGPT, Claude, or any modern chat assistant and watch words appear one by one, something concrete is happening on a computer. A program is reading your text, turning it into numbers, doing an enormous amount of arithmetic on those numbers, and turning the result back into more text. That program is a Large Language Model (LLM)LLMLarge Language Model — a neural network trained on huge text corpora to predict the next token given previous tokens.See in glossary → — a particular kind of neural networkneural networkA function built by stacking many simple operations — mostly matrix multiplies with nonlinearities between them — whose behavior is shaped by tuning billions of internal numbers (its parameters) from data.See in glossary → — and the act of running it to produce output is called inferenceinferenceRunning a trained model to produce outputs. Training learns the weights once; inference uses them many times.See in glossary →.

This essay is about how inference works. We will start from “what is a token” and end at “how does vLLMvLLMAn open-source LLM inference engine, originally from UC Berkeley, that introduced paged attention and is now one of the most widely used serving systems for open-weight models.See in glossary → keep a $30,000 GPU saturated with hundreds of concurrent users.” Every concept builds on the previous one, every new term is defined the first time it appears, and there are interactive widgets along the way so you can poke at the ideas instead of just reading about them.

Training vs inference

A neural network is, at heart, a giant function. It takes numbers in, multiplies them by other numbers (called parametersparametersThe numbers (weights) inside a model that get adjusted during training. A “7B model” has 7 billion of them.See in glossary →, or weights), passes the results through some simple nonlinear functionsnonlinear functionA function whose output isn't just a scaled, shifted copy of its input — e.g. ReLU, GELU, sigmoid. Stacking nonlinearities between matrix multiplies is what lets a neural net represent anything more interesting than scaling and rotation.See in glossary →, and produces numbers out. The interesting trick is that the parameters are learned from data. We start with random parameters, show the network billions of examples (“here is some text: predict what comes next”), and slowly nudge the parameters so it does better. That nudging process is called training.

Training a particular model version is expensive. Frontier models cost tens of millions of dollars and run for months on tens of thousands of GPUs. Once that version is deployed, its parameters are frozen (they are just a big file of numbers, hundreds of gigabytes for a flagship model) and serving uses those fixed parameters to answer users’ prompts. Developers can later train or fine-tune a new version, but that happens outside live inference.

What does an LLM actually compute?

This series focuses on autoregressive, decoder-style language models. They are trained for one job: given preceding text, predict the next token. We will say much more about what a “token” is in the next section, but for now think of it as roughly “a word or word-fragment”. The model is shown a chunk of text and asked: what comes next?

That sounds almost embarrassingly simple. But once you have a really good next-token predictor, you can string it together: predict the next token, append it to the input, predict the next token after that, and so on. That loop (generate one token, feed it back in, generate another) is what we call autoregressiveautoregressiveGenerating one token at a time, where each new token is conditioned on every token that came before it.See in glossary → generation. This is how most text chat assistants generate their responses.

The piece you type in is called the promptpromptThe input text fed to the model — what you want it to continue or respond to.See in glossary →. The text the model generates in response is the completioncompletionThe text the model generates in response to a prompt.See in glossary →. Everything you see being typed out token by token in a chat UI is the model running through its autoregressive loop. Each generated token requires another complete forward pass through the model: on the order of (10^11) arithmetic operations for a dense 70B-parameter model, before counting attention work that grows with the context.

Why is this hard?

If next-token prediction is the whole game, you might wonder why the engineering is interesting at all. It turns out the difficulty splits into two very different problems:

  1. The model is gigantic. A flagship open-weights model like Llama-3-70B has 70 billion parameters. At 16-bit precision that is about 140 GB of weights. That does not fit on an 80 GB H100, and even a 141 GB H200 leaves essentially no space for the KV cache and other runtime data. Even the smaller 8B model is 16 GB. Every single token you generate requires reading every parameter of a dense model at least once from GPU memory. During token-by-token generation at small batch sizes, the chip is usually limited by how quickly it can move those weights from memory, rather than by its arithmetic throughput. That asymmetry shapes many decisions in a serving system.

  2. Many users want answers at the same time. If you decode one request one token at a time, GPUs are usually underutilized: the step reads the model’s weights but performs only one token’s worth of work with them. Real systems pack many requests together so that one read of the weights serves many users, while juggling the fact that those requests have wildly different prompt lengths, completion lengths, and arrival times.

A modern inference engine like vLLM is, fundamentally, an answer to both problems. It is a memory-management system disguised as a model server.

What this essay covers

We will move in three phases:

  1. Foundations (sections 2–10). What is a token, what is an embedding, what is attention, what is a transformer block, culminating in a complete picture of “a forward pass through an LLM.” If you already know all of this, you can skim.

  2. How inference actually runs (sections 11–14). What it means for a token to be generated, why decoding splits into two phases, what the KV cacheKV cacheThe stored keys and values from all past tokens, so attention at step t only needs to compute Q for the new token.See in glossary → is, where bytes physically live on an Nvidia H100, and how a scheduler folds many requests into one batch.

  3. vLLM internals (sections 15–18). Paged attention, prefix caching, chunked prefill, and speculative decoding: techniques vLLM uses to improve the throughput–latency trade-off. Their benefit and achievable concurrency depend on the model, hardware, request mix, and configuration.

  4. Scaling out (sections 19–21). What changes when one GPU isn’t enough.

The end goal is to give you a complete mental model of what happens between you pressing Enter and a stream of tokens coming back. Let’s start with the very first step: turning your text into numbers.