LLM Inference

A long-form, interactive explainer

This is a walkthrough of how modern Large Language Models actually run — from the moment your text goes in, through every step of the arithmetic that picks the next word, to the memory tricks that production serving systems like vLLM use to keep the expensive chips they run on fully busy.

The v in vLLM stands for virtual: its signature trick borrows the virtual memory pagingvirtual memory pagingThe operating-system technique that gives each program the illusion of one large contiguous memory while physically storing it as small fixed-size pages scattered across RAM. vLLM's PagedAttention applies the same idea to the KV cache.See in glossary → idea operating systems have used to manage RAM since the 1960s, and applies it to the model's short-term memory. Chapter 15 is devoted to exactly how that works.

No prior machine-learning knowledge required. Every term gets defined the first time it shows up — you can hover any underlined word for a quick tooltip, or jump to the glossary at any time.

There are interactive widgets throughout: you can type text and watch it get split into the pieces the model actually reads, hover a chart of which earlier words the model looks back at, step through generation as the model's working memory fills up, and play with the same memory-allocation puzzle vLLM solves in production. They're meant to be played with, not just looked at.

Start reading →~2 hours, 22 sections

Contents

Foundations

  1. 01What is an LLM?— And what does "inference" mean?
  2. 02Tokens— Text → numbers the model can see
  3. 03Embeddings— Token IDs → vectors
  4. 04Attention— Queries, keys, and values
  5. 05Multi-head attention— Many attentions in parallel
  6. 06Positional encoding— Telling the model where each token sits
  7. 07The MLP block— Per-token nonlinear processing
  8. 08A full transformer block— Putting it together
  9. 09Stacking into a full model— From embeddings to logits
  10. 10Sampling— Logits → the next token

How serving actually works

  1. 11Prefill and decode— The two phases of inference
  2. 12The KV cache— Why decode is cheap and memory is expensive
  3. 13GPU memory hierarchy— Where data actually lives on H100s
  4. 14Continuous batching— Stop wasting GPU steps

vLLM internals

  1. 15Paged attention— KV cache as a page table
  2. 16Prefix caching— Shared system prompts for free
  3. 17Chunked prefill— Stop blocking decoders with one big prompt
  4. 18Speculative decoding— Draft fast, verify in bulk

Scaling out

  1. 19Scaling out— Tensor parallelism vs pipeline parallelism
  2. 20Throughput vs latency— What knobs move what

The 2026 frontier

  1. 21GLM-5.2— Tuning MTP speculative decoding to the limit
  1. 22Recap— And further reading