Recap

And further reading

If you read straight through, you’ve covered roughly the same material a senior ML systems engineer would expect from a new hire: what an LLM is, what attention does, why decode is memory-bound, why a KV cache matters, and how every clever idea in modern serving is some flavor of “manage that cache better.”

This page is a short recap and a handful of pointers if you want to go deeper.

The whole story in 16 bullets

1. Text becomes a list of integer token IDs via a tokenizer, often a BPE-style tokenizer. Each ID indexes into that model’s vocabulary, whose size is model-specific.
2. Each ID is mapped to a vector (“embedding”) of size $d_{\text{model}}$ (4096-ish). That row is looked up from a giant matrix.
3. Many current decoder-only LLMs inject position information with RoPE (rotating query/key vectors inside attention).
4. A standard dense transformer layer has attention (cross-token mixing) and an MLP (per-token nonlinear processing), supported by residual connections and normalization such as RMSNorm.
5. Attention computes scores $Q K^\top$, applies softmax + causal mask, takes a weighted sum of $V$. Multi-head splits this across many parallel “heads.”
6. Stacking 32-128 of these blocks, plus an embedding lookup and an LM head, is the model.
7. To generate, you take the final position’s logits, apply a sampling strategy (greedy / temperature / top-p), get a token, append, repeat.
8. The first pass on the whole prompt (prefill) is often compute-bound for long or well-batched prompts. Small-batch single-token decode is commonly memory-bandwidth-bound because it repeatedly reads weights from HBM.
9. To avoid redoing prior token projections during decode, we cache the keys and values: the KV cache. It can be enormous; many modern models use GQA to shrink it.
10. The serving story is dominated by HBM bandwidth and HBM capacity. The rest of the memory hierarchy (SRAM, PCIe, NVLink, RDMA NIC) determines what kinds of parallelism work.
11. Multiple users can share one GPU step via continuous batching: requests are admitted and completed dynamically as the scheduler has capacity.
12. KV memory can be managed with PagedAttention: HBM is split into fixed-size blocks and each request has a block table, bounding per-request internal fragmentation.
13. Pages can be shared across requests via prefix caching: identical completed prefix blocks can re-use the same physical KV blocks, reducing duplicated prefill and cache memory.
14. Long prompts are processed via chunked prefill so they don’t block decoders.
15. Speculative decoding lets a draft model propose K tokens that the target verifies in one pass. Correct rejection sampling preserves the target distribution; the speedup depends on acceptance, draft cost, and serving conditions.
16. Frontier models increasingly ship the draft model inside the model as multi-token-prediction (MTP) layers, then tune the whole stack for serving: GLM-5.2 shares the sparse-attention index and KV cache across draft steps, drafts probabilistically, and trains with a loss that directly optimizes acceptance length.

When a single GPU isn’t enough, tensor parallelism (every matrix split across GPUs, all-reduce per layer over NVLink) and pipeline parallelism (layers split across GPUs, activations forwarded once per stage) carry the load, with data parallelism stacking replicas on top.

Underneath all of this, the same arithmetic ratio governs everything: how many FLOPs you do per byte of memory you read. Every optimization in the essay is some flavor of pushing that ratio up.

Further reading

The vLLM ecosystem and the foundational papers are remarkably accessible. A small reading list:

The papers

• Vaswani et al., Attention Is All You Need (2017) — arXiv 1706.03762. The Transformer.
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) — arXiv 2309.06180. The vLLM paper.
• Dao et al., FlashAttention (2022) — arXiv 2205.14135, and FlashAttention-2 (2023) — arXiv 2307.08691. Widely used attention kernels.
• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (2022) — OSDI paper. Iteration-level scheduling.
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2023) — arXiv 2211.17192. An exact speculative-decoding protocol.
• Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling (2023) — arXiv 2302.01318. An independent formulation.
• Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (2021) — arXiv 2104.09864. RoPE.
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023) — arXiv 2305.13245. The K/V sharing method.
• Cai et al., Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads (2024) — arXiv 2401.10774.
• Li et al., EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024) — arXiv 2401.15077.
• Bai et al., IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse (2026) — arXiv 2603.12201. The IndexShare idea GLM-5.2 applies to its backbone and MTP layer.
• Li et al., Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling (2026) — arXiv 2606.12370. Rejection sampling and the end-to-end TV loss for MTP drafts.
• Z.ai, GLM-5.2: Built for Long-Horizon Tasks (2026) — blog post. The full production serving stack from §21.

Codebases

• vllm-project/vllm: the canonical reference implementation.
• pytorch/torchchat: minimal PyTorch decode loop, easy to read.
• karpathy/llama2.c: an entire Llama inference engine in ~700 lines of C. Probably the best single piece of code for internalizing §1–§10.
• karpathy/llm.c: same idea, training-focused.

Posts to read next

• Aleksa Gordić’s vLLM internals blog: the more advanced cousin of this essay, written by someone who’s worked deeply on vLLM. The structural inspiration for this site.
• The Annotated Transformer: Harvard NLP’s classic walkthrough of the original Transformer paper, with PyTorch code.
• Lilian Weng — Large Language Model Inference Optimization: broad survey, very readable.
• Horace He — Making Deep Learning Go Brrr From First Principles: the canonical explanation of arithmetic intensity vs memory bandwidth.

Once you have all of this, the productive next step is to clone vLLM, find one of its core files (scheduler.py, worker.py, the paged-attention kernel), and read it. Everything in this essay is in there, with the details and the rough edges that make it production code.

That’s all. Thanks for reading. Now go run something.