Throughput vs latency
What knobs move what
Throughout this essay we’ve kept saying “this knob trades throughput for latency.” This is the section where we make that explicit and look at the actual knobs a vLLM operator turns to hit an SLOSLOService Level Objective — a target like “p99 TTFT < 1 s”. Serving systems are tuned to maximize throughput subject to SLOs.See in glossary →.
The roofline you’re up against
A serving system on a single GPU has two hard ceilings:
- Compute ceiling: the GPU’s peak FLOPS (~990 TFLOPS BF16 on H100). Hit at very large batches and during prefill.
- Memory-bandwidth ceiling: the GPU’s HBM bandwidth (~3.35 TB/s on H100). This commonly limits small-batch decode; larger decode batches can become compute- or cache-capacity-limited instead.
Plot throughput (tokens/sec) against batch size, and you get a classic roofline shape: throughput rises nearly linearly until you hit the compute ceiling, then plateaus. The “saturation batch size” (the batch at which the curve bends) is where you’re most efficient.
The saturation batch depends on the model, precision, context length, kernels, and request mix. It should be measured on the workload you plan to serve. Below it, compute may be underused; above it, throughput flattens while per-token latency usually rises because each step takes longer.
The knobs
vLLM exposes controls in this family through vllm serve; exact flag names, defaults, and availability vary by release:
| Knob | Direction | Trades |
|---|---|---|
--max-num-seqs | up | More concurrency → higher throughput, more KV memory, more ITL |
--max-num-batched-tokens | up | Bigger per-step batches → higher throughput, more ITL |
| Chunked-prefill controls | on / larger token budget | Smooths ITL during long prompts; a larger chunk can improve prefill progress while delaying decode |
--enable-prefix-caching | on | Big TTFT and throughput wins for shared-prompt workloads |
--gpu-memory-utilization | up | Bigger KV pool → more concurrent requests, less safety margin |
--block-size | bigger | Less per-page overhead, more internal fragmentation |
--num-speculative-tokens (K) | up | Speculation; helps if acceptance is high, hurts otherwise |
--speculative-model | enabled | Speculation; net win if draft is well-matched |
--tensor-parallel-size | up | More GPUs per replica → bigger models, more cross-GPU sync |
--pipeline-parallel-size | up | More GPUs across pipeline → bigger models, more pipeline bubbles |
| Weight / KV quantization | lower precision | Smaller weights or cache → more memory headroom, with model- and kernel-dependent quality and speed trade-offs |
The combinatorics here are substantial. Operators normally benchmark representative workload mixes, choose candidate configurations, and evaluate their throughput/latency frontier against the service’s SLOs.
The three numbers you actually report
Any serving SLO comes down to three numbers:
- TTFT p99: what’s the worst time-to-first-token among the top 1% slowest requests? This is what users feel when they press Enter.
- ITL p99: same for inter-token latency. Choppy streaming is way worse than slow-but-steady streaming.
- Throughput: total tokens generated per second across all in-flight requests. This is what you ratio against GPU cost to compute $ / token.
A common target: “p99 TTFT < 1 s, p99 ITL < 50 ms, maximize throughput subject to those.”
Two stylized workloads, two configs
To make the tradeoffs concrete, contrast two services running the same Llama-3-70B model.
A: Code completion. Short prompts (~200 tokens), short completions (~50 tokens), high request rate, very tight latency.
- Small batch sizes (tight latency).
- Aggressive prefix caching (header files, project context repeat).
- Chunked prefill on (don’t let any one prompt dominate).
- Evaluate speculation on the actual workload; low-batch decode can benefit, but draft overhead and acceptance determine the result.
- KV cache moderate, lots of free pages so admissions are fast.
B: Document analysis. Long prompts (~16k tokens), long completions (~2k tokens), low request rate, latency-tolerant.
- Big batch sizes (KV cache is the bottleneck per-request anyway).
- Chunked prefill on (16k prompts must be split).
- Consider speculation if a draft method has good acceptance on the workload.
- Size the KV-cache pool and any preemption/offload policy around the desired concurrency and tail latency.
- Choose TP based on the model precision, available memory, topology, and context budget.
Same model. Same engine. Different configs because they have different shapes.
Quantization, briefly
The other lever we haven’t discussed is quantization: storing weights (and optionally KV) in fewer bits per number.
- FP16 / BF16: 2 bytes/param. The default. Llama-3-70B = 140 GB.
- FP8: roughly 1 byte/weight plus scales and metadata. It can preserve quality well with an appropriate quantization recipe, but results are model- and workload-dependent. A 70B weight file is roughly 70 GB before runtime buffers and KV cache, so fitting the weights on an 80 GB H100 does not by itself make it a practical serving configuration.
- INT4 (AWQ, GPTQ, GGUF Q4_K_M): roughly 0.5 byte/weight plus metadata. It can reduce memory traffic, but realized speed depends on compatible kernels and dequantization overhead as well as quality.
- KV cache quantization: FP8 or even INT4 KV. The cache fits 2-4× more concurrent requests for the same HBM.
Lower-precision formats can be highly cost-effective, especially when memory capacity or bandwidth is the bottleneck. They require evaluation: hardware support, quantization recipe, kernels, latency, and task quality all affect the result.
What we did not cover
This essay’s playbook is the mainline. A few important topics we skipped to keep the length manageable:
- FlashAttention: an IO-aware attention algorithm that computes and softmax in blocks without materializing the full attention matrix. It and related kernels are widely used, including in many paged-attention implementations.
- CUDA graphs: a way to record a sequence of kernel launches and replay them with low per-launch overhead. vLLM uses these in decode for the most common batch shapes.
- Mixture-of-Experts (MoE) routing: DeepSeek, Mixtral, etc. activate only a subset of experts per token. The serving story for MoE has its own set of tricks.
- Disaggregated P/D at scale: already mentioned in §17; the production engineering of a separated prefill cluster + decode cluster is its own subfield.
Before the recap, one frontier case study pulls nearly every lever from this essay at once: GLM-5.2 and its multi-token-prediction serving stack.