Section 13

GPU memory hierarchy

Where data actually lives on H100s

Sources: NVIDIA H100 Tensor Core GPU; NVIDIA Hopper Architecture In-Depth

Every optimization in vLLM ultimately bottoms out in one fact: bytes move at very different speeds depending on where they live. To reason about why prefill is fast, why decode is slow, why batching helps, why KV cache offload is a real option, and why multi-GPU serving requires NVLink, you have to know the rough numbers of the GPU memory hierarchy.

We’ll use the Nvidia H100 SXM5SXM5Server PCI eXpress Module, 5th generation — Nvidia's proprietary mezzanine board form factor for datacenter GPUs. (Despite the name, SXM bypasses PCIe entirely.) An H100 SXM5 module plugs directly into the motherboard via the SXM socket, which gives it more power (700 W vs ~350 W for PCIe), more NVLink bandwidth (900 GB/s per GPU), and higher HBM bandwidth than the PCIe variant of the same chip. Standard in HGX/DGX servers; what you get in most cloud H100 instances.See in glossary → as our concrete reference because it’s the workhorse of modern inference deployments. Other chips (A100, B100/B200, MI300X) have similar shapes with different magnitudes.

Where the tiers physically live — zooming out from one SM to a rack
Each step is one "zoom out" level. Every row maps to one or more entries in the bandwidth/latency table below.
Single SML1 instruction cache·TMA · texture unitsPartition 0warp scheduler · dispatch64 KB register fileFP32 ×32INT32 ×32FP64 ×164th-gen Tensor CoreFP16 / BF16 / FP8 matmulPartition 1warp scheduler · dispatch64 KB register fileFP32 ×32INT32 ×32FP64 ×164th-gen Tensor CoreFP16 / BF16 / FP8 matmulPartition 2warp scheduler · dispatch64 KB register fileFP32 ×32INT32 ×32FP64 ×164th-gen Tensor CoreFP16 / BF16 / FP8 matmulPartition 3warp scheduler · dispatch64 KB register fileFP32 ×32INT32 ×32FP64 ×164th-gen Tensor CoreFP16 / BF16 / FP8 matmulShared SRAM · L1 data cache (configurable, up to 228 KB)streaming multiprocessor — 4 partitions sharing an L1 / SRAM block~256 KB total registers · 228 KB shared SRAM / L1 · 4 tensor cores · 128 CUDA cores · TMASet of SMs132 SMs totalone GPU die · 132 SMs arranged on the silicon (H100)sharing 60 MB L2 cacheGPU packageHBMHBMHBMHBMHBMGPU die(132 SMs + L2)HBMHBMHBMHBMHBMdie + on-package HBM3 stacks under the metal lid80 GB HBM3 · 3.35 TB/s memory bandwidthSingle host (HGX node)GPU1GPU2GPU3GPU4GPU5GPU6GPU7GPU8NVSwitch · 900 GB/s per GPUCPU + host RAM (PCIe Gen5 link to each GPU)8 GPU packages · full NVLink mesh · CPU + host RAM640 GB HBM total · 900 GB/s NVLink per GPUSingle rackrackToRswitchInfiniBandmultiple hosts in one cabinet · NIC fabric · top-of-rack switchTBs HBM aggregate · InfiniBand / RoCE RDMA · ~50 GB/s/node

The tiers

From smallest and fastest at the top, to largest and slowest at the bottom:

TierLocationCapacityBandwidthLatency
Registersper-thread, on-die~256 KB/SMtens of TB/s (aggregate)a few cycles
L1 / Shared (SRAMSRAMStatic Random-Access Memory — the on-chip scratchpad / L1+shared memory inside each SM. Tiny (~100s of KB per SM) but ~10× faster than HBM.See in glossary →)per-SM scratchpadup to 256 KB/SMtens of TB/s (aggregate)tens of cycles
L2 cacheshared across SMs50 MBTB/s-scalehundreds of cycles
HBM3HBMHigh-Bandwidth Memory — the DRAM stack soldered next to the GPU die. H100 SXM has 80 GB at ~3.35 TB/s.See in glossary → (VRAM)on the GPU package80 GB3.35 TB/s~400 cycles
NVLinkNVLinkNvidia’s high-speed GPU-to-GPU interconnect. H100 NVLink ≈ 900 GB/s per GPU — much faster than PCIe.See in glossary →GPU ↔ peer GPU—900 GB/sµs
PCIe Gen5PCIeThe bus between the GPU and the host (CPU/RAM/NVMe). PCIe Gen5 x16 ≈ 64 GB/s — far slower than HBM.See in glossary →GPU ↔ host CPU/RAM—~64 GB/sµs
Host DDR5 RAMon the CPU512 GB–2 TB~400 GB/s~100 ns
NVMe SSDlocal diskTBs~14 GB/s~50 µs
NIC (RDMARDMARemote DMA — letting one node’s NIC write directly into another node’s memory without involving the CPU. The basis of InfiniBand and RoCE.See in glossary → / InfiniBand)node ↔ node—~50 GB/s for a 400 Gb/s linkµs-scale

The spread from registers to NIC is about 2,000× in bandwidth and even more in latency. Every order of magnitude matters.

What lives where

In a running vLLM worker, here’s where each kind of byte typically sits:

  • Model weights: HBM. 16 GB for Llama-3-8B, ~140 GB for 70B (split across multiple GPUs).
  • KV cache: HBM. Whatever is left after weights.
  • Activations (intermediate per-step tensors): HBM, briefly. SRAM/registers during a kernel.
  • The kernel’s actual operands: SRAM and registers. The matrix multiplies happen here.
  • Idle KV pages from paused requests (if offload is enabled): host DDR5 RAM, reached over PCIe.
  • Other GPUs’ shards of the same model: in their HBM, reached over NVLink.
  • Other nodes’ shards: in their HBM, reached over the NIC via RDMA + GPUDirect.
  • Cold model files: on NVMe disk until first load.

Try the scenarios

The visual below traces where bytes go in five common operations. Click through them and watch the animated packet follow its hops up and down the hierarchy.

Where the bytes live (Nvidia H100)
Pick a scenario; an animated packet follows the hops bytes actually take. Bandwidth/latency numbers are H100 SXM5 ballpark — not exact.
The exact same weight read happens for every decode step — but the math done with those weights is only ~1 token worth. This is why decode is memory-bandwidth bound and why batching helps so much: read the weights once, use them N times.
Registers
per-thread, on-die
~256 KB / SM
~120 TB/s
~1 cycle
L1 / Shared (SRAM)
per-SM scratchpad
~228 KB / SM
~33 TB/s
~30 cycles
L2 cache
shared across SMs
60 MB
~12 TB/s
~250 cyc
HBM3 (VRAM)
on-package DRAM
80 GB
3.35 TB/s
~400 cyc
NVLink → peer GPU
GPU ↔ GPU
— (transit)
900 GB/s
~µs
PCIe Gen5 ↔ host
GPU ↔ CPU/RAM
— (transit)
~64 GB/s
~µs
Host RAM (DDR5)
on the CPU
512 GB–2 TB
~400 GB/s
~100 ns
NVMe SSD
local disk
4–60 TB
~14 GB/s
~50 µs
NIC (RDMA / IB)
node ↔ node
— (transit)
~50 GB/s
~1–10 µs
Step 1 of 2
RegistersL1 / Shared (SRAM)L2 cacheHBM3 (VRAM)NVLink → peer GPUPCIe Gen5 ↔ hostHost RAM (DDR5)NVMe SSDNIC (RDMA / IB)
HBM3 (VRAM) → L2 cache
3.35 TB/s · ~5 ms per layer pass
Notice the bandwidth column spans 120 TB/s at the top down to ~50 GB/s at the bottom — over 2,000× from registers to the NIC. Every order-of-magnitude jump down the tier list is where serving systems gain (or lose) most of their performance.

A few things to notice:

  • HBM is fast, but not “free.” The H100’s 3.35 TB/s is extremely high DRAM bandwidth, but it is still far below on-chip storage bandwidth and tensor-core arithmetic throughput. A CUDA kernel tries to reuse data after loading it from HBM so that each byte supports as much arithmetic as possible.

  • PCIe is a cliff. Going off the GPU is 50× slower than HBM. This is why KV cache offload exists but isn’t free: pulling a 128 MB KV slab back from host RAM takes 2 ms, on top of whatever queueing the request did.

  • NVLink is why high-performance multi-GPU serving works. Tensor parallelism requires layer-by-layer collective communication. NVLink offers much more bandwidth and lower latency than PCIe, which can otherwise become a major bottleneck. The exact communication time depends on activation size, collective algorithm, and topology.

  • The NIC can be the bottleneck for multi-node serving. A 400 Gb/s link provides roughly 50 GB/s before protocol overhead; newer interconnects can provide more, but remain far below local HBM bandwidth. GPUDirect RDMAGPUDirectNvidia tech that lets the NIC or NVMe DMA straight into/out of GPU HBM, bypassing host RAM.See in glossary → avoids a round-trip through host RAM by letting the NIC DMA straight to/from HBM, which is important at scale.

Why decode is memory-bound, made concrete

Here’s the back-of-envelope for Llama-3-8B at fp16:

  • Weights: 16 GB.
  • HBM bandwidth: 3.35 TB/s.
  • Time to read all weights once: 16/3,350≈4.816 / 3{,}350 \approx 4.8 ms.
  • Matrix-multiply work per decode step: ~16 GFLOP (1 token through 8B weights, ×2 operations per parameter).
  • H100 Tensor Core (TC) throughput (BF16): ~990 TFLOPS.
  • Time to do the math: 16/990,000≈0.01616 / 990{,}000 \approx 0.016 ms.

The math takes about 16 microseconds in this idealized calculation. The weights take 4.8 milliseconds to read. This roofline estimate shows why batch-1 decode has very low tensor-core utilization: HBM traffic, plus other real-world overheads, dominates.

So: read-heavy work, slow when you only have one user, much faster when you have many. The scheduler’s job is to fold “many users” into single GPU steps cleanly. That’s continuous batching, and it’s where serving stops being about the model and starts being about systems.