Scaling out

Tensor parallelism vs pipeline parallelism

Sources: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Shoeybi et al., 2019; The Llama 3 Herd of Models — Grattafiori et al., 2024

For models that fit on one GPU, the previous sections cover the whole story. But Llama-3-70B at fp16 is about 140 GB, while an 80 GB H100 cannot hold those weights plus serving memory. At that precision on that hardware, you need multiple GPUs or another strategy such as lower-precision weights. How you split the model across GPUs determines what the math looks like and what interconnect you need.

There are two main strategies, with combinations on top of them.

Tensor parallelism: split every matrix

Tensor parallelism (TP)tensor parallelismSplitting each weight matrix across N GPUs. Every GPU does a slice of every layer; activations get all-reduced across them.See in glossary → shards most large weight matrices across $N$ GPUs. With $N = 4$, each GPU holds roughly one quarter of the sharded weights in every layer, while some small tensors may be replicated. The forward pass works like this:

1. Each GPU receives the same input activations (the residual stream).
2. Each GPU multiplies by its slice of the weight matrix, producing its slice of the output.
3. After certain operations (specifically: at the end of attention’s output projection and the MLP’s down-projection), the slices need to be summed across all GPUs to reconstruct the full output activations.

That summation is an all-reduceall-reduceA collective op where every GPU contributes a tensor and every GPU ends up with the sum (or other reduction). The TP workhorse.See in glossary →: every GPU sends its slice, every GPU receives the sum.

For a common Megatron-style Llama block, you do two all-reduces per layer (one after attention, one after the MLP). Their time depends on activation size, collective algorithm, topology, and batch shape; there is no universal 100 µs figure. NVLink or NVSwitch usually provides much better TP performance than PCIe, but PCIe-only TP can still be viable for some models and workloads.

Pipeline parallelism: split by layer

Pipeline parallelism (PP)pipeline parallelismSplitting the model layer-wise across GPUs. Each GPU owns a contiguous slab of layers; activations flow from one to the next.See in glossary → takes a different cut: each GPU owns a contiguous block of layers. For a 32-layer model on 4 GPUs, GPU 0 has layers 0–7, GPU 1 has layers 8–15, and so on. A token’s activations flow through GPU 0, then over the interconnect to GPU 1, then to GPU 2, …, then back to GPU 0 for the next token.

The interconnect cost per token is much smaller: you ship the residual stream once per stage boundary, not all-reduce per layer. This makes PP a good fit for cross-node scaling (over RDMA, where bandwidth is lower than NVLink).

The catch is pipeline bubbles. Dependent decode tokens from one autoregressive request cannot be pipelined ahead of one another, because token $T$ must be sampled before token $T+1$ begins. A pipeline can instead stay busy by interleaving independent requests or prefill microbatches; when it cannot, some stages are idle. PP is therefore more complex than TP and usually adds per-token latency because each token traverses every pipeline stage.

Combining them: TP within a node, PP across nodes

Many multi-node setups combine strategies:

• Inside a node (8 GPUs sharing NVSwitch): TP=8. The all-reduces ride NVLink.
• Across nodes (multiple racks): PP=N. Activations cross the InfiniBand fabric once per stage boundary via GPUDirect RDMA.

For very large models, the chosen combination depends on model architecture, precision or quantization, GPU memory, topology, and latency target. Mixture-of-experts models introduce further choices such as expert parallelism.

Data parallelism for throughput

The third axis is data parallelism (DP): just replicate the model and serve different requests on different replicas. This adds throughput linearly until you exhaust GPUs, with zero interconnect cost between replicas (they don’t talk to each other for inference; they share nothing).

In a multi-tenant deployment, DP is the most common shape on top of TP: “8 GPUs per replica via TP, 4 replicas via DP.” A load balancer routes requests to replicas; within each replica vLLM does its KV management.

What this means for memory

Each parallelism mode changes where bytes live:

• TP: each GPU holds roughly its shard of the model weights and the KV-cache shard for its share of attention heads.
• PP: each GPU holds (model weights / N) + a KV cache for the layers it owns across all active requests.
• DP: each replica holds its model partition and its own KV cache; in the simplest one-GPU replica, that is a full model copy.

The KV cache split is subtle. With TP, the KV cache is also sharded: each GPU stores the K/V values for its share of the heads. With PP, each GPU stores all heads but only for its layers.

The interconnect, revisited

Reviewing the memory hierarchy with these parallelism strategies in mind:

Pick the “Tensor-parallel all-reduce across 8 GPUs” scenario to see what TP looks like at the byte level: HBM → NVLink → peer GPU HBM, twice per layer. Pick “Multi-node: weights / activations cross the NIC” for PP: HBM → NIC → fabric → peer node’s HBM.

The takeaway: you do not pick a parallelism strategy by elegance. You pick it by the speed of the slowest interconnect that has to participate in the per-step critical path.

• All-reduce per layer benefits greatly from a fast local interconnect.
• Activations once per stage can traverse RDMA fabric when the added latency and bandwidth cost fit the deployment.
• Replicas can be placed independently, subject to application latency and routing requirements.

With one or many GPUs, many of the scheduling ideas from sections 14–18 still apply: paged KV management, continuous batching, prefix caching, chunked prefill, and speculative decoding where supported by the configuration. The model is bigger, but the same memory and scheduling principles remain relevant.

One last thing before we wrap up: a closing look at the throughput-vs-latency tradeoff and the knobs you actually turn in production.