When to use tensor vs pipeline parallelism for LLM inference

Key Takeaways

• Single-node inference fails when total serving footprint (weights + KV cache + overhead) exceeds available VRAM, not just when weights don't fit
• Tensor parallelism requires high-bandwidth interconnects and collapses at node boundaries; pipeline parallelism tolerates higher latency but introduces bubble overhead
• The decision matrix depends on model size, interconnect topology, latency SLA, and traffic patterns, not raw GPU count

Most LLM inference tutorials assume your model runs on one server. That assumption breaks the moment you try serving a 70B dense model with long context windows, scaling a mixture-of-experts architecture, or handling bursty production traffic. At that point, inference stops being a loading-a-checkpoint problem and becomes a distributed systems problem. DigitalOcean's new guide on multi-node parallelism tackles the engineering question most tutorials skip: given your model size, hardware, latency requirements, and traffic shape, how do you actually choose your tensor parallelism (TP), pipeline parallelism (PP), and data parallelism (DP) degrees?

Why single-node serving fails before you expect it to

A 70B parameter model in BF16 needs roughly 140 GB just for weights. That's before you account for the KV cache, activation buffers, CUDA graph memory, NCCL buffer allocations, runtime metadata, and other serving overhead. Many teams calculate only the weight memory and assume deployment is feasible. In production, that math is incomplete.

Here's what actually consumes GPU memory during inference: model weights, KV cache, activation buffers, NCCL communication buffers, scheduler state, CUDA graphs, quantization metadata, speculative decoding buffers (if enabled), and a margin for fragmentation. The KV cache alone scales with context length, concurrency, and output tokens. A model that fits comfortably at 4K context might not fit at 64K or 128K under realistic traffic, even when your GPUs have spare compute cycles.

The practical threshold for multi-node inference isn't a parameter count. It's the point where your total serving footprint can't fit on one node with reliable headroom. If your deployment works on paper but leaves no memory margin, expect instability under bursty traffic or large prompts.

The three parallelism strategies and when each applies

Tensor parallelism splits computation within individual layers across GPUs. Each GPU holds a slice of the weight matrices and computes part of every forward pass. This requires all-reduce operations after every layer, which means you need fast interconnects. NVLink or InfiniBand works. Ethernet across nodes usually doesn't.

Pipeline parallelism assigns different groups of transformer layers to different GPUs or nodes. GPU 0 runs layers 0-15, GPU 1 runs layers 16-31, and so on. Communication happens only between adjacent stages, so it tolerates higher latency. The tradeoff: pipeline bubbles. Some GPUs sit idle while waiting for activations to flow through the pipeline.

Data parallelism replicates the entire model stack so independent requests process simultaneously. This scales throughput linearly but doesn't help when a single replica can't fit in memory.

Production systems typically combine all three. The question is which degrees to set for each.

Why tensor parallelism collapses across node boundaries

Tensor parallelism demands high-bandwidth, low-latency communication because it synchronizes after every layer. Within a node, NVLink provides 600+ GB/s bandwidth between GPUs. Across nodes, even fast InfiniBand tops out around 100 GB/s, and standard Ethernet is far worse.

When you stretch TP across node boundaries, the all-reduce operations that were microsecond-scale become millisecond-scale. Inference latency spikes. The GPU utilization you expected collapses because GPUs spend more time waiting for communication than computing.

This is why most production deployments keep tensor parallelism within a single node (TP=8 for 8-GPU nodes) and use pipeline parallelism to span nodes. The communication pattern of PP, point-to-point between adjacent stages, tolerates higher latency because it happens less frequently.

Pipeline parallelism trades bandwidth for bubbles

The bubble problem in pipeline parallelism is real. During the warm-up and cool-down phases of each micro-batch, some pipeline stages sit idle. With 4 pipeline stages, you might see 25-30% of GPU cycles wasted on bubbles in naive implementations.

Modern inference frameworks use micro-batching and interleaved scheduling to reduce bubble overhead. vLLM, TensorRT-LLM, and DeepSpeed all implement some form of this. But the overhead never fully disappears. You're trading communication bandwidth requirements for computational efficiency.

The decision comes down to your bottleneck. If you have fast interconnects (NVLink everywhere), higher TP degrees work. If you're spanning nodes over Ethernet or slow InfiniBand, PP becomes necessary despite the bubbles.

The decision matrix: model size, interconnect, and SLA

There's no universal answer, but the guide provides a useful framework. Start with three variables: model size (total serving footprint, not just weights), interconnect topology (NVLink within node, InfiniBand or Ethernet across nodes), and latency SLA (p99 latency requirements from your product).

• If the model fits on one node with 20%+ memory headroom: stay single-node, use TP=8 for 8-GPU systems
• If the model spans two nodes but interconnect is fast InfiniBand: consider TP=8 within each node, PP=2 across nodes
• If interconnect is Ethernet or slow: maximize PP to minimize cross-node communication frequency
• For throughput-optimized workloads with relaxed latency: add DP replicas once your TP+PP config is stable

Framework defaults matter here. vLLM and TensorRT-LLM make different assumptions about parallelism configuration. Check what your framework defaults to before deploying, and benchmark with your actual traffic patterns.

Network topology beats GPU count

One counterintuitive finding: adding more GPUs doesn't always improve performance. If those GPUs are connected over slow fabric, you might be better off with fewer GPUs on faster interconnects. Eight GPUs on NVLink can outperform 16 GPUs spanning two nodes over Ethernet for latency-sensitive workloads.

This is why cloud providers like DigitalOcean are investing in high-bandwidth GPU-to-GPU networking. The compute isn't the bottleneck for large models. The communication is.

Frequently Asked Questions

What's the difference between tensor parallelism and pipeline parallelism?

Tensor parallelism splits individual layer computations across GPUs, requiring high-bandwidth synchronization after every layer. Pipeline parallelism assigns different layer groups to different GPUs, with communication only between adjacent stages. TP needs fast interconnects; PP tolerates higher latency but introduces bubble overhead.

When should I use multi-node inference instead of single-node?

When your total serving footprint (weights + KV cache + runtime overhead) can't fit on one node with 20%+ memory headroom. Don't just calculate weight memory. A 70B model that fits on paper might fail under long context or high concurrency.

How do I reduce pipeline parallelism bubble overhead?

Use micro-batching and interleaved scheduling, which modern frameworks like vLLM and TensorRT-LLM implement automatically. Bubbles never fully disappear, but good scheduling can reduce overhead from 30% to under 10%.

Does adding more GPUs always improve LLM inference performance?

No. If additional GPUs are connected over slow fabric (Ethernet vs NVLink), communication latency can negate the compute gains. Eight GPUs on fast interconnects often beat 16 GPUs with slow cross-node links for latency-sensitive workloads.