II · THE IDEA · ARTIFICIAL INTELLIGENCE
Splitting a Model Across Machines
▶ Listen · narrated
The largest open models exceed half a trillion parameters. No consumer card holds them whole, so running one locally means choosing how to divide work—the wrong choice turns hardware into a waiting room.
At a glance
- Tensor parallelism
- Split each matrix operation across devices; all communicate every layer
- Pipeline parallelism
- Assign whole layers to devices; each waits for the previous stage to finish
- Layer-wise sharding
- Load contiguous layer groups on separate devices; pass activations once per group
- Bandwidth requirement
- Tensor parallelism needs high-speed interconnect; pipeline and layer-wise tolerate slower links
Imagine you need to move a large pile of bricks from one side of a field to the other, but the pile is too heavy for one person to carry. You have three options. First, you could split each load: several people carry parts of the same load at the same time, meeting in the middle to hand off their portions. This is fast, but only if everyone can walk side by side — if the path is narrow, they will collide and slow down. Second, you could form a relay: the first person carries a load partway, sets it down, and while they walk back for the next load, the second person carries the first load onward. This works on a narrow path, but if you only have one brick to move, most people are standing around waiting. Third, you could have one person carry the entire brick all the way, then the next person carries the next brick. Simple, works on any path, but only one person works at a time. Splitting a model across machines is the same problem: you can split the work inside each layer (tensor parallelism, needs a wide path), split the model into stages and stream batches through (pipeline parallelism, needs multiple bricks in motion), or just pass the work from one device to the next (layer-wise sharding, one device active at a time).
Tensor parallelism partitions weight matrices and splits matrix multiplications across devices. For a linear layer with weight matrix W and input X, each device holds a portion of W and computes its portion of the output Y = XW. This requires an all-reduce operation to sum the partial results, and because every layer involves such operations, communication happens continuously. Megatron-LM splits both the attention QKV projection and the feed-forward layers, using column-wise partitioning for some and row-wise for others to minimise communication. The method achieves near-linear scaling when devices are connected by high-bandwidth interconnects like NVLink or InfiniBand, but performance degrades sharply when bandwidth is limited because the all-reduce becomes a bottleneck.
Pipeline parallelism divides the model into stages, assigning contiguous layers to each device. A batch is split into micro-batches, and the system schedules them so that while device k processes micro-batch i, device k+1 processes micro-batch i-1. GPipe introduced a fill-drain schedule that minimises idle time, but there is always a pipeline bubble at the start and end where some devices are idle. The bubble size is proportional to the number of stages and inversely proportional to the number of micro-batches, so large batches are essential for good utilisation. Communication happens only at stage boundaries, so bandwidth requirements are lower than tensor parallelism, but the method is poorly suited to inference workloads where batch sizes are small.
Layer-wise sharding is a degenerate case of pipeline parallelism with one micro-batch. Each device processes its assigned layers in sequence, then passes activations to the next device. At any moment, only one device is active. The method is simple to implement and works over any interconnect, including slow network links, because communication is infrequent — once per stage boundary, not once per layer. The cost is utilisation: if you split a model across N devices, N-1 are idle at all times, and total latency is the sum of per-device latencies plus communication overhead. For local inference, where you cannot fill a pipeline and often lack high-bandwidth interconnects, this is the default choice despite its inefficiency.
Look closer
Tensor parallelism splits the arithmetic inside a single layer
A large matrix multiplication — say, transforming a hidden state through a feed-forward layer — can be divided so that each GPU computes part of the output. The input activations must be broadcast to all devices, each performs its fraction of the work, then the partial results are gathered and summed. This happens at every layer, so the devices communicate constantly. Megatron-LM demonstrated this at scale, training models with billions of parameters by splitting attention and feed-forward operations across multiple GPUs connected by NVLink, NVIDIA's high-bandwidth interconnect.
Pipeline parallelism assigns whole layers to different devices and streams the batch
Instead of splitting operations, you split the model vertically: the first few layers live on GPU zero, the next few on GPU one, and so on. A batch is divided into smaller micro-batches that flow through the pipeline in sequence, so while GPU two is processing micro-batch three, GPU one is working on micro-batch four. GPipe introduced this scheme, showing that careful scheduling of these micro-batches could keep all devices busy most of the time. Communication happens only at layer boundaries, so bandwidth requirements are much lower than tensor parallelism, but devices spend time idle waiting for the previous stage unless the batch is large enough to keep the pipeline full.
Layer-wise sharding is pipeline parallelism with a batch size of one
You place contiguous layer groups on separate devices and pass activations from one to the next, but you process only a single sequence at a time. There is no pipelining, no micro-batching, and most of the time only one device is working while the others wait. It is the simplest scheme to implement and tolerates slow interconnects — even CPU-to-CPU links over a network — because communication is infrequent. The cost is poor utilisation: if you have four devices, three are idle at any moment. For inference on a single prompt, where you cannot fill a pipeline, this is often the only practical choice when the model does not fit on one machine.
The story
When a model grows beyond the memory of a single accelerator, you must distribute it. Three strategies have emerged, each trading off bandwidth, latency and utilisation differently.
Tensor parallelism divides the work inside each layer. A matrix multiplication that would normally happen on one device is split so that each device computes a portion of the result. The devices must exchange data before and after every operation: inputs are broadcast, outputs are gathered and summed. This means tensor parallelism demands a fast link between devices. Megatron-LM, published by researchers at NVIDIA, used this approach to train models with multiple billions of parameters by splitting both the attention mechanism and the feed-forward layers across GPUs connected by NVLink, which offers substantially higher bandwidth than PCI Express. The method scales well when the interconnect is fast, but performance collapses over slower links because communication becomes the bottleneck. Every forward pass and every backward pass during training requires multiple all-reduce operations, and if the time to move data between devices exceeds the time to compute, the GPUs spend more time waiting than working.
Pipeline parallelism takes a different approach. Instead of splitting operations, you split the model vertically: the first few layers run on one device, the next few on another, and so on. A batch of inputs is divided into smaller micro-batches that flow through the pipeline in sequence. While one device processes micro-batch three, the next device works on micro-batch four. GPipe demonstrated that with careful scheduling, this could keep all devices busy most of the time, training models too large for any single accelerator. Communication happens only at layer boundaries — one device sends its output activations to the next — so bandwidth requirements are much lower than tensor parallelism. The trade-off is utilisation: unless the batch is large enough to fill the pipeline, devices sit idle waiting for the previous stage. Training benefits from large batches, but inference often processes one prompt at a time, which makes pipeline parallelism less attractive there.
Layer-wise sharding is the simplest scheme and the one most commonly used for local inference. You place contiguous groups of layers on separate devices and pass activations from one to the next, processing a single sequence with no attempt at pipelining. At any given moment, only one device is working while the others wait. If you have split a model across four GPUs, three are idle at all times. Utilisation is poor, but the method tolerates slow interconnects — even CPU-to-CPU links over a network — because communication is infrequent. For someone running a large model on consumer hardware, often across mismatched devices, this is the only practical option. The alternative schemes require either high bandwidth or large batches, neither of which a local setup is likely to provide.
The choice of strategy depends on what you are optimising for. Tensor parallelism maximises utilisation when the interconnect is fast, making it the preferred method for training clusters with purpose-built networking. Pipeline parallelism works when you have large batches and can tolerate some idle time, and it needs less bandwidth. Layer-wise sharding sacrifices utilisation entirely in exchange for working on any hardware, which makes it the default for inference on models that do not fit on one device. The interconnect is the deciding factor: a slow link between devices rules out tensor parallelism and limits how well pipeline parallelism can perform, leaving layer-wise sharding as the only viable choice despite its inefficiency.
Why it mattered then
Megatron-LM and GPipe were both published in 2019, at a moment when model size was increasing faster than single-GPU memory. Training a billion-parameter model required either buying more expensive hardware or finding a way to distribute the work across cheaper devices. Tensor parallelism worked well within a single machine where NVLink provided high bandwidth between GPUs, and it became the standard approach for training large models in data centres. Pipeline parallelism offered a way to scale across machines connected by slower network links, though it required large batches to keep the pipeline full. Both methods were designed primarily for training, where batches are large and gradients must flow backward through the entire model. The papers demonstrated that distribution was practical, not just possible, and that careful attention to communication patterns could make multi-device training competitive with single-device speeds.
Why it matters now
The techniques remain relevant because models have continued to grow and because the hardware landscape has diversified. Cloud providers offer multi-GPU instances with fast interconnects, making tensor parallelism practical for training and for high-throughput inference. Meanwhile, the availability of open-weight models has created a new use case: individuals running very large models on consumer hardware, often splitting them across mismatched devices or even across separate machines on a local network. For these users, layer-wise sharding is often the only option, and understanding why it is so inefficient helps explain why local inference on large models is slow. The rise of mixture-of-experts architectures has also created hybrid cases where different experts can be placed on different devices, blending aspects of all three strategies. The fundamental trade-off — between communication cost, utilisation, and the flexibility to run on varied hardware — has not changed, and anyone working with models too large for a single device must still choose which costs to pay.
The surprising detail
Pipeline parallelism introduces a subtle problem called the pipeline bubble. At the start of processing a batch, the first device begins working while all others are idle. As micro-batches flow through, more devices become active, but there is always a ramp-up period where the pipeline is filling and a ramp-down period where it is draining. During these phases, some devices are idle, and the larger the number of pipeline stages, the larger the bubble. GPipe's key contribution was a scheduling scheme that minimised this waste, but it could not eliminate it entirely. The bubble is why pipeline parallelism needs large batches: only when the pipeline is full for most of the time does the utilisation approach that of tensor parallelism. For inference, where you often process one sequence at a time, the bubble is the entire workload, which is why pipeline parallelism is rarely used there despite its lower bandwidth requirements.
Remember this
Tensor parallelism needs fast links because it communicates every layer. Pipeline and layer-wise sharding communicate less often, but layer-wise leaves most devices idle most of the time.
Test yourself
You have two machines, each with one GPU, connected by a one-gigabit Ethernet link. You want to run a 70-billion-parameter model that does not fit on one GPU. Which parallelism strategy is practical, and what specific behaviour will you observe during inference?
Layer-wise sharding is the only practical choice. Tensor parallelism would require communicating large tensors after every layer, and a gigabit link is far too slow for that — the GPUs would spend most of their time waiting for data to arrive. Pipeline parallelism requires multiple micro-batches in flight to keep both devices busy, but during inference you typically process one prompt at a time, so one device would be idle while the other works. With layer-wise sharding, you will observe that one GPU is active while the other is idle, then the active one goes idle while the second one works. The model will generate tokens, but slowly, because only half your hardware is ever in use. You will also notice pauses at layer boundaries as activations are sent over the network. The method works, but it is inefficient, and the slow link makes it slower still.
Go deeper
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism · arXiv · Mohammad Shoeybi et al. · 2019-09-17
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism · arXiv · Yanping Huang et al. · 2018-11-16
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.