II · THE IDEA · ARTIFICIAL INTELLIGENCE
Model Parallelism with Pipeline Bubbles
▶ Listen · narrated
A model too large for one device must be cut into stages. The cost is not only communication: whole stretches of the pipeline can sit waiting while other stages finish.
At a glance
- Problem
- Devices idle at the start and end of each mini-batch
- Mechanism
- Split each mini-batch into micro-batches and pipeline them
- Schedule
- Forward passes fill the pipe; backward passes drain it
- Bubble cost
- Grows with stages; shrinks as micro-batches increase
- Sync style
- Synchronous updates after all micro-batches complete
Think of a kitchen pass with four cooks in a line. Each dish must visit cook 1, then 2, then 3, then 4. If you send only one dish at a time, cooks 2–4 stand idle until the plate reaches them, and cook 1 stands idle once the plate has moved on and is waiting for the “all clear” from the far end.
Now send a tray of many small plates one after another. As soon as cook 1 finishes plate 1, plate 2 starts. The line fills, and for a long stretch every cook is busy. At the very start and very end there are still quiet moments — the pipeline bubbles — but they are a smaller fraction of the whole service.
GPipe does the same for a neural network split across devices. A training mini-batch is cut into micro-batches. Forward passes of those micro-batches fill the pipeline; backward passes drain it. Gradients add up, and weights update once per mini-batch, so the maths matches ordinary training. The empty gaps never fully disappear; they just stop dominating the clock.
GPipe partitions a network into K consecutive stage cells mapped to devices and splits each mini-batch into M micro-batches. The default schedule performs the forward pass of micro-batch 1…M in order (pipeline fill), then the backward passes in reverse dependency order (pipeline drain), accumulating gradients so the parameter update is synchronous and equivalent to a single mini-batch step.
Bubble overhead is the idle fraction implied by the fill and drain tails. Under a balanced-stage model it scales with (K − 1) relative to the M micro-batch continuum: increasing M for fixed K improves utilisation; increasing K for fixed M worsens it. This is schedule geometry, independent of kernel efficiency.
Activation memory would grow with M if all forwards were cached. GPipe uses re-materialisation (gradient checkpointing): discard most forward activations and recompute them during the backward pass from retained checkpoints. That raises FLOPs per step but caps activation residency roughly in line with depth per stage rather than M, which is what makes large M practical.
Limitations: uneven layer costs re-introduce bubbles as load imbalance; collectives and peer transfers add latency not captured by the K–M idealisation; and re-materialisation’s recompute cost can erase throughput gains if stages are already compute-bound. The method does not change optimiser semantics; it changes when each device has useful work.
Look closer
Where the empty slots appear
Partition a network into consecutive stage cells, one per accelerator. The first stage can start the next micro-batch as soon as it finishes the previous forward pass, but the last stage has nothing to do until the first micro-batch has traversed every earlier cell. At the far end of the mini-batch the reverse happens: early stages finish and wait while later stages complete their backward work. Those waiting intervals are the pipeline bubbles — visible as diagonal bands of idle time on a device–time diagram.
Why micro-batches help
A single mini-batch moving as one unit leaves almost every device idle almost all of the time. Splitting that mini-batch into M micro-batches lets stage k start micro-batch i+1 while stage k+1 still works on micro-batch i. The startup and shutdown tails stay roughly proportional to the number of stages K; the busy middle grows with M. GPipe’s analysis treats the relative bubble overhead as shrinking when M is chosen comfortably larger than K, so hardware spends most of the step doing real compute rather than waiting for neighbours.
Forward fill, backward drain
In the GPipe schedule the pipeline is first filled with forward passes of successive micro-batches, then drained with the corresponding backward passes. Gradients from every micro-batch are accumulated, and parameters update once per mini-batch, so the mathematical result matches ordinary synchronous training. Memory pressure is further eased by re-materialisation: activations from the forward pass need not all be retained; they can be recomputed during the backward pass at the cost of extra compute. The bubble pattern and the re-materialisation choice are separate levers — one about utilisation across devices, the other about fitting a deeper cell on each device.
The story
When a neural network no longer fits in the memory of a single accelerator, one practical response is to cut it into consecutive groups of layers and place each group on its own device. Activations flow forward along this chain; gradients flow back. That arrangement is model parallelism in its pipeline form. It sounds efficient until you watch a clock: the second device has nothing to compute until the first device finishes its piece of the current batch, and the first device has nothing useful to do while it waits for gradients to return from the far end. The resulting idle stretches are pipeline bubbles.
GPipe, described in 2018, makes the bubble a first-class scheduling object rather than an embarrassment to be ignored. The training mini-batch is divided into micro-batches of equal size. As soon as device 1 finishes the forward pass for micro-batch 1, it begins micro-batch 2, while device 2 works on what device 1 just sent. The pipeline fills. After the forwards, backward passes propagate in the reverse direction, accumulating gradients across micro-batches. Parameters are updated only once the whole mini-batch is complete, so the optimisation step remains synchronous and equivalent to training the same network on a single device with the same mini-batch size.
The bubble does not vanish. At the start of each mini-batch the later stages still wait for work to arrive; at the end the earlier stages still wait for the backward wave to finish. What changes is the proportion. With K pipeline stages and M micro-batches, the relative cost of those tails shrinks as M grows relative to K. Choose M only a little larger than K and much of the step is still idle time; choose M comfortably larger and the devices spend most of the step on real forward and backward compute. Communication still happens at stage boundaries, but the dominant waste GPipe targets is compute left on the table while a neighbour finishes.
Memory is a second constraint. A naive pipeline would hold activations for every micro-batch until its backward pass, which quickly exhausts device memory as M grows. GPipe pairs the schedule with re-materialisation: during the backward pass, needed activations are recomputed from saved checkpoints rather than stored in full. That trades extra FLOPs for a lower activation footprint, so each stage can be deeper and the number of devices needed for a given model can fall. The two ideas work together — micro-batching attacks the bubble; re-materialisation attacks the memory wall that would otherwise limit how far micro-batching can go.
The editorial point is modest but practical. Pipeline parallelism is not only about where layers sit; it is about the order in which forward and backward work is issued across those places. Interleaving micro-batch forwards to fill the pipe, then draining with backwards, is a concrete schedule that improves utilisation without changing the loss, the gradients’ mathematical definition, or the synchronous character of the update. The bubble remains visible on a timeline. It is simply smaller relative to the useful work.
Why it mattered then
By the late 2010s, model depth and width were outrunning single-accelerator memory. Data parallelism alone still required each replica to hold the full parameter set. Pipeline parallelism offered a way to place different layer groups on different devices, but naive schedules left accelerators idle for large fractions of each step. GPipe’s contribution in that moment was a practical recipe — micro-batches, a fill-then-drain schedule, gradient accumulation for synchronous updates, and re-materialisation for activation memory — that made pipeline training of very deep networks workable on then-current hardware without abandoning standard optimisers or exact mini-batch semantics.
Why it matters now
Models continue to be partitioned across devices whenever a single chip cannot hold weights, optimiser state and activations together. Pipeline bubbles still appear whenever stages are chained, and the same levers still apply: more micro-batches relative to stages reduce relative idle time; recomputing activations still trades compute for memory; synchronous accumulation still preserves a familiar training dynamic. Understanding why empty slots open at the start and end of a mini-batch remains useful when reading utilisation plots, choosing micro-batch counts, or deciding whether a pipeline stall is a communication problem or simply the geometry of the schedule.
The surprising detail
The bubble is not a communication failure and not a bug in the partition. Even with instantaneous transfers, K stages need a startup wave before the last device is busy and a shutdown wave before the first device can finish the mini-batch. Micro-batching does not remove those waves; it only buries them under a longer stretch of steady work. The idle time is structural — a consequence of depth in time as much as depth in layers.
What is disputed
GPipe’s published analysis frames bubble overhead in terms of stage count and micro-batch count under a particular synchronous schedule. Real systems also pay for communication volume, load imbalance across uneven stages, and framework overhead; those effects can dominate the simple K-versus-M picture when stages are poorly balanced or interconnects are slow. Treat the bubble formula as a schedule-geometry lower bound on idle time, not a full performance model.
Remember this
Pipeline bubbles are idle tails fixed by stage count; micro-batches stretch the busy middle so those tails matter less.
Test yourself
You double the number of pipeline stages K while holding the mini-batch size and the micro-batch count M fixed. What happens to relative bubble overhead, and what single counter-adjustment does GPipe’s reasoning suggest if you want utilisation to recover?
Relative bubble overhead rises, because the startup and shutdown tails scale with the number of stages while the amount of micro-batch work stays the same. To recover utilisation you increase M — more micro-batches per mini-batch — so the busy steady-state region grows again relative to those tails. (You must still watch activation memory; re-materialisation is the usual companion when M grows.)
Go deeper
- [1811.06965] GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism · arxiv.org
- [1911.05946] A Scalable Approach for Facial Action Unit Classifier Training UsingNoisy Data for Pre-Training · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.