Skip to content
The Daily Triptych084 / 365
Pipeline parallelism across three machines

Each machine processes its layers, then sends activations to the next. The network crossing between machines adds tens of milliseconds per token, far more than the microseconds typical of memory transfers within a single GPU.

Try it in the local lab

Estimate the network overhead for your own setup

If you have two machines on the same network, you can measure the round-trip time for a realistically sized tensor to understand whether distributed inference latency would be acceptable for your workload. This does not require a model, just Python and network access between the machines.

$ # On machine A, run a simple server that echoes data back:
# Save this as echo_server.py
import socket
import sys

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('0.0.0.0', 9999))
s.listen(1)
print('Listening on port 9999')
conn, addr = s.accept()
print(f'Connected by {addr}')
while True:
    data = conn.recv(1048576)  # 1MB chunks
    if not data: break
    conn.sendall(data)
conn.close()
$ # On machine B, send a tensor-sized payload and measure round-trip time:
# Save this as measure_latency.py
import socket
import time
import sys

host = sys.argv[1]  # IP of machine A
size_mb = int(sys.argv[2]) if len(sys.argv) > 2 else 10

payload = b'x' * (size_mb * 1024 * 1024)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, 9999))

start = time.time()
s.sendall(payload)
received = b''
while len(received) < len(payload):
    chunk = s.recv(1048576)
    if not chunk: break
    received += chunk
elapsed = time.time() - start

print(f'Sent and received {size_mb}MB in {elapsed*1000:.1f}ms')
print(f'Effective bandwidth: {(size_mb*2/elapsed):.1f} MB/s')
s.close()
$ # Run on machine A:
python3 echo_server.py
$ # Run on machine B (replace 192.168.1.10 with machine A's IP):
python3 measure_latency.py 192.168.1.10 10

A typical activation tensor between layers in a large model is 5-20 MB for a single sequence. Multiply the measured round-trip time by the number of network crossings (devices minus one) to estimate the per-token network overhead. If you plan to batch, divide by your batch size. This gives you a floor for latency; actual inference adds computation time on top.

II · THE IDEA · ARTIFICIAL INTELLIGENCE

A Cluster of Personal Computers

Hardware and local inference · Pipeline and tensor · Seconds per token, not milliseconds

▶ Listen · narrated

The largest open-weight models will not fit on one machine. Splitting them across several is possible, but the network between them becomes part of the forward pass.

At a glance

What it enables
Running models too large for a single consumer GPU's memory
The cost
Network round-trips between layers, adding seconds rather than milliseconds
When it is worth it
Batch inference, translation, summarisation — anything that does not require conversational speed
Common approach
Pipeline parallelism: consecutive layers on different machines

Imagine a factory assembly line where each station adds a part to a product. If all the stations are in one building, the product moves between them on a conveyor belt in seconds. If the stations are in three separate buildings across town, the product must be driven from one to the next, and each trip takes ten minutes. The work at each station is the same, but the total time is much longer because of the travel.

A neural network model is a sequence of layers, each performing a mathematical transformation on the data. Normally all the layers live on one graphics card, and data moves between them through the card's memory almost instantly. But if the model is too large to fit on one card, you can split the layers across several cards in different computers. The data must then travel over the network between computers, which is thousands of times slower than moving through memory on a single card. This added travel time means generating each word takes much longer. It is acceptable if you are processing many documents at once and can wait, but not if you need quick responses in a conversation.

Look closer

  1. The network is now in the critical path

    In a single-machine model, data moves between layers through GPU memory at hundreds of gigabytes per second. When layers live on separate machines, that movement happens over Ethernet or InfiniBand, typically at one to ten gigabytes per second under real conditions. Every layer boundary crossed by the network adds a round-trip measured in milliseconds or tens of milliseconds. For a model with dozens of layers, those waits compound. The result is that generating one token may take several seconds rather than the tens of milliseconds typical of a local single-GPU setup.

  2. Pipeline parallelism splits the model vertically

    The simplest distribution strategy assigns consecutive layers to different machines. Machine A holds layers one through twelve, machine B holds thirteen through twenty-four, and so on. A token's activations flow through A, cross the network to B, continue through B's layers, and eventually produce an output. This is called pipeline parallelism. It was formalised in systems like GPipe, which also introduced micro-batching to keep all machines busy rather than waiting idle while one processes. The alternative, tensor parallelism, splits individual weight matrices horizontally across machines and requires even tighter coordination.

  3. Batch size determines whether the overhead is tolerable

    If you generate one token at a time for a single user, most of the wall-clock time is spent waiting for data to cross the network. But if you process a batch of thirty prompts together, all thirty share the same round-trips, and the cost per token drops substantially. This is why distributed consumer setups are practical for offline workloads — translating a document, summarising a corpus, running evals — but rarely for interactive chat. The latency is the same either way; batching just amortises it over more useful work.

The story

A seventy-billion-parameter model in sixteen-bit precision requires roughly one hundred and forty gigabytes of memory just to hold the weights, before accounting for activations or the key-value cache. No consumer graphics card offers that much. But three cards with forty-eight gigabytes each do, if the model can be split among them.

The simplest way to split a model across machines is to assign consecutive layers to each one. Machine A loads the first third of the layers, machine B the second third, machine C the final third. When you prompt the model, the input tokens are embedded and processed through A's layers. The resulting activations — a tensor of floating-point numbers representing the intermediate state — are serialised and sent over the network to machine B. B processes them through its layers and sends the result to C. C produces the final logits, samples a token, and the process repeats for the next token in the sequence.

This approach is called pipeline parallelism, and it was developed for training very large models in data centres. Systems like GPipe, published by researchers at Google in 2019, and Megatron-LM, from NVIDIA, formalised the techniques and introduced micro-batching to keep all machines working in parallel rather than sitting idle. The same principles apply when the machines are consumer desktops connected by Ethernet rather than server nodes linked by InfiniBand, but the performance characteristics change sharply.

In a data centre, the network between machines is often InfiniBand or a similar fabric running at twenty-five to two hundred gigabits per second, with latencies under ten microseconds. In a home or office, the network is typically gigabit or ten-gigabit Ethernet, with latencies measured in hundreds of microseconds or low milliseconds. That difference matters. Sending a tensor of a few megabytes over gigabit Ethernet takes tens of milliseconds. If the model has forty layers and you have split it across three machines, every token requires at least two network crossings, adding perhaps fifty to one hundred milliseconds of pure transfer time before any computation happens. For a conversational model expected to produce tokens every twenty or thirty milliseconds, this is prohibitive.

But if you are not generating tokens one by one for a waiting user, the calculus changes. Suppose you are translating fifty documents, or running a benchmark suite, or summarising a week of meeting transcripts. You can process all fifty prompts as a batch. The activations for all fifty flow together from A to B to C. The network round-trip happens once per layer per batch, not once per token. The overhead is the same in absolute terms, but it is now amortised over fifty outputs instead of one. Total throughput — tokens per hour across the entire batch — can be quite high, even though latency per token remains poor.

The alternative to pipeline parallelism is tensor parallelism, where individual weight matrices are split horizontally across machines. Each machine holds a slice of every layer, and they must synchronise during the forward pass within each layer. This requires more frequent communication and is generally only practical with very fast interconnects. For consumer setups, pipeline parallelism is the usual choice.

The software to orchestrate this exists, though it is less polished than single-machine inference engines. Frameworks like Megatron-LM and DeepSpeed support pipeline parallelism, and there are open-source projects that adapt these techniques specifically for distributed inference on consumer hardware. The setup is not trivial — you must partition the model correctly, configure network addresses, and manage the batch queue — but it is within reach of someone comfortable with command-line tools and Python environments.

Why it mattered then

Pipeline parallelism was developed to train models that were too large to fit on a single accelerator, even in well-funded research labs. By 2019, models with billions of parameters were becoming common, and the largest experimental models were pushing toward a trillion parameters. No single GPU could hold them. Splitting the model across multiple GPUs within a single machine helped, but even that had limits. The next step was splitting across multiple machines, which required rethinking how data flowed through the network during both forward and backward passes. GPipe introduced the idea of micro-batching to keep all stages of the pipeline busy, rather than having most of the hardware sit idle while one stage processed. Megatron-LM combined pipeline parallelism with tensor parallelism and demonstrated that the combination could scale to models with hundreds of billions of parameters, trained on clusters of hundreds of GPUs. These techniques were not designed for inference, and they were certainly not designed for consumer hardware, but the underlying mathematics is the same whether the machines are in a data centre or on a desk.

Why it matters now

The release of open-weight models with tens or hundreds of billions of parameters has made distributed inference relevant outside research labs. A small organisation or an individual researcher may want to run a seventy-billion-parameter model for evaluation, fine-tuning experiments, or batch processing, but cannot afford a single card with enough memory. Using several older or cheaper cards becomes an option if the latency penalty is acceptable. The technique is also relevant for hobbyist clusters — people who have accumulated several consumer GPUs over time and want to put them to use together. The latency means this is not a replacement for a fast single-GPU setup when you need interactive performance, but it expands the range of models you can run at all. It is also a reminder that the boundary between training infrastructure and inference infrastructure is not absolute. Techniques developed for one can often be adapted for the other, though the trade-offs shift.

The surprising detail

The network overhead is not evenly distributed across the generation process. The first token — the prefill phase, when the entire prompt is processed at once — can often be quite fast, because the batch size is effectively the prompt length and all that data is moving through the pipeline together. It is the subsequent tokens, generated one by one in the decode phase, that suffer most from the latency. This creates a strange user experience: the model appears to think quickly, then produces output very slowly. Some distributed inference setups therefore use speculative decoding or other techniques to generate multiple candidate tokens at once, then verify them, amortising the round-trip cost again. The engineering effort required to make distributed consumer inference feel acceptable is substantial, which is why most people who need speed still choose a single large card, and most people who choose multiple cards are doing batch work where the wait is tolerable.

Remember this

Splitting a model across consumer machines trades latency for capacity. It works when you can batch the work and wait for the answers.

Test yourself

You have three machines, each with one GPU and twenty-four gigabytes of memory. You want to run a fifty-billion-parameter model in sixteen-bit precision across them using pipeline parallelism. Explain one specific reason why you might not be able to use all twenty-four gigabytes on each machine, even though the weights alone would fit comfortably.

Go deeper

Image: Original diagram, The Daily Triptych. Licence: Original work. Source.

← Back to day 84