II · THE IDEA · ARTIFICIAL INTELLIGENCE
Continuous Batching in LLM Serving
▶ Listen · narrated
A single slow decode can strand an entire batch. Serving systems that keep the batch membership fluid waste far less of each GPU cycle on padding and waiting.
At a glance
- What it is
- Admitting and retiring requests inside a running decode batch
- Core problem
- Requests differ in prompt length and generation length
- Idle source
- Static batches pad and wait for the longest member
- SARATHI idea
- Piggyback decodes with chunked prefills
Think of a café espresso machine that can pull several shots in one go. In static batching the barista waits until every cup in the group is done before starting any new order—so one slow cappuccino leaves the machine half-idle while finished cups sit there. Continuous batching is the same machine, but as soon as one cup is handed over, the next ticket is pulled into that free slot. The group at the machine keeps changing.
For language models the “cups” are requests generating tokens. Prefill is grinding the beans for a new order (reading the whole prompt); decode is pouring the next small shot (one new token). If someone arrives with a huge bag of beans, grinding it all at once stops the pouring for everyone else. Chunked prefills grind that bag a handful at a time, interleaved with pouring other people’s shots—the idea SARATHI emphasises as piggybacking decodes with chunked prefills.
Continuous batching (iteration-level scheduling) maintains a dynamic set of active sequences and runs a forward step over whatever is currently admitted. After each step, sequences that emitted EOS are removed; waiting requests may be admitted, subject to memory for KV cache. Static batching instead fixes membership and pads to max length until the slowest sequence completes, wasting compute on padding and on finished rows.
Prefill computes keys and values for all prompt positions and is typically compute-bound; decode attends with the cached KV and is typically memory-bandwidth bound. Under continuous batching the two phases share the device. A full prefill inserted into a decode-heavy batch creates a latency bubble for interactive decodes. SARATHI-style chunked prefills split the prompt into token chunks and schedule those chunks in the same steps as ongoing decodes—piggybacking—so each iteration does a bounded amount of prefill work plus decode for live sequences. Limits remain: KV memory caps admission, chunk size trades stall reduction against prefill efficiency, and scheduling policy must still respect fairness and max batch constraints.
Look closer
Batch membership is no longer fixed
In a static batch the set of sequences is chosen once, padded to a common length, and held until every member finishes. Continuous batching treats the batch as a sliding roster: when one sequence emits an end-of-sequence token, its slot can be given to a waiting request on the next step instead of sitting empty or filled with pad tokens for the rest of the run.
Prefill and decode compete for the same device
Prefill is compute-heavy and processes many prompt tokens at once; decode is memory-bandwidth heavy and emits one token per sequence per step. Under continuous batching these two phases interleave on the GPU. A large prefill dropped in whole can stall ongoing decodes. Chunking the prefill, as SARATHI does, breaks that stall into smaller pieces that can share steps with decode work already in flight.
Utilisation shows up in the gaps
What you would observe on a busy server is not a single long kernel per request but a steady cadence of mixed steps: some sequences advancing by one generated token, others chewing through the next chunk of a prompt. The batch size in flight stays closer to the hardware sweet spot because new work arrives as old work retires, rather than in stop-start waves.
The story
Large language model serving is awkward for classical batching. Each request arrives with its own prompt length and then generates an unpredictable number of tokens. If a server gathers a fixed group of requests, pads them to the longest prompt, and runs until the slowest generation finishes, much of the GPU’s capacity is spent on pad positions or on sequences that have already completed.
Continuous batching changes the unit of scheduling. Instead of committing to a closed group for the lifetime of the longest member, the server maintains an open batch of active sequences and, at each decoding step, decides which sequences still need a next token. When a sequence finishes, it leaves. When a slot frees and a queued request is waiting, that request can enter—often after a prefill phase that builds its key-value cache—without waiting for every other sequence to drain.
The difficulty is that prefills and decodes have different shapes. Prefill reads the full prompt and is typically compute-bound; decode reads the growing cache and is typically bound by memory bandwidth. Naively inserting a full prefill into a batch of decodes can create a latency spike for everyone already generating tokens. The SARATHI line of work addresses this by chunking prefills: the prompt is split into smaller segments, and those segments are scheduled alongside ongoing decode steps. Decodes are piggybacked with chunked prefills so that the GPU stays busy on useful work without stalling interactive generations for the duration of a large prompt.
The practical picture is a pipeline that never quite empties. Tokens from many users advance in the same step; padding shrinks because sequences are not held to a common finish line; and the server can trade a little scheduling complexity for higher device occupancy. Continuous batching does not invent new model mathematics. It rearranges when each token’s work is allowed onto the GPU so that variance in request length stops dominating utilisation.
Why it mattered then
As soon as LLM serving moved from single-request demos to multi-tenant endpoints, fixed batches became a visible tax. Interactive traffic mixes short chat turns with long documents and uneven generation lengths. Under static batching that mix translated directly into idle cycles and inflated tail latency: one long decode held the batch open while finished slots produced nothing useful. Continuous batching, and refinements such as SARATHI’s chunked prefills, mattered because they attacked that structural waste without requiring a different model architecture. They made it feasible to keep GPUs nearer saturation on the traffic patterns operators actually saw.
Why it matters now
Production traffic is still dominated by mixed lengths, and cost is still dominated by GPU time. Every serving stack that aims at high tokens-per-second under concurrent load—whether open-source engines or managed APIs—depends on some form of continuous or iteration-level batching, often with prefill chunking or related scheduling so that heavy prompts do not freeze live decodes. Understanding the mechanism clarifies why adding hardware alone does not fix latency, why prompt length affects neighbours in the batch, and why scheduling policy remains a first-order lever beside model size and quantisation.
The surprising detail
The expensive mistake is often not decode at all but a monolithic prefill. A single large prompt, admitted whole into a batch of ongoing generations, can stall every interactive user sharing the device for the duration of that prefill. SARATHI’s counter-intuitive move is to break the prompt into chunks and deliberately interleave those chunks with other people’s decode steps—piggybacking—so throughput stays high without turning one user’s pasting of a long document into everyone else’s latency spike.
What is disputed
Public descriptions of continuous batching and of SARATHI-style chunked prefills agree on the stall mechanism and the piggybacking remedy, but exact throughput gains depend on hardware, model size, and traffic mix. Treat the technique as a scheduling pattern, not a fixed speed-up factor.
Remember this
Continuous batching keeps the GPU busy by letting requests join and leave each step; chunked prefills stop one long prompt from stalling everyone else’s decode.
Test yourself
A server runs continuous batching. A long prompt arrives while several short chats are decoding. Why might admitting that prompt as one unbroken prefill hurt the chat users, and what does chunking the prefill change about each decode step?
An unbroken prefill is a large, compute-heavy block that can monopolise the GPU for many milliseconds, delaying the next decode token for every chat already in the batch. Chunking splits the prompt into smaller pieces scheduled beside ongoing decodes, so each step still advances live generations while only a fraction of the new prompt is processed—piggybacking decodes with chunked prefills instead of pausing them.
Go deeper
- [2308.16369] SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills · arxiv.org
- [2311.03384] Serious Games in Digital Gaming: A Comprehensive Review of Applications, Game Engines and Advancements · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.