II · THE IDEA · ARTIFICIAL INTELLIGENCE
Batching, Throughput and Latency
▶ Listen · narrated
The model on your machine and the one behind an API endpoint may be identical, yet one answers in two seconds and the other in twenty. Batching explains most of the gap.
At a glance
- Throughput
- Total tokens generated per second across all requests
- Latency
- Time from submitting one request to receiving its complete answer
- Batching gain
- Processing ten prompts together can be nearly as fast as processing one
- Single-user cost
- Hardware sits idle between tokens; no other request can share the wait
Think of a bakery with one oven. If the baker makes one loaf at a time, the oven spends most of its energy heating empty space. If the baker waits until there are twelve orders and bakes them all together, the oven is full and the energy cost per loaf drops dramatically. The twelve customers wait the same amount of time for their own loaf to bake — the oven does not get faster — but the bakery produces twelve loaves in barely more time than it took to produce one. That is batching. A cloud API is the busy bakery: many customers, one oven, high efficiency. Your laptop is the same oven baking one loaf at a time. The hardware is identical, but the economics are completely different because there is no one else to share the fixed costs with.
Batching in transformer inference works because the forward pass is dominated by matrix multiplications, and matrix multiplication on a GPU scales sub-linearly with batch size up to the hardware's memory and parallelism limits. Processing a batch of 32 prompts takes perhaps twice as long as processing one, not thirty-two times as long, because the same weights are reused and the operations are parallelised across the batch dimension. Throughput, measured in tokens per second summed across all requests, scales nearly linearly with batch size until memory becomes the constraint. Latency for any individual request is unaffected by batching during the prefill phase — processing the prompt — but during autoregressive decoding, each step must wait for the entire batch to complete, so latency per token can increase slightly as batch size grows. Continuous batching, described by Pope et al., decouples request lifetime from batch lifetime: requests join and leave the batch dynamically between decode steps, so the batch size stays near its maximum without delaying short requests behind long ones. PagedAttention, from Kwon et al., eliminates memory fragmentation by splitting the KV cache into fixed-size blocks allocated non-contiguously, which allows much larger batch sizes before memory is exhausted. The combination of the two techniques is why modern serving systems can sustain batch sizes of 100 or more on consumer GPUs, whereas naive implementations struggle beyond 8. For single-user local inference, none of this applies: there is no batch to fill, so the GPU is underutilised and the effective cost per token is much higher.
Look closer
The GPU is mostly idle during generation
Generating a token requires a forward pass through the entire model, but then the system must wait while that token is appended, the new prompt assembled, and the next pass prepared. During that wait — often measured in milliseconds — the accelerator does nothing. If ten requests are batched, the same idle time is shared across all of them, and the hardware runs closer to full utilisation. A single-user setup pays the full wait cost for every token with no one else to amortise it over.
Continuous batching fills gaps as requests finish
Older serving systems waited until every request in a batch completed before starting a new batch, which meant short answers were delayed by long ones. Continuous batching, described by Pope et al. and implemented in systems like vLLM using PagedAttention, removes finished requests and adds new ones between each decode step. The batch size stays high, throughput stays high, and no request waits for an unrelated one to finish. It is the difference between a bus that departs on a schedule and one that leaves when full but lets passengers board and alight at every stop.
Memory fragmentation was the earlier bottleneck
Each request carries a cache of key and value vectors for every token generated so far — the KV cache — and its size grows with the conversation. Naive allocators reserved the maximum possible space up front, wasting gigabytes. PagedAttention, introduced by Kwon et al., borrows the paging idea from operating systems: the cache is split into small blocks that can be allocated non-contiguously and shared between requests when prompts overlap. The result is that a server can fit several times more requests in the same memory, which directly increases the throughput.
The story
Imagine a commercial API serving a hundred users at once. Each user submits a prompt, and each prompt requires the model to generate a response token by token. The naive approach would be to handle them one at a time: process the first user's prompt, generate all their tokens, then move to the second user. The GPU would be busy, but the total time to serve everyone would be the sum of all individual response times. If each response takes ten seconds, the hundredth user waits sixteen minutes.
Batching changes the arithmetic. Instead of processing one request to completion, the server processes one decode step for every request simultaneously. The model sees a batch of a hundred prompts, generates a hundred tokens in a single forward pass — only slightly slower than generating one — then moves to the next step. The GPU does more work per unit of time because it is never waiting for a single request to finish. Throughput, measured in tokens per second across all users, climbs steeply. The cost per token drops because the fixed overhead of loading weights and moving data is shared.
But latency, the time any individual user waits, does not improve and may worsen slightly. Each decode step now processes a hundred requests instead of one, so it takes a little longer. The user still waits for every token in their response to be generated sequentially. Batching is a throughput optimisation, not a latency one.
Now consider a laptop running the same model for a single user. There is no second request to batch with. The GPU performs one forward pass, generates one token, then idles while the software prepares the next step. The hardware is capable of far more work, but there is no work to give it. Throughput per token is identical to the cloud case, but there is only one token being produced at a time, so the absolute throughput — tokens per second summed across all activity — is a fraction of what the hardware could sustain. The user pays the full cost in time and energy for each token, with no economy of scale.
Continuous batching, the technique described by Pope et al. and implemented in serving frameworks like vLLM, improves on static batching by allowing requests to enter and leave the batch between decode steps. A request that finishes is immediately replaced by a new one from the queue. The batch size stays near its maximum, the GPU stays busy, and short requests do not wait for long ones to complete. The system behaves like a pipeline that never drains.
PagedAttention, introduced by Kwon et al., solved a memory problem that had been limiting batch sizes. The KV cache for a request can grow to gigabytes, and older systems allocated the maximum possible space for each request up front. Most of that space went unused, and memory filled up long before the GPU was fully utilised. PagedAttention splits the cache into small fixed-size blocks and allocates them on demand, non-contiguously, like an operating system managing virtual memory. When two requests share a prompt prefix, they can share the same cache blocks for that prefix. The memory savings are large enough that a server can often double or triple its batch size, which directly translates to higher throughput and lower cost per token.
The local single-user case cannot exploit any of this. There is no batch to keep full, no queue of requests to draw from, and no memory pressure because only one conversation is active. The model runs correctly, but the hardware runs inefficiently. This is why the same model on the same hardware can feel fast in a commercial setting and slow on a desktop: the cloud provider is serving fifty requests in the time it takes you to generate fifty tokens, and their cost per token is a fiftieth of yours.
Why it mattered then
The papers by Kwon et al. and Pope et al. were published in 2023, at a moment when the cost of serving large language models had become a bottleneck for anyone trying to offer them commercially. Inference at scale was expensive not because the hardware was slow, but because it was being used inefficiently. Memory limits forced batch sizes down, and static batching meant that hardware sat idle whenever batch sizes fluctuated. The techniques described in these papers — paged memory management and continuous batching — were direct responses to those inefficiencies. They allowed providers to serve more users on the same hardware, which lowered the cost per token and made certain business models viable that had not been before. The improvements were large enough that they changed the economics of API access within months of deployment.
Why it matters now
Continuous batching and PagedAttention are now standard in production serving systems. If you use a commercial API, you are almost certainly benefiting from both. They are part of the reason that API pricing has fallen while model size has grown: the cost per token has dropped faster than the models have become more expensive to run. For anyone running models locally, understanding these techniques clarifies why the experience is different. A single-user setup cannot amortise costs across many requests, so it will always feel slower and less efficient than a shared service, even when the underlying model is identical. The techniques also matter for anyone building infrastructure: if you are serving more than one user, implementing continuous batching is one of the highest-leverage optimisations available, often doubling throughput without changing the model or the hardware.
The surprising detail
PagedAttention borrows its core idea — paging — from operating systems designed in the 1960s. The problem it solves is structurally identical: how to manage a large address space when physical memory is limited, and how to avoid fragmentation when allocations are unpredictable. The KV cache behaves like virtual memory, and the blocks behave like pages. The analogy is close enough that the implementation can reuse many of the same algorithms, including copy-on-write sharing when multiple requests reference the same data. It is a reminder that infrastructure problems often have older solutions waiting to be recognised, and that a technique from one domain can sometimes transfer wholesale to another that looks entirely different on the surface.
Remember this
Batching makes serving many users cheap per token. Serving one user fast requires different optimisations entirely, and batching does not help.
Test yourself
A server using continuous batching is processing requests with a maximum batch size of 64. At a given moment, 40 requests are active. A new request arrives that will take 200 tokens to complete, and the average request in the current batch will finish in 30 tokens. Should the server add the new request immediately, or wait until the batch is fuller? What is the trade-off?
The server should add it immediately. Continuous batching is designed to keep the batch size high by filling gaps as soon as they appear, and waiting wastes GPU cycles. The trade-off is subtle: adding a long request now means it will occupy a batch slot for 200 steps, during which shorter requests will come and go around it. If the server waited, it might briefly process a smaller batch, but the long request would start later and finish later, increasing its latency. The system is optimised for throughput, and throughput is maximised by keeping the batch full at every step. Delaying requests to balance batch composition is a more complex strategy that some systems explore, but the default behaviour in continuous batching is to admit requests as soon as there is space.
Go deeper
- Efficient Memory Management for Large Language Model Serving with PagedAttention · arXiv · Woosuk Kwon et al. · 2023-09-12
- Efficiently Scaling Transformer Inference · arXiv · Reiner Pope et al. · 2022-11-09
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.