II · THE IDEA · ARTIFICIAL INTELLIGENCE
Bandwidth, Not FLOPs
▶ Listen · narrated
If you have ever watched a large model run locally and wondered why it generates so slowly despite the advertised teraflops, the answer is simpler than you might expect: the weights are heavy.
At a glance
- Operation per token
- Read every active weight once, then multiply and add
- Dominant cost
- Moving billions of parameters from VRAM to compute units
- Prediction method
- Divide memory bandwidth by model size in bytes
- Batch size effect
- Larger batches amortise the weight-reading cost across more tokens
Imagine a library where you must fetch every book, one at a time, to write a single sentence. Even if you can read extraordinarily fast, your sentence-writing speed is limited by how quickly you can carry books from the shelves to your desk. A GPU generating tokens faces the same constraint. The model's weights are the books, stored in video memory. To produce one token, the chip must fetch all of them — billions of numbers — across a memory bus with a fixed width. That fetching takes time, and it takes the same amount of time regardless of how fast the chip can multiply numbers once they arrive. A GPU advertised as capable of trillions of calculations per second might generate only thirty tokens per second, because it spends most of its time waiting for the weights to arrive. Halving the size of each weight — by storing them as smaller numbers — doubles the speed, because twice as many weights fit through the bus in the same time.
Autoregressive token generation performs one forward pass per token. For a model with N parameters stored in 16-bit precision, that requires reading 2N bytes from VRAM. On a GPU with B bytes per second of memory bandwidth, the theoretical maximum token rate is B divided by 2N, before accounting for activations, attention or kernel overhead. A 7-billion-parameter model in FP16 is 14 GB; on a card with 500 GB/s bandwidth, the ceiling is roughly 35 tokens per second. Actual performance is lower — typically 60 to 80 percent of the theoretical maximum — because intermediate activations must also be read and written, and because attention requires additional memory traffic proportional to sequence length. Quantising to INT8 halves the bytes per parameter, effectively doubling bandwidth-limited throughput. Quantising to INT4 doubles it again, though with greater accuracy loss. Batching amortises the weight-reading cost: generating B tokens in parallel reads the weights once but produces B outputs, so the effective cost per token is divided by B. At sufficiently high batch sizes, the workload becomes compute-bound rather than bandwidth-bound, and FLOPs begin to matter. For single-user inference, however, bandwidth remains the dominant constraint, which is why memory bus width and clock speed predict performance better than shader count or tensor core throughput.
Look closer
The weights are read, not computed
Generating one token from a 7-billion-parameter model stored in 16-bit precision requires reading roughly 14 gigabytes from memory — the chip must fetch every weight, once, to perform the forward pass. A consumer GPU with 500 GB/s bandwidth can therefore read those weights about 35 times per second, which sets an upper bound near 35 tokens per second for a single user. The actual figure is lower once you account for activations, attention and overhead, but the ceiling is visible from the bandwidth spec alone.
FLOPs are abundant, bandwidth is not
A modern accelerator may advertise hundreds of teraflops — trillions of multiply-accumulate operations per second. But each weight must arrive at the compute unit before any arithmetic happens. If the memory system cannot supply data fast enough, the arithmetic units idle. This is why a high-end datacenter GPU and a consumer card with the same memory bandwidth often produce similar token rates for single-user inference, despite a tenfold difference in peak FLOPs. The compute is waiting for the weights.
Batching changes the arithmetic
Serving multiple users at once, or generating multiple candidate completions in parallel, means the same weights are reused across many tokens in one pass. A batch of 32 prompts reads the 14 gigabytes once but produces 32 tokens, so the effective cost per token drops by a factor of 32. Large-scale serving infrastructure is built around this principle: keep batch sizes high and memory bandwidth becomes less dominant, allowing the abundant FLOPs to matter again. Single-user inference has no such luxury.
The story
When you ask a model running on your own hardware to generate text, the process feels smooth but slow. A capable GPU might produce twenty or thirty tokens per second — fast enough to read comfortably, but nowhere near the trillions of operations per second printed on the specification sheet. The mismatch is not a software bug or a driver inefficiency. It is a fundamental consequence of what the chip must do.
Every token generated requires a full forward pass through the model. That means reading every active parameter, multiplying it by the corresponding activation, and accumulating the results. For a model with seven billion parameters stored in 16-bit floating point, that is 14 gigabytes of data. The GPU must fetch all of it from its own video memory, move it across the memory bus to the compute cores, and only then perform the arithmetic. The time spent reading dwarfs the time spent multiplying.
Consider a consumer card with 500 gigabytes per second of memory bandwidth. Dividing 500 by 14 gives roughly 35 — meaning the card can, in principle, read the entire model 35 times per second. That is the ceiling for token generation with a single prompt. Real performance falls somewhat short, because the model also produces intermediate activations that must be written back to memory, and because attention mechanisms require their own reads and writes. But the rough prediction holds: bandwidth divided by model size gives you tokens per second.
This is why quantisation — storing weights in 8-bit or even 4-bit integers instead of 16-bit floats — produces such dramatic speedups on consumer hardware. Halving the precision halves the bytes, which doubles the number of times per second you can read the model. The arithmetic becomes less accurate, but the arithmetic was never the bottleneck. You were waiting for the data to arrive.
Batching changes the equation. If you generate text for thirty-two users at once, you read the 14 gigabytes once but produce thirty-two tokens. The cost per token drops by a factor of thirty-two, and suddenly the FLOPs start to matter again, because the compute units have enough work to do while the next batch of weights arrives. This is why large-scale API providers care intensely about batching, request routing and queue depth: they are trying to keep the memory bus busy and the arithmetic units fed. A lone user running a model locally has no such option. The weights are read, one token emerges, and the process begins again.
Why it mattered then
The recognition that memory bandwidth, not floating-point throughput, limits transformer inference became explicit in research around 2022, though practitioners had observed the pattern earlier. Tri Dao and collaborators published FlashAttention in 2022, demonstrating that attention mechanisms could be rewritten to minimise memory movement rather than minimise FLOPs, producing speedups of several times on identical hardware. Reiner Pope and colleagues at Google published work on efficiently scaling transformer inference in 2023, making the bandwidth-bound nature of autoregressive generation explicit and deriving the same rule of thumb: divide bandwidth by model size. The insight mattered because it clarified where optimisation effort should go. Buying a faster GPU helped less than expected; reducing precision, rewriting kernels to respect cache hierarchies, and batching requests helped more. It also explained why purpose-built inference accelerators, which often had lower peak FLOPs than training GPUs but much higher memory bandwidth per parameter, could outperform them in production.
Why it matters now
The constraint has not gone away. Consumer GPUs in 2024 still have memory bandwidth in the hundreds of gigabytes per second, which sets a ceiling of tens of tokens per second for models in the 7-to-13-billion-parameter range. Quantisation remains the most effective local speedup, because it directly reduces the bytes that must move. The rise of mixture-of-experts architectures, which activate only a subset of parameters per token, is partly a response to the same pressure: if you cannot read all the weights fast enough, read fewer weights. Meanwhile, batching remains the dominant strategy in serving infrastructure, and the gap between single-user and multi-user performance continues to widen. Understanding the bandwidth ceiling also clarifies why certain optimisations matter and others do not: a faster matrix multiplication library helps only if memory is already feeding the compute units fast enough, which in autoregressive generation it usually is not.
The surprising detail
The bandwidth-bound nature of inference means that a high-end datacenter GPU and a much cheaper consumer card can produce nearly identical token rates for a single user, provided they have similar memory bandwidth. A card with twice the FLOPs but the same bandwidth will not generate tokens twice as fast. This has made memory bandwidth per dollar a more useful metric than FLOPs per dollar for anyone running models locally, and it has created a market for older or less powerful cards that happen to have wide memory buses. It also means that the teraflops figure prominently displayed in GPU marketing materials is, for this workload, largely irrelevant.
Remember this
Token generation reads every weight once per token. Bandwidth divided by model size gives you the ceiling; FLOPs do not.
Test yourself
You have a 13-billion-parameter model in 16-bit precision and two GPUs: one with 1000 GB/s bandwidth and 500 teraflops, another with 600 GB/s and 100 teraflops. Which will generate tokens faster for a single user, and roughly how much faster?
The first GPU will be faster, by about 67 percent. The model is 26 gigabytes, so the first card can read it roughly 38 times per second and the second roughly 23 times per second. The difference in teraflops is irrelevant here: the bottleneck is moving the weights from memory, not multiplying them. The 400 teraflop gap between the cards contributes nothing to single-token generation, because the arithmetic units are idling while they wait for data. If you were batching requests heavily — say, 64 or 128 at once — the teraflops would start to matter, because the same weights would be reused across many tokens and the compute would no longer be starved. But for one user, bandwidth is the only number that predicts performance.
Go deeper
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness · arXiv · Tri Dao et al. · 2022-05-27
- Efficiently Scaling Transformer Inference · arXiv · Reiner Pope et al. · 2022-11-09
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.