II · THE IDEA · ARTIFICIAL INTELLIGENCE
TensorRT and Kernel Fusion
▶ Listen · narrated
Inference cost is not only arithmetic. Much of the time a GPU spends on a neural network is moving activations in and out of memory between tiny kernels that could have shared a single pass.
At a glance
- What it is
- Merging several operations into one GPU kernel at compile time
- Main aim
- Fewer round trips between on-chip compute and device memory
- What shrinks
- Stores and loads of intermediate activations, not the maths itself
- Where it sits
- In the compiler and runtime path used for inference
Think of a kitchen line where every cook must walk to the cold store after each tiny step: chop, walk, store; mix, walk, store; plate, walk, store. The recipes are correct, but the walking dominates. Kernel fusion is like letting one cook finish a short sequence at the board and only then walk back with the finished pan. The recipe did not change; the trips did.
On a GPU, each small operation is often a separate kernel. Between kernels, results are written to device memory and read again. That traffic costs time. A compiler that fuses operations builds one kernel that does several steps in order and keeps the in-between values nearby, in fast on-chip storage. You still get the same numerical job done, but with fewer large movements of intermediate activations. That is why fusion can speed inference without retraining the model.
Neural nets lower to a graph of ops; GPU backends schedule those ops as kernels. An unfused edge typically means a global-memory write of an intermediate tensor by producer and a subsequent read by consumer, plus launch overhead. Fusion is a graph rewrite and code-generation strategy that replaces a matched subgraph with a single kernel in which producers’ results remain in registers or shared memory for consumers, subject to dependence, footprint, and correctness constraints.
The optimisation targets bytes moved and launches issued, not a reduction in FLOPs required by the abstract network. It helps most when the unfused sequence is memory-bound or launch-bound. Element-wise chains and epilogues attached to larger stages are common successful patterns; mismatched tilings, reductions, or working sets that exceed on-chip capacity limit what may merge.
Inference compilers (TensorRT among them) apply such rewrites when building an engine for a target device. Two engines can implement the same function with different fusion degrees and therefore different bandwidth curves. Limitations remain: fusion is pattern-dependent, can interact with layout and precision choices, and must not change the defined numerics beyond the stack’s stated tolerance. Profiling before and after an engine build is the practical check—look for fewer kernel launches and less intermediate traffic, not for a different model.
Look closer
The journey costs more than the sum
A kernel that runs alone must usually read its inputs from device memory and write its outputs back before the next kernel can start. When a network is expressed as a long chain of small operations, those repeated store–load pairs become a large share of the step time, even when each individual arithmetic stage is modest. Fusion is interesting precisely because it targets that traffic, not because it invents new mathematics.
What is allowed to merge
Adjacent operations fuse only when the compiler can keep the network’s defined result, fit working data in on-chip resources, and still emit one valid launch. Short element-wise chains are comparatively cooperative; stages with clashing access patterns often are not. The useful observation is conditional: when fusion succeeds, intermediate values need not become full memory residents between stages.
The omitted store–load pair
In an unfused sequence the first operation writes an intermediate tensor and the second reads it again. Inside a fused kernel that hand-off can stay in registers or other on-chip memory. The bandwidth that would have moved the whole intermediate is simply not spent. That is the concrete saving the optimisation is after.
The story
A deployed neural network is not executed as one indivisible program. It is lowered into a graph of operations, and on a GPU those operations become kernels—short programs launched across many parallel threads. Between launches, results ordinarily live in device memory. The next kernel reads them back, does its arithmetic, and writes again. The arithmetic is what the model description names; the repeated traffic is what the machine often waits on.
Kernel fusion is a compiler response to that pattern. Instead of emitting one kernel per operation, the stack looks for short sequences it can legally combine. The fused kernel performs the same mathematical steps in order, but keeps intermediate values where the execution units can still see them. The large intermediate tensor that would have been stored and reloaded may never become a full trip across the memory bus.
The gain is therefore easy to misread. Fusion does not magically reduce the amount of useful arithmetic the network requires. It reduces how often the machine must pay bandwidth and launch overhead to feed that arithmetic. On workloads where layers are memory-bound—where the GPU finishes the sums and products quickly and then waits on data—removing those trips shortens the step. On workloads already limited by compute, the same rewrite may matter less.
What the compiler may merge is constrained. It has to preserve the result the graph specified, respect on-chip capacity, and produce a single kernel the hardware can run. Some patterns cooperate: chains of element-wise work, or a stage followed by a bias and an activation when the schedule allows partial results to stay local. Other patterns resist fusion because the data access shapes do not line up, or because the working set would no longer fit. The optimisation is therefore opportunistic rather than universal.
Inference stacks such as TensorRT sit in this space: they take a trained network, apply graph-level rewrites including fusion, and emit an engine tuned for a chosen target. The editorial point is not a vendor checklist but a systems fact. Once you see the GPU as a machine that charges for traffic between kernels, fusion stops looking like a mysterious speed trick and becomes a straightforward attempt to stop paying for memory movements the model never semantically required.
That framing also explains why small graph choices matter. An extra operation inserted for convenience during export can break a fusible chain and reintroduce a round trip. A layout or precision decision can change whether intermediates stay on-chip. The visible speed difference after an engine build is often less about a new algorithm and more about how many of those invisible trips the compiler managed to delete.
Why it mattered then
As networks were deployed for interactive inference, the gap between peak arithmetic throughput and what real graphs achieved became hard to ignore. Many production models spent a striking share of each step on launching small kernels and shipping activations through device memory. Compiler-side fusion addressed that mismatch directly: it kept the trained weights and the defined computation, yet changed the mapping onto the GPU so that intermediate traffic shrank. In its own moment, that mattered because latency and cost were already set by the memory path as much as by the maths.
Why it matters now
The same pressure has only sharpened. Larger models, tighter latency budgets, and the habit of running open-weight networks on finite hardware all make needless memory traffic expensive. Fusion remains one of the main reasons two engines built from the same abstract graph can differ in speed without differing in the answers they are meant to compute. Anyone who exports a model, builds an inference engine, or compares backends is still negotiating this trade: how much of the step is arithmetic, and how much is movement the compiler might still remove.
The surprising detail
The surprising part is how un-glamorous the win is. No new layer is invented; no training run is repeated. A fused kernel can perform essentially the same multiplies and adds as the unfused sequence, yet finish sooner because a bulky intermediate never travels to device memory and back. Performance moves because of absence—the missing store, the missing load—rather than because the model learned something new.
What is disputed
Public discussion of fusion often leans on vendor documentation and engineering lore rather than a single canonical measurement. How much any one network gains depends on shape, precision, hardware generation, and which patterns the compiler actually matches; treat the mechanism as solid and the size of the win as workload-specific. The sources supplied for this lesson do not themselves document TensorRT internals, so no product-specific claim beyond the general compiler pattern is asserted here.
Remember this
Fusion keeps intermediate values on-chip so the GPU does less memory traffic between operations that could have shared one kernel.
Test yourself
Two inference builds run the same trained network and produce matching outputs, yet one is faster on the same GPU. In the light of kernel fusion, what should you suspect first about the slower build’s execution, and what would not need to differ?
Suspect that the slower build still materialises more intermediate tensors in device memory—more separate kernels, more store–load round trips—while the faster build fused some of those stages so intermediates stayed on-chip. The trained weights and the mathematical definition of the network need not differ; the mapping onto kernels and memory traffic can account for the gap.
Go deeper
- [2005.14176] Marcinkiewicz-Zygmund Inequalities for Polynomials in Bergmann and Hardy Spaces · arxiv.org
- [2104.05754] The role of relatedness and strategic linkages between domestic and MNE sectors in regional branching and resilience · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.