Kernel fusion

The hidden cost of doing things one step at a time

A transformer’s forward pass is built from many individual operations — matrix multiplications, activation functions, normalisation steps — chained one after another. Run naively, each of these operations reads its input from GPU memory, computes its result, and writes that result back out to memory before the next operation reads it back in again. For an intermediate result that is never needed anywhere except as input to the very next step, every one of those writes and reads is pure overhead: time spent moving data that nothing outside this one chain of operations will ever use.

What fusion actually does

Kernel fusion combines several of these operations into a single GPU kernel that performs all of them in one pass, keeping intermediate results in fast on-chip memory — close to the GPU’s compute units — instead of round- tripping them out to the GPU’s much slower main memory and back. The computation performed is identical to running the operations separately; what changes is eliminating the unnecessary memory traffic between steps that served no purpose except moving data from one operation to the next.

What to watch for

This is the canonical reference lecture the field itself points to for this exact topic — the GPU MODE community’s own dedicated session on fusing kernels, which is more authoritative here than a smaller individual tutorial, even a well-made one like the deeper link.

Why this is such high-leverage work

This connects directly to arithmetic intensity and the roofline model: fusion does not reduce the amount of useful computation an operation performs, but it does reduce the bytes of memory traffic required to perform it, which directly raises arithmetic intensity. For an operation that was memory-bound — sitting under the sloped memory roof, limited by data movement rather than compute — this is exactly the lever that helps. A compute-bound operation, by contrast, gains little from fusion, since memory movement was never its bottleneck. Because so much of a transformer’s per-token decode work is memory-bound, as covered in memory-bound vs compute-bound, fusion is one of the highest-leverage optimisations available to a serving engine, which is exactly why every major inference engine covered in the previous lesson implements extensive kernel fusion internally rather than leaving it to chance.

A concrete example of the chain being fused

A common target for fusion in a transformer is the sequence of operations right after an attention or feed-forward block: adding the block’s output back to its input (a residual connection), normalising the result, and applying an activation function, all one after another. Run unfused, that is three separate memory round trips for what is conceptually one small step. Fused into a single kernel, the GPU reads the input once, performs all three operations while the data sits in fast on-chip memory, and writes the final result out once — three memory round trips reduced to two, repeated for every single one of these junctions throughout every layer of the network, which is where the savings compound into something substantial across a full forward pass.

Not something you usually write yourself

In practice, most engineers using a mature inference engine never write a fused kernel by hand — the engine’s own optimized kernels already apply this technique throughout the model’s forward pass. Understanding what fusion is and why it matters is valuable regardless, because it explains a real chunk of why a mature engine outperforms a naive implementation so dramatically, and it is directly relevant for anyone writing a custom operation — a new activation function, a specialised attention variant — who wants that custom code to perform anywhere near as well as the engine’s own built-in kernels.