Arithmetic intensity and the roofline model

A ratio that predicts the bottleneck before you measure anything

Arithmetic intensity is the ratio of floating-point operations performed to bytes of data moved from memory for a given operation. A high arithmetic intensity means an operation does a lot of computation per byte it reads — it keeps the GPU’s compute units busy relative to how much it depends on memory movement. A low arithmetic intensity means the opposite: comparatively little computation happens per byte moved, and the operation spends more of its time waiting on memory than computing.

The roofline model, built from two lines

The roofline model plots this relationship for a specific piece of hardware. One line — the memory roof — rises with arithmetic intensity, representing the maximum performance achievable given the GPU’s memory bandwidth: at low arithmetic intensity, performance is capped by how fast data can be moved, regardless of how much raw compute the GPU has. A second, flat line — the compute roof — represents the GPU’s peak possible throughput, which no operation can exceed no matter how favourable its arithmetic intensity. These two lines meet at a single point for any given GPU, called the ridge point.

What to watch for

This is the only candidate found anywhere that teaches the roofline model by name, specifically for LLM inference, walking through arithmetic intensity and the compute/memory-bound distinction directly rather than gesturing at the underlying ratio without naming the framework it belongs to. The channel is an individual educator rather than an established organisation, which is worth a correctness check, but the content maps directly onto this lesson’s exact scope.

Reading an operation’s position on the chart

An operation’s arithmetic intensity determines where it sits on this chart, and that position tells you immediately which of the two roofs actually bounds it. An operation with arithmetic intensity below the ridge point sits under the sloped memory roof — it is memory-bound, and no amount of additional raw compute throughput would make it faster, only more memory bandwidth would. An operation above the ridge point sits under the flat compute roof — it is compute-bound, and more memory bandwidth would not help at all, only more raw compute throughput would.

Why this is the formal version of a lesson already covered

This is exactly the framework underneath memory-bound vs compute-bound, stated precisely rather than qualitatively: decode sits at low arithmetic intensity, since each step moves the entire set of weights through memory to perform comparatively little arithmetic on one new token, which places it firmly under the memory roof on essentially every current GPU. Prefill, processing many tokens of the prompt at once, achieves far higher arithmetic intensity and typically sits under the compute roof instead. Knowing an operation’s actual arithmetic intensity, rather than guessing which regime it falls into, is what lets an engineer predict in advance whether a given optimisation will help at all — a change that adds raw compute throughput does nothing for an operation that is already memory-bound, and vice versa.