FlashAttention and PagedAttention

Two different wastes, two different fixes

This lesson covers two of the most influential systems optimisations in modern LLM serving, each targeting a different kind of waste. PagedAttention addresses waste in how the KV cache’s memory is allocated. FlashAttention addresses waste in how the attention computation itself moves data through memory. Both are exact optimisations — neither changes the model’s output at all, only how efficiently that identical output is computed.

PagedAttention: memory management borrowed from operating systems

A naive KV cache implementation has to reserve memory for each request before knowing how long that request will actually run — since a request’s final length is not known in advance, the naive approach reserves enough contiguous memory for the worst case. This wastes most of that reservation for the many requests that turn out to be much shorter, and the fragmented leftover space often cannot be reused for anything else, per KV cache size. PagedAttention fixes this by borrowing a well-established idea from operating systems: allocate the cache in small, fixed-size blocks — pages — that do not need to sit next to each other in memory and can be assigned to a request incrementally, as it actually generates tokens, rather than reserved all at once upfront. This eliminates both the waste of over-reservation and the fragmentation that comes with it.

What to watch for

This explains PagedAttention exactly as this lesson frames it — the operating-system virtual-memory analogy, block tables mapping logical positions to physical memory pages, and what happens under memory pressure — in a tight, focused explainer. It covers PagedAttention specifically rather than both techniques; FlashAttention’s mechanism, covered below, is this lesson’s own written material, with the deeper link available for a fuller treatment.

FlashAttention: keeping the computation on-chip

Computing attention naively involves materialising a full matrix of attention scores — one entry per pair of positions in the sequence — in the GPU’s main memory, then reading that matrix back in for the next step of the computation. For anything beyond a short sequence this matrix is large, and writing and then re-reading it is exactly the kind of unnecessary memory traffic kernel fusion exists to eliminate. FlashAttention restructures the computation to avoid ever fully materialising this matrix in slow memory at all — processing the attention computation in smaller tiles that fit in fast, on-chip memory throughout, and only writing the final result back out. The output is identical to naive attention; what changes is that far less data makes the expensive round trip to and from main memory along the way.

Why both matter together

PagedAttention and FlashAttention target different parts of the same problem — memory management for the cache, and data movement within the attention computation itself — and a mature serving engine implements both rather than choosing between them. Together they are a large part of why a properly built inference engine, covered in inference engines, outperforms a naive implementation so substantially: neither technique changes what the model computes, only how much wasted memory traffic that computation requires to produce the identical result.