KV cache mechanics

Why caching exists at all

As the previous lesson covered, each decode step needs attention computed between the new token and every token that came before it. Recomputing that attention from scratch at every single step — redoing the work for every earlier token, every time a new one is generated — would make decode dramatically more expensive than it needs to be, since the vast majority of that computation produces exactly the same result it produced the step before. The KV cache avoids this by storing the result once and reusing it.

What is actually stored

At every attention layer, for every attention head, a token’s contribution to attention is captured in two vectors: a key and a value. Once a token has been processed, its key and value vectors are fixed and will never change — they depend only on that token and the ones before it, not on anything generated afterward. The KV cache stores exactly these vectors, for every past token, at every layer and head, so that a new decode step only has to compute the key and value for the one new token and compare it against everything already cached, rather than reprocessing the whole sequence.

What to watch for

This continues the same serving-fundamentals series as the previous lesson, specifically covering what the cache stores and why it avoids recomputing attention — exactly this lesson’s scope, from the serving engine’s own memory-management perspective rather than a purely conceptual one.

Why the cache is often the biggest thing in memory

Cache size scales with three things at once: context length (more tokens means more cached key/value pairs per request), batch size (more concurrent requests means more sets of cached pairs held simultaneously), and model size (more layers and heads means more key/value vectors per token). See KV cache size for the exact arithmetic. Unlike the model’s own weights, which occupy a fixed amount of memory regardless of traffic, the cache’s footprint grows directly with how much context and how many concurrent users a server is actually handling — which is why, at long context lengths or high concurrency, the cache routinely becomes the largest single consumer of GPU memory, sometimes dwarfing the weights it is serving alongside.

A concrete sense of the memory involved

A model with several dozen layers and tens of attention heads per layer, each storing a key and value vector of a few thousand numbers for every single token in context, adds up quickly. A single request holding a context of a few thousand tokens can occupy several gigabytes of cache on its own, and that figure multiplies directly with however many such requests a server is holding concurrently. This is why context length and concurrency are not independent knobs a serving system can tune freely — doubling either one roughly doubles the cache memory required, and a server that comfortably handles a hundred short conversations at once may struggle badly the moment even a handful of those conversations grow into long ones.

Why this matters for everything downstream

Managing this memory efficiently, rather than wasting it, is the direct motivation behind PagedAttention, which manages the cache the way an operating system manages virtual memory to avoid fragmentation and waste. Sharing a cached prefix across multiple requests that share a common prompt start is the whole idea behind prefix caching. And the cache’s sheer size at scale is a direct input into choosing hardware — sizing a GPU for a workload means budgeting for the cache’s likely footprint, not just the model’s weights.