Prompt caching

Two phases, one of them repeated

Serving a model call splits into two mechanically different phases. Prefill processes the entire input in one parallel pass across all positions at once, building the key and value tensors for every input token at every attention layer — the KV cache described in the context window lesson. Decode then produces output tokens one at a time, each step reading the accumulated cache. Prefill is compute-heavy and scales with input length; decode is comparatively cheap per token but has to happen sequentially. If your next request starts with the same two thousand tokens as the previous one — the same system prompt, the same tool definitions, the same long reference document — every bit of the prefill work for that shared prefix is being redone from scratch, even though the result would be identical.

What caching actually stores and reuses

Caching keeps the computed key and value tensors for a prefix across calls, keyed by the exact token sequence. When a new request’s prefix matches a cached one, the provider skips recomputing that portion of prefill entirely and reuses the stored tensors, continuing only from the first token that differs. The cached portion typically costs a small fraction of its normal per-token price and adds negligible latency, since the expensive parallel computation for it was already done. This is purely a serving-side optimisation — it changes nothing about what the model can attend to or how it reasons, only how much redundant computation the provider has to perform to get there.

Why the match has to be exact, from the first token

The cache is keyed on an identical prefix starting from position zero — not on semantic similarity, not on “close enough.” The moment a single token differs, from that position onward nothing is served from cache, because the key and value tensors for every subsequent token depend on everything before it through the attention mechanism. This is why prompt layout becomes a genuine performance decision: order a prompt from most stable to least — system instructions, tool definitions and long reference material first, since these rarely change between calls; retrieved documents next; the user’s specific message and anything request-specific, such as a timestamp or a session identifier, last. Placing a timestamp near the top of a prompt, purely out of habit, invalidates the cache on every single call, because no two requests ever share that value.

A worked numeric illustration

Consider a system prompt and tool definitions totalling three thousand tokens, reused across every call in an agent loop that makes twenty iterations per task. Without caching, that three thousand tokens of prefill work is repeated in full twenty times — sixty thousand tokens’ worth of prefill computation for content that never actually changed. With a cache hit on each of the nineteen repeat calls, only the first call pays full price for that portion, and every subsequent call pays the steeply discounted cached rate for it, with full price applying only to the new, request-specific content appended at the end. The saving scales directly with how much of the prompt is shared and how many times it is reused, which is exactly why agent loops — where the bulk of the prompt is identical across many iterations — benefit more than single-turn chat.

Practical details that vary by provider

Providers differ on the minimum prefix length eligible for caching, on how long a cached entry survives before eviction, and on whether caching is fully automatic or requires an explicit marker in the request indicating where the cacheable prefix ends. These specifics move with provider updates and are worth checking directly rather than assumed, but the underlying shape of the optimisation — stable content first, variable content last — is universal across every provider that offers this at all, and it remains one of the cheapest, highest-leverage cost reductions available to most LLM applications, tying directly into the broader economics covered in cost and latency.