Disaggregated serving

Two phases that want different hardware

Prefill and decode established that these two phases of generating a response have almost opposite resource profiles: prefill is compute-bound, processing an entire prompt in one burst of parallel work; decode is memory-bandwidth-bound, running as a sustained, steady sequence of smaller steps for as long as generation continues. A single machine configured to serve both phases together is, by construction, a compromise — whatever balance of compute and memory bandwidth it strikes serves neither phase as well as hardware dedicated to just one of them would.

Disaggregation: serve each phase on hardware built for it

Disaggregated serving takes this observation to its logical conclusion: run prefill on one pool of GPUs, provisioned and scaled for prefill’s bursty, compute-heavy demand, and run decode on an entirely separate pool, provisioned and scaled for decode’s sustained, memory-bandwidth-heavy demand. Each pool can be sized, and even chosen on different hardware entirely, to match exactly what its one job actually needs, rather than forcing a single configuration to serve both reasonably rather than either one well.

What to watch for

The official NVIDIA explainer for Dynamo, the project that popularised disaggregated prefill/decode serving at scale, directly from the team that built it — covering the architecture and the reasoning behind it in genuine depth. It runs close to ninety minutes as a full technical walkthrough rather than a tight explainer; this is the only verified candidate found for this specific lesson after a dedicated search, so treat it as a reference to return to at a relevant chapter rather than something to watch in one sitting.

The cost disaggregation introduces

Splitting the two phases onto separate hardware is not free: the KV cache computed during prefill, per KV cache mechanics, has to actually move from the prefill machine to whichever decode machine will continue generating that specific request, over a network connection between them. This transfer takes real time, and it is a cost a single combined machine never has to pay at all, since the cache never has to leave the device it was computed on. Disaggregation only pays off when the efficiency gained from giving each phase hardware matched to its own needs exceeds this transfer cost — which is generally true at real production scale, where the efficiency gains are substantial, but is a genuine engineering tradeoff rather than a free win.

Where this fits in a full serving architecture

Disaggregated serving is usually combined with the model parallelism techniques from the previous lesson rather than replacing them — a large model might still need tensor parallelism within each individual prefill or decode machine, with disaggregation operating at the level of deciding which pool of machines handles which phase at all. This combination of techniques — splitting a model across GPUs where it does not fit on one, and splitting the two phases of serving it across differently-provisioned pools — is what large-scale production serving for the biggest current models actually looks like, well beyond what a single machine running a single copy of an inference engine, covered earlier in this roadmap, is built to handle.