Testing and safe deployment

Changing a running system without breaking it

Every other lesson in this roadmap covers getting an inference system running well. This one covers changing it safely once it is already live and serving real traffic — a new model version, a different quantization scheme, an updated serving configuration — without a bad change reaching every user at once, or the system going down while the change rolls out.

Shadow testing: watch without affecting anyone

Shadow testing sends a copy of real, live production traffic to a new version running alongside the current one, without ever showing that new version’s output to any actual user. This is the lowest-risk way to observe how a change behaves against genuine traffic patterns — real prompt lengths, real context, real edge cases that a synthetic test might never think to include, per the same reasoning covered in benchmarking inference systems — because whatever the new version does, correct or not, never reaches a real user while it is being observed this way.

Canary deployment: a small, deliberate slice

Once shadow testing has given some confidence, a canary deployment sends a small, deliberately limited fraction of real traffic to the new version — commonly starting as low as one percent, or even lower for a particularly high-stakes change — and widens that fraction gradually as confidence grows, rather than switching every user over in one step. If something goes wrong, only the canary’s limited slice of users is affected, and the rollout can be halted and reverted before the blast radius grows any further. This is exactly the same staged-exposure principle used throughout software deployment generally, applied here to a model or serving-configuration change specifically.

What to watch for

A full conference session on shadow-testing methodology for safely rolling out model and serving-container changes, including a real case study. It predates the specific LLM-serving concerns this roadmap covers — quantization scheme regressions, model-version latency shifts — and is framed around general ML model endpoints rather than GPU-based LLM inference specifically. This is genuinely the weakest video pick in this roadmap: extensive searching found no conference talk or platform video addressing canary or shadow rollout specifically for an LLM serving stack, which is worth knowing plainly rather than pretending the fit is closer than it is. The underlying methodology transfers directly even though the examples do not match this roadmap’s specifics.

What is specific to rolling out an LLM serving change

A regression in a traditional ML model endpoint usually shows up as a clear error or an obviously wrong prediction. An LLM serving regression can be far subtler: a new quantization scheme might degrade output quality in ways that are not obviously wrong, only slightly worse — missed by a shallow smoke test but real and measurable at scale. This is exactly why the eval sets and quality metrics covered in the previous lesson have to be part of what a canary rollout actually checks, not just latency and error rate: a canary that only watches for crashes and slow responses can sail straight past a genuine quality regression that a dedicated quality check would have caught immediately.

Where this leaves the whole roadmap

Testing and safe deployment is the last stage of the loop this roadmap has built toward across every group: choose hardware and a serving stack, tune it for speed and cost, monitor it in production, and change it carefully when something needs to improve — then watch the result and repeat. Nothing in production inference is ever finished in the sense a training run finishes; it is operated, observed, and improved continuously, which is the discipline this entire roadmap has been describing from its very first lesson onward.