Shipping and monitoring

Someone else’s capacity

The defining operational fact of shipping an LLM feature is that the most important component of it runs on hardware you do not own, shared with every other customer of that provider at the same moment. Rate limits are not an error condition to be surprised by; they are the ordinary, expected way shared capacity gets rationed across customers. Overload responses arrive during other customers’ traffic peaks that have nothing to do with your own usage pattern. Latency genuinely varies by time of day for reasons entirely outside your own system’s control. Designing around this reality, rather than treating every individual occurrence as a one-off incident to investigate, is the single biggest mindset shift from operating a typical internal service to operating one built on a third-party model API.

Canary and shadow deployment, adapted for non-determinism

A canary deployment rolls a change out to a small percentage of real traffic first, watching key metrics closely before expanding to everyone — a standard practice already familiar from ordinary software deployment. Applied to an LLM feature, the metrics watched need to include quality signals specific to this domain, not only the usual latency and error rate: a canary for a new prompt version should also be compared against the eval set score and against real-time proxies like retry rate on that slice of traffic, not solely against infrastructure-level health signals that would miss a purely behavioural regression entirely. A shadow deployment goes a step further, running a new prompt or model version in parallel against a real copy of live traffic without ever actually serving its output to any user, comparing its outputs directly against the current production version’s outputs on exactly the same real inputs before the new version serves a single real user — valuable specifically because it surfaces behavioural differences risk-free, before any user is ever exposed to whatever the new version might get wrong.

Feature flags for prompts and models

Treating a specific prompt version and a specific model version as values behind a feature flag, rather than hard-coded directly into application logic, allows an instant, zero-deploy rollback the moment a regression is detected in production — reverting the flag rather than needing to revert and redeploy a code change. It also enables targeted experimentation, running two prompt variants side by side on different user segments and comparing their real outcomes directly, which is the A/B testing half of evaluation applied specifically to production traffic rather than an offline eval set.

Fallbacks worth actually having in place

A second provider or model, with its prompt already tested and known to work against it ahead of time, is the fallback teams most often skip entirely and most often regret skipping specifically during an outage — and it is far easier to set up calmly in advance than to improvise under the pressure of a live incident. A degraded mode — serving retrieval results alone without a final generation step, for instance — is frequently genuinely useful on its own and costs essentially nothing to serve when the generation model itself is unavailable. A cached answer for genuinely common, repeated requests can serve something slightly stale rather than nothing at all during an outage. An honest failure message, used when none of the above options actually applies to the situation at hand, stating plainly that the feature is temporarily unavailable, is reliably better than a fabricated answer produced under pressure to appear to be working when it genuinely is not.

Versioning prompts and models as first-class deployed artefacts

Prompts are logic, exactly as covered in prompt anatomy, and a change to one can alter a system’s behaviour as thoroughly as any ordinary code change can. Treat them accordingly: held in version control, reviewed by another person before merging, deployed together with the rest of the application rather than edited live and directly in a provider’s console, and rolled back through exactly the same mechanism as any other code change. Pin specific model versions wherever a provider allows it — an endpoint that silently points at whatever the provider’s latest model happens to be at any given moment is effectively an undeployed, unreviewed change to your product landing on its own schedule, and you will generally find out about it from an affected user rather than from your own release notes. Pinning converts that silent, uncontrolled drift into a deliberate upgrade you schedule yourself, run your full eval set against before switching, and roll out on your own terms. Recording which specific prompt version and which specific model version produced each individual trace means that when quality visibly shifts in either direction, the first diagnostic question — what actually changed — has a direct, concrete, answerable answer rather than requiring reconstruction after the fact from memory.

SLOs adapted for a genuinely non-deterministic system

Traditional service-level objectives, built around uptime percentage and fixed latency percentiles, still apply here and remain worth tracking, but they are not sufficient on their own for an LLM feature, because a feature can be fully “up” by every traditional measure and still be producing systematically poor answers. A meaningful SLO set for an LLM feature usually adds an explicit quality-related objective — eval score staying above a defined floor, or a user-facing proxy like negative-feedback rate staying below a defined ceiling — tracked and alerted on with the same operational seriousness as the traditional infrastructure metrics. A postmortem culture for this kind of system has to adapt accordingly too: “this specific run could not be exactly reproduced” is an expected, routine property of a probabilistic system working as designed, not itself evidence that the original investigation was somehow incomplete, and a useful postmortem here focuses on the trace of what actually happened in the specific failing run, the category of failure it represents, and what eval case or guardrail should be added to catch that category going forward — rather than chasing an exact reproduction that a non-deterministic system may simply never reliably give you.

The loop this whole roadmap has been building toward

Watching quality in production is not a separate concern bolted onto the end of building a feature; it is the thing every other group in this roadmap has been assembling pieces of. Ship a feature. Trace every real run, per tracing and observability. Read a genuine sample of the failures that show up in those traces, weekly if at all possible, rather than relying purely on automated metrics to surface everything that matters. Turn the patterns found in that reading into new, concrete eval cases per building an eval set. Fix what the evals now catch. Ship the fix, and watch it again. That closed loop — ship, trace, read, measure, fix, ship again — is the actual discipline behind every lesson in this roadmap, far more than any single technique covered along the way.