Continuous delivery for models

The handoff that CD removes

By the time a model has passed its evaluation gate — it cleared the quality bar, it beat the current production model, it’s the output of a continuous training pipeline that did its job — there’s still a gap between “we have a good model” and “that model is running in production.” In a shop without continuous delivery, that gap is a person: someone pulls the model file off wherever training left it, checks it matches the run that was approved, figures out what dependencies it needs, builds a serving container by hand, and pushes it somewhere. Every one of those steps is a place for drift to creep in — a slightly different library version in the hand-built container than what was used at training time, a wrong file picked up because two runs finished close together, a preprocessing script that got updated after training but before the manual packaging step. Continuous delivery for models is the automation of exactly that handoff: given a model that has passed its gate, produce, deterministically and without a human in the loop, a packaged artifact that is ready to deploy.

What “packaged and deployable” actually requires

The core mistake to avoid here is treating a model’s deployable artifact as just its weights file. A set of weights on its own is close to useless: a downstream service also needs the exact preprocessing logic that turned raw input into the features the model expects, the exact versions of the libraries the model was built with (a different version of a tokenizer or a numeric library can silently change output), and metadata tying the artifact back to the training run, code commit, and dataset version that produced it. This is the training-serving consistency problem showing up again in a new form: if the CD pipeline packages the weights but not the matching preprocessing code, you can end up with a model trained on one feature encoding being served behind different preprocessing — producing predictions that look plausible and are systematically wrong. A model-packaging format (containers, or a standard format like the one covered earlier in this roadmap) exists precisely to bundle all of this together as one unit so that “the artifact that passed evaluation” and “the artifact that gets deployed” are provably the same thing, not just probably the same thing.

The registry as the system of record

The piece that makes this pipeline actually automatable end to end, rather than just scripted-but-manual, is a model registry: a tracked store of model versions, each tagged with a stage — staging, production, archived — and its full lineage back to the run that created it. Without a registry, “deploy the latest approved model” has no unambiguous answer; someone has to know which of several candidate artifacts is the right one. With a registry, promotion from “passed evaluation” to “production” is a status change on an already-packaged artifact, and a deployment step can be as simple as querying the registry for whatever is currently tagged production and pulling it. This is also what makes rollback tractable: if a newly promoted model misbehaves in production, reverting means pointing the stage label back at the previous version, not reconstructing it from scratch. The registry is what turns “a model that passed its tests” into something a fully automated pipeline can act on without a person interpreting the result.

Why this closes the loop

Put the whole group of lessons together and the shape becomes clear: CI catches problems in code, data, and config on every change; layered testing makes sure a passing pipeline actually means a trustworthy model; continuous training keeps candidates coming without waiting on a human’s schedule, gated so a worse model can’t silently replace a better one; and continuous delivery takes a gated, approved candidate and turns it into something that can actually be deployed, consistently, every time. Skip any one of these and the others lose most of their value — automated training without automated, trustworthy delivery just produces validated models that still require a risky manual step to ship. Together, they’re what makes “deploy models as often as the data demands” something a team can do safely rather than something that only works until the one time a manual step gets skipped.