Testing ML systems

One number is not a test suite

A software test suite is a list of specific claims: given this input, expect that output. Accuracy, by contrast, is a single number averaged over an entire dataset, and averages are excellent at hiding exactly the kind of failure you most need to catch. A spam classifier can sit at 97% accuracy while reliably misclassifying every email that mentions a competitor’s product name, because that pattern shows up in a small enough slice of the test set to barely move the aggregate. A resume screener can look fine in overall accuracy while systematically scoring one demographic lower, again invisible in the top-line number. Treating “accuracy is high” as equivalent to “the model is tested” throws away the one thing that makes testing useful in traditional software: the ability to assert something specific and know immediately when it breaks. ML systems need that same specificity, applied at three different layers, because each layer fails in a different way and for a different reason.

Layer one: does the data look like what we expect

The earliest and cheapest place to catch a problem is before training even starts. Data validation tests assert things about the input: this column should never be null, this categorical field should only take these five values, this numeric field should fall in this range, the label distribution shouldn’t have shifted by more than X from last week’s. None of this checks the model — it checks the ingredients. The reason this layer has to exist is that a corrupted or drifted dataset usually doesn’t cause training to fail; it causes training to succeed on bad data, producing a model that learned the wrong thing with total confidence. By the time accuracy on a held-out set looks off, the bad data is already baked into every weight, and diagnosing “which of the thousand recent changes caused this” is far harder than it would have been to catch at ingestion.

Layer two: does the training code do what it’s supposed to

This layer looks the most like conventional software testing, because it is: ordinary unit tests against the deterministic parts of a training pipeline. Does the feature-engineering function turn a raw timestamp into the right day-of-week encoding? Does the class-balancing step actually produce balanced batches? Does a custom loss function return the expected value on a hand-computed example? These are fast, isolated, and exactly as valuable as unit tests always are — they catch a broken transform before it ever touches real data, and they’re the layer most teams already know how to write, because it requires no ML-specific tooling at all, just ordinary testing discipline applied to the pipeline’s functions.

Layer three: does the trained model behave sanely

The layer most teams skip is testing the model itself, not just the code that produced it, against specific known cases — the ML equivalent of a unit test’s “given this input, expect that output.” A sentiment model should flip its prediction when you negate a sentence (“great service” vs. “not great service”); if it doesn’t, something about how it weights negation is wrong, and no aggregate accuracy score reliably surfaces that single failure mode. A credit model should behave consistently when you swap a protected attribute like a name associated with a different demographic while holding everything else fixed; if the prediction moves, that’s a specific, testable fact about the model worth catching before deployment, not after a complaint. These behavioral tests don’t replace a held-out evaluation set — they complement it by asserting invariants (the model should behave this way under this transformation) and performance on hand-picked, known-hard cases that a random sample might never happen to include.

Why all three together

Each layer catches a failure mode invisible to the other two: bad data can produce a model that trains cleanly and scores fine on data drawn from the same bad distribution; broken training code can compute a correct-looking loss while silently never updating the right weights; and a model can have excellent aggregate accuracy while failing hard on a specific, predictable case nobody wrote a test for. Treating any one layer as sufficient is how teams end up surprised by a production failure that, in hindsight, a five-line assertion would have caught during review. Building this layered discipline is also what makes the next step possible: once you can trust that a model has actually passed real tests rather than just “the pipeline ran,” you can start letting retraining happen automatically instead of only by hand, which is exactly the subject of continuous training.