The model was fine at deploy. It is not fine now.

What actually happens

A model that has degraded does not throw errors. It returns confident answers that are gradually less correct, which is the hardest kind of failure to notice.

A model is validated, approved and deployed. Its accuracy on the validation set is documented and its serving infrastructure is monitored for latency, error rate and throughput.

The world it serves then changes. Customer behaviour shifts, an upstream system starts populating a field differently, a category that barely existed at training time becomes common. The input distribution moves away from the one the model was trained on.

Nothing fails. The model receives inputs, produces outputs, and returns them within its latency budget. Every operational metric is green, because every operational metric is measuring the serving layer rather than the predictions.

The predictions get worse. Not catastrophically, and not uniformly: they degrade most on the inputs that have drifted furthest, which are often the ones that matter most because they represent whatever is new in the business.

It surfaces when someone downstream questions the numbers, typically weeks later, and the first question asked is whether the model has always been like this. Answering that requires ground truth that may not have been retained.

Every operational metric was green throughout, because every operational metric was watching whether the model responded, not whether it was right.

The same deployment, two ways

Measured in weeks of degraded decisions rather than minutes, and in whether the evidence to diagnose it still exists.

There is no clean cost figure to offer here, and any that we invented would be worse than none. The cost of degraded predictions depends entirely on what decisions they drive, and only you know that.

What is generalisable is the detection mechanism, and the reason the current one fails: the serving layer and the prediction quality are different things, and monitoring the first tells you almost nothing about the second.

Input distribution drift is observable without ground truth, which is what makes it usable. Waiting for labelled outcomes to measure accuracy means waiting weeks or months. Comparing the serving input distribution against the training reference is possible on the first day it moves.

Correlating drift back to upstream changes is what makes it actionable rather than merely alarming. Drift on its own tells you the model is now serving a different world; the pipeline change that caused it tells you whether to fix the data or retrain the model.

Who decides to press go

Rolling back a model, restricting a segment and retraining are three different actions with three different risk profiles.

Drift detection, upstream correlation, segment impact quantification and option generation run under policy. None of them change what is served.

The Action Ticket carries the options with their consequences, the affected segments and the rollback path. The model owner approves the specific action before anything changes in serving.

Rollback is treated as a first-class option rather than a failure state, because reverting to a model that was correct for the previous distribution is frequently the right immediate answer while retraining runs.

Model serving normally. Latency, error rate and throughput all green. Input distribution moving away from the training reference on several features.

Upstream data sources, recent schema and pipeline changes, and affected feature segments correlated. Upstream field population change identified as the cause.

Action Ticket raised with rollback, segment restriction and retraining options, each with its consequences. Held for the model owner to choose.

Training input distribution retained as a standing reference for every deployed model. Upstream schema changes wired into the drift correlation so cause arrives with detection.

This is a platform capability, not a published customer deployment for this exact scenario. The mechanism, which is cross-system correlation followed by governed MOP execution with pre-check, post-check, rollback and approval gating, is running in production today across managed estates; see governed day-2 operations across 2,000+ nodes and closed-loop network automation. The timings shown are modelled, not measured at a named customer.