MLOps: Getting Models to Production and Keeping Them Honest
The hard part of machine learning is not training a model. It is everything that happens after — deploying it, trusting it, noticing when it quietly stops working, and being able to fix it without a forensic investigation. That discipline is MLOps, and it is where most enterprise ML value is won or lost. Plenty of organisations have a model that scored well in a notebook and no idea whether the version running in production today is still any good. This is a guide to closing that gap.
A useful way to frame MLOps: a machine learning model is the only kind of software that can be completely broken while every server is healthy, every request returns 200, and every dashboard is green. Nothing crashes. The predictions simply become wrong, and without specific instrumentation, nobody finds out until a person downstream notices the decisions have been bad for a while. Traditional operations does not catch this, because traditional operations was not built to.
The problem that makes ML different: drift
A model encodes patterns from the data it was trained on. It implicitly assumes the world it sees in production resembles the world it learned from. The moment that stops being true, accuracy erodes — and the world always changes eventually. Customer behaviour shifts, a new product launches, an upstream system starts formatting a field differently, a pricing change alters the mix of transactions. The model keeps answering with total confidence based on a reality that has moved.
This is drift, and it comes in two flavours worth separating. Data drift is when the inputs change — the distribution of what the model sees no longer matches training. Concept drift is subtler and nastier: the inputs look the same, but the relationship between input and correct answer has changed, so the very thing the model learned is now wrong. A fraud model is the classic case; fraudsters adapt precisely to defeat it, so a model that was excellent last quarter can be actively harmful this quarter with no change in the inputs' appearance. Monitoring for both is not optional overhead — it is the core of running a model responsibly.
The foundations: versioning and reproducibility
Before monitoring, you need to be able to answer a deceptively hard question: exactly what produced this prediction? In mature software you can trace a behaviour to a specific commit. In ML you need more, because a prediction is the product of three things that all change independently — the code, the model weights, and the data it was trained on. Version all three. A model artifact in production should be traceable to the exact training data, the exact code, and the exact parameters that created it. Without that, debugging a bad prediction is guesswork, and reproducing a result from six months ago is impossible.
Reproducible training pipelines follow from this. Training should be a defined, automated pipeline that anyone can re-run to get the same model, not a sequence of notebook cells a single data scientist executed in an order only they remember. This is the point where MLOps meets ordinary engineering discipline, and where it leans directly on solid infrastructure automation & DevOps practice: source control, CI, artifact registries, and infrastructure as code apply to models just as they do to services.
Deployment: promote on evidence, and keep a way back
A new model version should never reach production because someone believed it was better. It should reach production because it demonstrably outperformed the current one on a held-out evaluation the pipeline ran automatically. Bake that gate in: a candidate is promoted only if it beats the incumbent on metrics that reflect the business outcome, not just an abstract accuracy score that may not correlate with anything users care about.
Even then, treat every deployment as reversible. Rollback for models is a first-class capability, exactly as it is for application releases — because a model that passes offline evaluation can still behave badly on live traffic in ways the test set did not capture. Progressive rollout helps: route a slice of traffic to the new version, compare its live behaviour against the old, and expand only if it holds up. The ability to put the previous model back in minutes is what makes it safe to iterate at all.
Monitoring: watch inputs, outputs, and outcomes
Production monitoring for ML operates on three levels, and skipping any one leaves a blind spot. Watch the inputs for data drift and for plain data-quality failures — a feature that suddenly arrives null or in a new unit is a leading indicator of trouble. Watch the outputs for shifts in the distribution of predictions: if a model that historically approved thirty percent of cases starts approving sixty, something has changed whether or not it is correct. And, where you can get it, watch the outcomes — the actual ground truth as it arrives — because that is the only true measure of whether the model is still right.
Outcomes are the hard one, because ground truth is often delayed or expensive. You may not learn whether a loan was a good decision for months. This is where honest MLOps does design work: building feedback loops to capture real outcomes as they become known, and using drift in inputs and outputs as the early-warning system for the period before truth arrives. The same instinct powers AIOps on the infrastructure side — detecting that something has changed before the damage is obvious.
Keeping a human in the loop
The most important MLOps decision is often not technical: it is deciding which cases the model is allowed to decide alone. A well-run system routes its low-confidence or high-stakes cases to a person rather than acting on them, and that routing is a designed feature, not a fallback. This matters more than any single accuracy metric, because it bounds the damage a wrong prediction can do. A model that is right ninety-five percent of the time and escalates its uncertain five percent is far safer than one that is right ninety-seven percent of the time and acts on everything. Design for the failures, not just the average.
Does every model need all of this?
No — and pretending otherwise is how MLOps becomes bureaucracy. The weight of process should match the stakes. A model that ranks internal search results and whose errors are mildly annoying does not need the governance of one that approves credit or flags fraud. Scale the discipline to the blast radius. But every model in production needs at least the minimum: know what version is running, know whether it is still working, and have a way to put the last good one back. Teams that skip even that are not running a model; they are hoping.
Getting from a promising prototype to a model you can trust in production, with the right amount of process for its stakes, is precisely the work our AI & machine learning practice does — the unglamorous operational engineering that decides whether ML pays off or quietly stops working.