The gap between "the model works in my notebook" and "the model is serving predictions reliably in production" is one of the most underestimated engineering challenges in modern software. Studies consistently show that 85–90% of ML projects never make it to production — not because the models are bad, but because the infrastructure, reliability, and operational requirements are vastly more complex than the data science work that preceded them.
The Production ML Stack
A production ML system is not just a model file behind an API. It includes: a feature store that provides consistent features at training and inference time, a model registry (MLflow, Weights & Biases) that versions every trained model with its hyperparameters and training metrics, a serving infrastructure that handles batching, caching, and autoscaling, and a monitoring system that detects data drift and model degradation before they affect users.
- Feature store: Feast or Tecton ensure that the features used during training are identical to those at inference — the "training-serving skew" problem that silently degrades model performance.
- Model serving: FastAPI for low-latency REST endpoints; Triton Inference Server for high-throughput GPU serving; BentoML for managed deployment with built-in monitoring.
- Canary deployments: Route 5% of traffic to a new model version, compare prediction distributions and latency against the baseline, and roll forward only if metrics are satisfactory.
Model Drift: The Silent Killer
A model trained in January on winter shopping patterns will begin to degrade as summer approaches. This is concept drift — the statistical relationship between features and the target variable has changed. Data drift is the related problem: the distribution of input features has shifted (e.g., a new marketing campaign attracting a different demographic). Both require continuous monitoring of prediction distributions, feature distributions, and downstream business metrics.
Retraining Pipelines
Automated retraining pipelines are the final piece: a scheduled job (weekly, or triggered by drift alerts) that pulls recent labelled data, retrains the model with the same hyperparameters (or runs a lightweight hyperparameter sweep), evaluates against a holdout set, registers the new model if it outperforms the current production version, and deploys via canary. This loop — monitor → detect → retrain → evaluate → deploy — is what separates teams that have ML in production from teams that had it in production until it quietly stopped working.