Concept Drift — in a constantly changing world

Concept drift is defined as a change in the perceived accuracy of an ML model's predictions over time, caused by changes in user expectations, behaviour, and the operational environment. In plain terms: the model has not changed, but the environment it was trained on has. What was once an accurate reflection of reality gradually becomes a snapshot of a world that no longer exists.

The operational environment is not static. Customer behaviour shifts. Markets respond to external events. Cultural and societal changes reshape what people do, want, and expect. A model trained on historical data cannot anticipate these changes — it can only reflect the patterns it has already seen. Consider a straightforward example: the impact of a marketing campaign. If a campaign successfully changes how potential customers behave, the patterns that once predicted their actions may no longer hold. The model has not failed in a technical sense — but its outputs have become less accurate and less useful. Concept drift can be gradual — seasonal shifts in behaviour that accumulate slowly over months. Or it can be abrupt — triggered by external shocks that no training dataset could have anticipated. The COVID-19 pandemic is the clearest recent example. Models used for sales projections and stock market forecasting were built on assumptions about a world that changed overnight. The models did not break. The world did.

Systems that are prone to concept drift should be regularly tested against their agreed ML functional performance criteria. The goal is to detect any occurrence of drift early enough that it can be mitigated before it becomes a material business risk. This is not a one-time activity. It requires an ongoing monitoring and testing discipline — one that treats model performance in production as a live quality concern, not a post-deployment afterthought.

Detection is only the first step. Typical mitigations fall into two categories.

• Retire the system — if the operational environment has changed to a degree where the model's original purpose can no longer be fulfilled reliably, retirement may be the responsible outcome
• Retrain the system — using up-to-date training data that reflects the current operational environment, followed by a structured confirmation and regression testing cycle

Retraining is not simply a technical exercise. It requires the same rigour as the original model release. Confirmation testing validates that the retrained model meets its performance criteria. Regression testing ensures that existing capabilities have not degraded. And where risk warrants it, A/B testing provides the final assurance — the updated model must demonstrably outperform the original before it is trusted in production.