Model drift

In plain English

A trained model is a photograph of the world on the day its data was collected. The world keeps moving. Customers change, fraudsters adapt, prices rise and new sensors arrive. Model drift is the growing gap between that photograph and reality, and it shows up as worse predictions.

Model drift is the symptom. Two causes explain most cases:

• Data drift: the inputs change, such as a new customer group.
• Concept drift: what the inputs mean changes, such as fraudsters shifting tactics.

Our data drift vs concept drift comparison shows how to tell them apart. A quick test: did the inputs change, or did the right answer move?

Example

Elmstead Clinics, a fictional outpatient group, uses a model to predict which patients will miss appointments, so staff can call them first. Recall was 78% at launch. Eighteen months later, monitoring shows it has fallen to 61%, below the 70% alert threshold.

The model owner diagnoses before acting:

1. Input checks show far more video appointments than the training data contained: data drift.
2. Outcome checks show patients with long journeys miss fewer appointments, because video visits removed the trip: concept drift.
3. The decline is worst for patients over 70, so fairness is affected too.

The team retrains on recent, representative data and re-validates accuracy and fairness. It updates the model card and releases through change management with rollback ready. It also adds appointment type to the drift dashboard.

Why it matters

• NIST. The AI RMF notes that AI systems may need more frequent maintenance because of data, model or concept drift (Appendix B). MEASURE 2.4 calls for monitoring system behavior in production. MANAGE 4.1 calls for post-deployment monitoring plans that include change management.
• EU AI Act, providers. Providers of high-risk systems run post-market monitoring (Art. 72). Systems that keep learning must reduce the risk of biased outputs feeding future inputs (Art. 15(4)). Retraining not pre-determined at the initial conformity assessment can be a substantial modification that needs a new assessment (Arts. 3(23) and 43(4)).
• EU AI Act, deployers. Deployers monitor operation and inform the provider of risks (Art. 26(5) of the AI Act).
• Vendor models. Deployers usually cannot retrain a bought model. Their levers are thresholds, configuration, data refreshes, human review and escalation to the vendor.

How the AIGP exam tests it

Drift appears in lesson III.C.4 from the provider's side and in lesson IV.C.2 from the deployer's side.

• The tell. Accuracy sliding after deployment is model drift. Then ask what caused it.
• Trap 1: calling every drop concept drift. New users or new sensors are data drift. The same inputs with a new right answer are concept drift.
• Trap 2: overfitting vs drift. Overfitting is a training problem: the model memorized its training data. Drift happens after deployment, when the world changes.
• Trap 3: the two extremes. "Freeze the model" and "retrain only when it visibly breaks" are both wrong. The answer is monitoring plus a schedule plus triggers.
• Trap 4: emergency retraining. An alert justifies diagnosis first. Retraining still goes through re-validation and change management. Retraining only on the last few weeks of data can introduce temporal bias.