Data drift vs concept drift

Side by side

When each applies

Data drift happens when the population feeding the model moves away from the training data. A lender expands to a new country. A hospital installs scanners that produce brighter images. A voice assistant gains many users with regional accents. The model may still be right for the old population, but it now sees cases it learned little about. Detection can be fast because it needs no labels. Statistical distance measures, such as the population stability index, compare feature distributions over time.

Concept drift happens when the world changes the answer. Fraud rings change tactics. After a rate shock, the same debt ratio predicts more defaults. Users start treating promotional newsletters as spam. The inputs can look normal, so input monitoring may miss it entirely. Outcome data is the reliable way to reveal it. The research literature describes concept drift as sudden, gradual, incremental or recurring, as with seasonal patterns.

Either can cause model drift, the performance decay you actually observe. A related idea, label or prior shift, is a change in the mix of outcomes, such as fraud rising from 1% to 3%.

Governance frameworks expect you to watch for all of these. Appendix B of the NIST AI RMF notes that AI systems may need more frequent maintenance because of data, model or concept drift. Its MEASURE 2.4 calls for monitoring system behavior in production. Under the EU AI Act:

• providers must reduce the risk of biased feedback loops in systems that keep learning (Art. 15(4));
• retraining not pre-determined in the initial conformity assessment can be a substantial modification needing a new assessment (Arts. 3(23) and 43(4));
• providers run post-market monitoring (Art. 72), and deployers monitor operation and suspend use if a risk appears (Art. 26(5)).

Worked example

Quillfield Credit Union, a fictional lender, scores personal-loan applications with a model trained on five years of past applicants.

Data drift. Quillfield launches a loan product for gig workers. Applicants now show irregular income patterns that were rare in the training data. Input monitoring flags the income-volatility feature within the first month. No defaults have happened yet, but the model is scoring people it barely saw in training. The team routes the new segment to human underwriters and collects data for retraining.

Concept drift. A year later, interest rates rise sharply. Applicants look much as before, and the input monitors stay quiet. Six months on, loans with debt-to-income ratios the model rated safe are defaulting more often. The link between that ratio and repayment has changed, and only outcome tracking caught it. The team retrains on recent labeled outcomes, re-validates accuracy and fairness, and releases the update through change management.

Input monitoring alone would have missed the second problem. Outcome monitoring alone would have caught the first one months late.

The exam trap

The classic confusion is calling every performance drop "concept drift," or calling a new population "concept drift." The tell is one question: did the inputs change, or did what they mean change?

• New users, new sensors or a new market: data drift.
• Inputs that look the same, but the right answer moved: concept drift.
• Accuracy sliding over time: model drift, the symptom rather than a third cause.

Watch for the look-alikes too. Covariate shift is data drift. Overfitting is a training problem, not drift. Retraining only on the last few weeks of data can introduce temporal bias. An alert justifies diagnosis first, then a controlled retrain. Practice sorting the two in lesson III.C.4.