Glossary
Overfitting
Written with AI from the sources below and checked against our facts sheet. Last checked . How this site is made · Spot an error?
Overfitting is when a machine learning model learns its training data too closely, noise and quirks included. It then scores well on that data but performs poorly on new data. The tell is a large gap between training and test performance. It is a training problem, unlike drift, which appears after deployment.
In plain English
Think of a suit tailored so tightly to one mannequin that it fits no real customer. An overfit model has done the same with its training data. It has learned the general pattern plus every quirk, coincidence and bit of noise. Those quirks don't exist in new data, so performance drops.
The opposite failure is underfitting: a suit cut so loosely it fits nobody well. The model is too simple to capture the pattern at all.
To catch both, teams split data three ways. They train on one part, tune on a validation set and estimate real-world performance once on a sealed test set.
Example
Glenfield Insurance, a fictional insurer, builds a model to route claims for review. It scores 98% accuracy on training data and 71% on the held-out test set.
That gap signals overfitting. The team finds a large model trained on only 900 claims. It had learned to recognize individual adjusters' writing habits rather than features of risky claims.
The fixes:
- Gather more claims, representative of all regions and claim types.
- Use a simpler model and remove adjuster-specific features.
- Re-test once on a fresh, sealed test set against a threshold agreed before testing.
- Record the results in the training and test records and the model card.
Why it matters
NIST. Under "valid and reliable," the AI RMF warns that systems poorly generalized to data and settings beyond their training increase risk. Accuracy measures should show the system generalizes beyond its training conditions. Test sets should represent the conditions of expected use.
EU AI Act. High-risk training, validation and testing data must be relevant, sufficiently representative and, as far as possible, error-free and complete (Art. 10(3)). Systems must reach an appropriate level of accuracy and robustness, with accuracy metrics declared in the instructions for use (Art. 15).
Fairness. A model can overfit to the majority group in its data and perform worse for smaller groups. Testing by subgroup catches this. See algorithmic bias.
How the AIGP exam tests it
The testing toolkit, including overfitting, is taught in lesson III.B.3.
Common traps:
- Gap vs floor. A large train-test gap is overfitting. Poor scores everywhere is underfitting.
- Overfitting vs drift. Failing from day one is a training problem. Decaying after launch is model drift.
- Overfitting vs leakage. Data leakage, where a feature secretly carries the answer, makes test scores look too good. Contamination means duplicate records sit on both sides of the split.
- Tuning on the test set. Repeated tuning against the test set turns it into a validation set, so the final score is no longer credible.
The tell: always ask how the model performs on data it has never seen.
Frequently asked questions
What is the difference between overfitting and underfitting?
An overfit model is too closely tuned to its training data: excellent on that data, poor on new data. An underfit model is too simple or undertrained to capture the pattern, so it performs poorly everywhere, training data included. A big gap between training and test scores points to overfitting; low scores on both point to underfitting.
How do you detect overfitting?
Compare performance on the training data with performance on a held-out test set the model has never seen. A large gap is the classic sign. Teams also use a separate validation set during development and check results across subgroups. The test set must stay sealed until the end, or the comparison means nothing.
How do you prevent overfitting?
Common methods include more representative training data, a simpler model and regularization, which penalizes complexity. Teams also use cross-validation, stop training before the model memorizes, and remove features that carry quirks rather than real signal. Governance adds separate test sets, acceptance thresholds set in advance and independent validation.
Is overfitting the same as model drift?
No. Overfitting is a training problem: the model fails on new data from day one. Drift is a monitoring problem: a model that worked at launch decays as the world changes. Data drift means the inputs change; concept drift means the link between inputs and the right answer changes.