Govern the collection and use of data in training and testing the AI model and system.
- ✓Assess and document lawful rights to every source before training.
- ✓Check quality, quantity, integrity and fitness for purpose as separate dimensions.
- ✓Fit for purpose includes representing the population the system will serve.
- ✓Rights and origin before technical work; cleaning before splitting.
- ✓Data owners decide how data may be used; data stewards keep it in good order. Lawful still has to pass an ethics check.
- Lawful rights to data:
- The documented legal basis, consent, license or contract that permits an organization to collect and use a dataset for a specific purpose.
- Purpose limitation:
- The principle that data collected for one purpose shouldn't be reused for an incompatible one.
- Data integrity:
- Assurance that data has stayed complete and unaltered, protected from corruption and tampering.
- Fitness for purpose:
- Whether data suits the specific task and population a model will serve, beyond being large and accurate.
- Data minimization:
- Using only the personal data a purpose actually needs.
- Data owner vs. data steward:
- The owner is the accountable leader who decides how a dataset may be used; the steward manages its quality, documentation and day-to-day handling.
Exam tips: Lawful rights come FIRST. Options that start cleaning, deduplicating or modeling before rights are checked are premature. · 'Publicly available' isn't 'licensed for training', and 'anonymize it and the rights problem disappears' is a trap. · If the stem concedes the data is big and accurate but a reviewer still objects, the answer is fitness for purpose. · Cross-border data: review origin, destination and controlling law BEFORE any processing.
- ✓Provenance is origin; lineage is the journey to the model.
- ✓Both make accuracy, representativeness and bias checkable.
- ✓They power incident forensics, rights requests and reproducibility.
- ✓Use source records, transformation logs, dataset versioning and access logs, automated where possible.
- Data provenance:
- The origin of data: who created or collected it, when, and under what authority, consent or license.
- Data lineage:
- The traceable path of data from its source through every transformation into a model.
- Dataset versioning:
- Keeping identifiable versions of datasets so each model can be linked to the exact data it used.
- Chain of custody:
- An unbroken record of who handled something and what they did, borrowed from evidence handling.
Exam tips: Provenance = where it came from (origin, authority, license). Lineage = what happened to it (transformations). Swapped definitions are a classic distractor. · 'BEST way to ensure training data is accurate, representative and unbiased' → track lineage and provenance. · Never 'assume purchased data is accurate'. Vendor promises are not evidence. · Diagnose with lineage before fixing. Synthetic top-ups or threshold tweaks before understanding the cause are premature.
- ✓Train on training data, tune on validation data, judge once on test data.
- ✓Watch for overfitting, underfitting, leakage and contamination.
- ✓Seven test types: unit, integration, validation (meets requirements), performance (speed and scale), security, bias, interpretability.
- ✓Automate for coverage, test edge cases and malicious inputs, add human interpretation, and choose metrics that match the harm.
- Validation set:
- Data held back from training and used during development to tune settings and compare model versions.
- Test set:
- Data kept completely separate and used once, at the end, to estimate real-world performance.
- Overfitting:
- When a model memorizes its training data and performs poorly on new data.
- Underfitting:
- When a model is too simple or undertrained to capture the pattern, performing poorly everywhere.
- Data leakage:
- When a training feature carries information about the outcome that won't be available at prediction time, inflating scores.
- Precision and recall:
- Precision is the share of flagged cases that are truly positive; recall is the share of true positives the model catches.
Exam tips: Training and test data must stay separate. Options that evaluate on training data, or tune against the test set, are traps. · Overfitting = great on training, poor on new data. Underfitting = poor everywhere. · Full user-base coverage → automated testing. Blind trust in automated metrics → automation bias. The BEST evaluation adds human interpretation. · Bias test fails before release → diagnose, remediate, re-test, document. 'Ship and monitor' and 'just delete the protected attribute' are traps.
- ✓NIST groups bias as systemic, statistical/computational and human.
- ✓Know the causes: sampling, historical, measurement, temporal, computational, confirmation.
- ✓One dataset can carry several biases, each needing its own fix.
- ✓Examine, pick metrics, mitigate, re-test, document, then keep watching.
- Sampling bias:
- Bias from a training sample that doesn't represent the population the system will serve.
- Historical bias:
- Bias from data that accurately records a past shaped by unfair treatment.
- Measurement bias:
- Bias from using a feature or label that is a poor or uneven proxy for what you actually care about.
- Temporal bias:
- Bias from data that is unbalanced across time, so it no longer reflects current reality.
- Systemic bias:
- Bias embedded in institutions and society that flows into data even when each step looks neutral.
- Confirmation bias:
- The human tendency to favor information that supports existing beliefs.
Exam tips: Bias questions hinge on the CAUSE: who was sampled (sampling), when (temporal), what was measured (measurement), an unfair past (historical), an external skew (computational). · 'Remove the protected attribute' is rarely the BEST answer: proxies carry the signal. · 'More data' only helps if it's the RIGHT data. More of a skewed sample or an outdated era makes things worse. · Temporal bias fix = recent, representative data (reweight or restrict the window), not more history.
- ✓Data poisoning corrupts training data; defend the pipeline with lineage, access control and integrity checks.
- ✓Models can memorize and leak personal data, so minimize first.
- ✓Differential privacy adds noise; federated learning keeps data home; synthetic data fills gaps; homomorphic encryption, SMPC and TEEs protect data while it's being computed on.
- ✓Anonymization is irreversible; pseudonymization isn't.
- Data poisoning:
- An attack that inserts corrupted or manipulated data into a training set so the model learns harmful or wrong behavior.
- Backdoor:
- A hidden behavior planted through poisoning that activates only when a specific trigger appears.
- Differential privacy:
- Adding calibrated statistical noise so individuals can't be singled out while overall patterns survive.
- Federated learning:
- Training a model across separate data holders so raw data stays put and only model updates are shared.
- Synthetic data:
- Artificially generated records that mimic real data, used to fill gaps, cover rare cases and improve accuracy.
- Pseudonymization:
- Replacing identifiers with codes that a separately held key can reverse; still personal data for whoever can re-link it.
Exam tips: Map each PET to ONE job: noise (differential privacy), train in place (federated learning), generate new records (synthetic), strip identifiers (anonymization), swap for a key (pseudonymization), compute on encrypted data (homomorphic encryption), joint result without sharing inputs (SMPC), hardware enclave (TEE). 'Interchangeable' is the trap. · Poisoning = training time; evasion = inference time. Poisoning defenses are data-governance controls: lineage, access control, integrity checks. · Raw data legally can't move → federated learning. Real examples too scarce → synthetic data. · Pseudonymized isn't anonymized. If re-linking is possible, it's still personal data.
- ✓Documentation lets others validate results, establishes compliance and manages risk.
- ✓Datasheets describe datasets in seven sections, from motivation to maintenance.
- ✓Model-card inputs (data, metrics, factors, per-group results) are captured during testing.
- ✓Test reports record versions, methods, thresholds, failures, fixes and sign-off.
- Datasheet for a dataset:
- A standard document describing a dataset's motivation, composition, collection, processing, uses, distribution and maintenance.
- Model card:
- A short summary of a trained model's purpose, performance (including by group), limitations and intended use.
- Test report:
- A dated record of what was tested, how, against which criteria, the results including failures, remediation and sign-off.
- Reproducibility:
- The ability for others to obtain consistent results by repeating the documented process.
Exam tips: Datasheet = the DATASET (motivation, composition, collection…). Model card = the MODEL (purpose, performance, limits). Don't swap them. · Records must include failures and remediation, not just the final passing score. · Thresholds are documented BEFORE testing. Anything set after seeing results is goalpost-moving. · The model card is finalized at release (III.C), but its inputs come from training and testing records.
- ✓TEVV runs across the life cycle: verification checks the spec, validation checks the real-world fit.
- ✓False positives are false alarms, false negatives are misses; context decides which hurts more.
- ✓Accuracy hides rare-event failures; precision, recall and F1 tell the truth.
- ✓Fairness metrics (parity, equal opportunity, calibration) can conflict, so choose and document one up front.
- ✓Explain globally and locally (SHAP, LIME, counterfactuals), and check calibration and repeatability so confidence means something.
- TEVV:
- Test, evaluation, verification and validation: NIST's umbrella term for the checks run across the AI life cycle.
- Verification vs. validation:
- Verification confirms a system meets its specification (built it right); validation confirms it fulfills the requirements for its intended use (built the right thing).
- False positive / false negative:
- A false positive flags something that isn't there (a false alarm); a false negative misses something that is.
- F1 score:
- A single score that balances precision and recall; it falls sharply if either one is poor.
- Fairness metrics:
- Yardsticks such as demographic parity, equal opportunity and calibration that compare outcomes across groups; when base rates differ, they can't all be met at once.
- Counterfactual explanation:
- An explanation that states the smallest change to the inputs that would have changed the outcome.
Exam tips: Rare outcome plus a big accuracy number = the base-rate trap. Ask for recall, precision and results by group. · Precision guards against false alarms; recall guards against misses. Pick the metric that matches the costlier error in the scenario. · Verification = built it right (meets the spec). Validation = built the right thing (fits the intended use). Repeatability = same team, same setup; reproducibility = someone else, from your documentation. · No model meets every fairness metric when base rates differ: the BEST answer picks one up front, ties it to the harm and documents why. 'What would I need to change?' calls for a counterfactual explanation.
- ✓Write the test plan first: objectives, scope, edge and malicious cases, criteria, roles.
- ✓Independent validation, peer review and gates make results credible.
- ✓Re-test after every change: regression testing keeps old passes valid.
- ✓Watch for label noise, class imbalance, poor calibration, environment mismatch and feedback loops.
- ✓Layer mitigations and weigh their trade-offs.
- Test plan:
- A document written before testing that sets objectives, scope, scenarios, metrics, pass/fail criteria and roles.
- Regression testing:
- Re-running tests after any change to data, code, features, hyperparameters or thresholds, to confirm earlier results still hold.
- Inter-rater reliability:
- How consistently independent labelers give the same label to the same item; low agreement signals label noise.
- Feedback loop:
- When a model's outputs influence the data it is later trained or tested on, reinforcing its own patterns.
- Defense in depth:
- Layering safeguards at different points in a system so a single failure doesn't cause harm.
Exam tips: Plan before, report after. Pass criteria written after seeing the results are goalpost-moving. · Any change (data, features, code, hyperparameters, thresholds) → regression testing. 'It was only a small tweak' is the trap. · Builders don't validate their own work alone. Look for independence, peer review and formal gates. · Layered mitigations beat one perfect filter, and every layer has trade-offs (over-blocking, delay, cost) that must be weighed and documented.
Independent study aid, not affiliated with the IAPP. Written with AI from primary sources; laws change, so check the sources. Last updated 7 Nov 2026.
