A lender trains a default model only on past loans that were approved, then uses it to score all applicants. What is the main risk?
Options
- Class imbalance
- Concept drift
- Underfitting because there are too few features
- Selection bias: there are no outcomes for the kind of applicants who used to be rejected, so scores for them are unreliable (Correct option)
Answer
Selection bias: there are no outcomes for the kind of applicants who used to be rejected, so scores for them are unreliable
Why
Outcomes (repaid or defaulted) exist only for approved applicants, so the training sample is not representative of everyone who will be scored. The model extrapolates for groups the old policy excluded, and an evaluation on approved loans looks better than reality.
Mitigations: a small controlled group that is approved regardless, reject inference techniques (used with care), conservative thresholds at first, and monitoring of the new population.
Cross-validation cannot detect this, because every fold shares the same bias.
Remember
Labels exist only where past decisions allowed them.