A churn model has an offline AUC of 0.99 but only about 0.6 in production. One of its features is days_since_account_closed. What is the most likely cause?
Options
- Target leakage: the feature is only known after the customer has already left (Correct option)
- Class imbalance in the training data
- Too many trees in the ensemble
- Underfitting: the model is too simple
Answer
Target leakage: the feature is only known after the customer has already left
Why
A feature that is only filled in after the outcome carries the answer. In training it makes the task almost trivial; at prediction time it is missing or empty, so the score collapses.
Ask of every feature: will I know this value at the moment I need the prediction?
A suspiciously high offline score is a typical sign of leakage.
Remember
Too good to be true: look for target leakage.