Positives are very rare (0.1%). Why can ROC-AUC look excellent while the model is poor at finding them?
Options
- ROC-AUC ignores the positive class
- The false positive rate divides by the huge number of negatives, so many false alarms barely move it, while precision collapses (Correct option)
- ROC-AUC is computed from accuracy
- ROC-AUC cannot be calculated with imbalanced classes
Answer
The false positive rate divides by the huge number of negatives, so many false alarms barely move it, while precision collapses
Why
The ROC curve plots recall against the false positive rate, FP / (FP + TN). When negatives dominate, TN is enormous, so even thousands of false alarms leave the rate tiny.
Precision, TP / (TP + FP), is hit directly by those false alarms. A PR curve or average precision shows the problem that ROC hides.
ROC-AUC is still defined with imbalance; it just answers a different question from the one you usually care about.
Remember
Rare positives: check the PR curve, not only ROC.