How would you choose evaluation metrics for a fraud detection model?
Answer
- Start from the business: what does a missed fraud cost, and what does a false alarm cost (review time, annoyed customers)?
- Fraud is rare, so skip accuracy. Use precision, recall and PR-AUC; ROC-AUC is a secondary view.
- Fix an operating point: for example the best recall at a precision the review team can handle, or the threshold that minimises expected cost.
- Split by time, not at random, and keep the latest period as the test set.
- Check segments (new customers, countries, amounts) for hidden weak spots.
- After launch, track the same metrics on labelled cases, plus the number of alerts per day.
Practice this question in Repaso
