A missed fraud costs 50 and a false alarm costs 1. If the model's probabilities are well calibrated, at which predicted probability of fraud should you start flagging payments?
Options
- 0.5
- About 0.02 (Correct option)
- 0.25
- About 0.98
Answer
About 0.02
Why
Flag when the expected cost of ignoring is higher than the cost of flagging: p × 50 > (1 - p) × 1, which gives p > 1 / 51, about 0.02.
The default of 0.5 is only right when both errors cost the same. 0.98 flips the costs.
This works only if the probabilities are calibrated, so check calibration first.
Remember
Threshold = cost of a false alarm / (cost of a false alarm + cost of a miss).