What should an automated canary check before it increases the traffic share of a new model?
Options
- Health metrics (errors, latency, resource use), the distribution of predictions and the business metrics against the baseline, over a large enough sample, with thresholds that trigger a halt or a rollback (Correct option)
- Only that the container starts
- That nobody complained on chat
- That the new model is larger than the old one
Answer
Health metrics (errors, latency, resource use), the distribution of predictions and the business metrics against the baseline, over a large enough sample, with thresholds that trigger a halt or a rollback
Why
A canary is useful only if it decides on evidence. Compare the new version with the baseline running at the same time on comparable traffic, so that time-of-day effects cancel.
Check the service (errors, latency), the model (prediction distribution, a sudden constant or shift) and the product (conversion or whichever metric matters), and define in advance the thresholds and the minimum sample size.
If a threshold is crossed, stop the ramp-up and roll back automatically.
Remember
Canary gates compare against the baseline with thresholds set in advance.