A new model version is live and the error rate and latency of the service jumped. What is the fastest safe action?
Options
- Roll back to the previous model version, then investigate (Correct option)
- Debug it while it keeps serving everyone
- Retrain the new model right away
- Add more traffic to see if it improves
Answer
Roll back to the previous model version, then investigate
Why
Reduce the damage first, then find the cause. A rollback is only quick if the previous version is still available and deployable, which is a reason to keep versions in a registry.
Investigation then happens offline or in staging, with logs and sample requests from the incident.
Good practice: decide the rollback rule before the release.
Remember
Roll back first, investigate after.