Your app worked well for months, then answer quality drops after the model provider updates a model. How do you protect yourself from this?
Options
- Nothing can be done: models never change
- Set temperature to 0, which freezes the behaviour forever
- Pin a specific model version where the provider allows it, run your eval suite on a schedule and before upgrades, and monitor quality in production (Correct option)
- Read the release notes and trust them
Answer
Pin a specific model version where the provider allows it, run your eval suite on a schedule and before upgrades, and monitor quality in production
Why
Providers update, retire and sometimes silently adjust models. The same prompt can behave differently after a change.
A pinned version removes surprises until you decide to move. Your eval set tells you if the new version is better or worse for your own tasks, and production monitoring catches what the test set misses.
Release notes do not describe how your particular prompts will react.
Remember
Pin versions, run evals on every change.