You rewrite a prompt and the answers feel better. What should you do before shipping it?
Options
- Run it on a fixed set of representative test cases and compare the scores with the old prompt (Correct option)
- Ship it and watch the support tickets
- Ask the same model whether the new answers are better and ship if it says yes
- Test it with three inputs you wrote while editing
Answer
Run it on a fixed set of representative test cases and compare the scores with the old prompt
Why
"Feels better" on a few examples is how regressions reach production: a change that fixes one case often breaks another.
An eval set is a fixed list of real and tricky inputs with a way to score the output (exact checks, rubrics, human review or a calibrated judge model). Run old and new on it and compare.
Add each production failure to the set so the same bug cannot return unnoticed.
Remember
Evals before changes, not opinions.