How do you build an eval set for an LLM feature and keep it useful over time?

Start from real inputs: logs, support tickets, or examples written with domain experts. Cover the normal cases, the edge cases and the known failures.
Define the expected result for each case: an exact answer, a checklist or rubric, or the facts that must be present or absent.
Choose the scoring method per case: code checks first (format, exact match, tests), then human review, then a calibrated judge model for the subjective rest.
Keep a small, trusted set for every change, and a bigger one for release checks, so that evals are quick to run.
Split the scores by type of case, because an average can hide a regression in one group.
Add every production failure as a new case, version the set, and never tune the prompt on the cases you use as the final check.