Most requests to your assistant are simple, and a few are hard. How do you cut cost without hurting quality on the hard ones?
Options
- Always use the largest model
- Route by difficulty: send simple requests to a smaller, cheaper model and escalate to a larger one when the request is hard or the first answer looks unreliable, measuring the quality with evals (Correct option)
- Randomly send half of the traffic to a cheaper model
- Truncate every prompt to 100 tokens
Answer
Route by difficulty: send simple requests to a smaller, cheaper model and escalate to a larger one when the request is hard or the first answer looks unreliable, measuring the quality with evals
Why
Smaller models are cheaper and faster, and good enough for many tasks. The saving comes from not paying for the big model when it is not needed.
A router can be rules (the type of request), a classifier, or a confidence signal from the first model. Verify with evals that quality holds on each route.
Random routing and blind truncation lose quality in an unpredictable way.
Remember
Route easy work to small models, escalate when needed.