Your monthly LLM bill quadrupled with the same traffic after you started pasting a full 30-page manual into every prompt. What is the best fix?
Options
- Ask the model for shorter answers
- Block heavy users
- Lower the temperature
- Retrieve only the relevant passages instead of the whole manual, and use prompt caching for any part that stays the same (Correct option)
Answer
Retrieve only the relevant passages instead of the whole manual, and use prompt caching for any part that stays the same
Why
Cost scales mainly with the tokens you send, and the manual is sent on every call. Shorter answers only reduce the output part, which was not the problem.
Retrieving the relevant passages sends a few hundred tokens instead of tens of thousands. Many providers also offer prompt caching, which makes a repeated prefix cheaper and faster.
Measure tokens per request before and after to confirm the saving.
Remember
Send less input: retrieve and cache.