Your RAG assistant gives wrong answers. How do you find out whether retrieval or generation is at fault?
Options
- Check, on a labelled set of questions, whether the retrieved chunks contain the answer (for example recall@k), and separately whether the answer follows from those chunks (Correct option)
- Raise the temperature and see if answers change
- Compare the length of right and wrong answers
- Only collect thumbs up and thumbs down from users
Answer
Check, on a labelled set of questions, whether the retrieved chunks contain the answer (for example recall@k), and separately whether the answer follows from those chunks
Why
RAG has two stages that fail in different ways. If the right passage was never retrieved, no prompt can fix it. If it was retrieved and the answer is still wrong, the problem is in the prompt or the model.
Measure the stages separately: retrieval with recall@k or hit rate against known relevant passages; generation with faithfulness to the retrieved text.
Thumbs up and down are useful signals but do not say where the failure is.
Remember
Evaluate retrieval and generation separately.