In your RAG system the right chunk is usually around rank 30 of the vector search, so the top 5 passed to the model are mediocre. What helps?
Options
- Retrieve a wider set (for example 30 to 50) and rerank it with a stronger model, then keep the best few (Correct option)
- Pass all 30 chunks to the model
- Lower the temperature to 0
- Make the chunks one token long
Answer
Retrieve a wider set (for example 30 to 50) and rerank it with a stronger model, then keep the best few
Why
Vector search is fast but approximate: it compares two separately computed vectors. A reranker (often a cross-encoder) reads the question and each candidate together and scores them more accurately, but is too slow to run on the whole corpus.
So the pattern is two steps: cheap wide retrieval, then precise reranking of the candidates.
Passing all 30 raises cost and can hide the key passage among noise.
Remember
Retrieve wide, rerank narrow.