Your RAG system retrieves chunks that are on the right topic, but the exact answer sits in the paragraph just after the chunk. What do you try first?
Options
- Lower the temperature
- Chunk along the document structure (headings, paragraphs) with some overlap, or add the neighbouring chunks to the context (Correct option)
- Make every chunk a single sentence
- Shorten the user's question
Answer
Chunk along the document structure (headings, paragraphs) with some overlap, or add the neighbouring chunks to the context
Why
Fixed-size cuts can separate a statement from the text that completes it. Splitting at headings or paragraphs keeps ideas together, and an overlap between chunks keeps the boundary text in both.
Another option is to retrieve a chunk and then include the chunks before and after it.
Temperature only changes how the answer is written, not what was retrieved.
Remember
Retrieval problems need chunking and retrieval fixes.