You put 40 retrieved chunks into the prompt, and the model often ignores facts that sit in the middle of that text. What is a sensible response?
Options
- Increase the temperature
- Repeat the question five times
- Send fewer, better ranked chunks, and put the most important ones near the start or the end (Correct option)
- Assume the model uses everything in a large window equally well
Answer
Send fewer, better ranked chunks, and put the most important ones near the start or the end
Why
Studies of long-context models ("Lost in the Middle") found that they tend to use information at the beginning and the end of the input better than information in the middle.
A large window means the text fits, not that every part is used equally.
Reranking, trimming and ordering the context usually beat adding more text.
Remember
A long context is not used evenly: send less, better ordered.