What is the context window of a model?
Options
- The amount of data the model was trained on
- The maximum number of tokens it can take into account in one request (Correct option)
- The time the model remembers you between days
- The number of users it can serve at once
Answer
The maximum number of tokens it can take into account in one request
Why
The context window is the model's working memory for one request: the instructions, the conversation or documents you send and, in most APIs, the answer it writes all have to fit inside it.
What is outside the window does not exist for the model in that request.
A bigger window lets you send more text, but it also costs more and is slower.
Remember
Context window = how many tokens fit in one request.