Why does a long chat get slower and more expensive as it goes on?
Options
- The model gets tired and needs more compute
- Each request resends the whole history, so the number of input tokens keeps growing (Correct option)
- The model weights grow with each message
- The provider limits long users on purpose
Answer
Each request resends the whole history, so the number of input tokens keeps growing
Why
Because the model has no memory between calls, the app sends the conversation again every time. A chat of 50 messages carries all 50 in every new request.
Providers charge by tokens and the model needs to read all of them, so cost and latency grow.
Common fixes: summarize old messages, drop what is not needed, or cache the stable beginning of the prompt.
Remember
More history = more input tokens on every call.