Users complain that your chat feature feels slow because nothing appears for several seconds. What helps most with the feeling of speed?
Options
- Increase the maximum output length
- Send the requests one after another
- Stream the response token by token as it is generated, and shorten the prompt (Correct option)
- Raise the temperature
Answer
Stream the response token by token as it is generated, and shorten the prompt
Why
Total generation time depends on the length of the answer, but users judge speed by the time until the first words appear.
Streaming shows text as soon as it is produced. A shorter prompt, a smaller model, or a cached prefix also reduces the wait before the first token.
A longer maximum output or a higher temperature does nothing for speed.
Remember
Stream the answer; the first token matters most.