You send exactly the same prompt twice and get two different answers. What is the most common reason?
Options
- The model learned from the first answer
- The second request went to a different language
- The model picks each token with some randomness (sampling), controlled by settings like temperature (Correct option)
- The prompt was cached wrongly
Answer
The model picks each token with some randomness (sampling), controlled by settings like temperature
Why
At each step the model gives a probability for every possible next token, and the app usually samples from them instead of always taking the top one.
Temperature controls how random that sampling is. A low value makes answers more similar between runs, but even then they are not guaranteed to be identical.
The model does not learn from your conversation.
Remember
Sampling makes answers vary.