What is a token in a large language model?
Options
- Exactly one word of the text
- One sentence of the prompt
- A password that gives access to the model
- A piece of text (a word, part of a word or a symbol) that the model reads and writes as one unit (Correct option)
Answer
A piece of text (a word, part of a word or a symbol) that the model reads and writes as one unit
Why
Models do not read letters or words directly. A tokenizer cuts the text into pieces called tokens, and the model works with those.
A token is often a short word or part of a longer one, so the number of tokens is different from the number of words, and it changes with the language.
Limits and costs are usually counted in tokens, so this is the unit to think in.
Remember
Models read and write tokens, not words.