token

The unit of text a language model actually reads and predicts -- often a word piece, not a whole word or a single character.

A token is the atomic unit of a tokenizer's output vocabulary. Modern subword tokenizers (byte-pair encoding, unigram/SentencePiece) choose tokens shorter than most words and longer than single characters, balancing vocabulary size against sequence length.

Examples

"unbelievable" might tokenize as three tokens, "un", "believ", "able", depending on the tokenizer.