token ID

The integer a tokenizer assigns to each vocabulary entry, and the actual number a model reads in and predicts.

A token ID is the integer index a tokenizer assigns to one entry of its vocabulary. A model never sees text directly -- tokenization converts a string into a sequence of token IDs, and generation is the reverse: the model predicts a probability distribution over every token ID in the vocabulary, one integer chosen at a time.

Examples

A tokenizer might map the token "the" to ID 262 and "cat" to ID 3797 -- the model's output layer produces one probability per such integer.