Skip to content
Free public beta until December 4, 2026: every course is open, no card needed. After that, the first chapter of each course stays free and full access is a one-time $39 ($19 for up to 100 founding members chosen for helpful feedback). Your progress is kept. How the beta works

tokenization

Splitting raw text into the sequence of tokens a model actually processes.

Tokenization is the process of converting a string of text into a sequence of tokens, each mapped to an integer token ID from the tokenizer's fixed vocabulary, before any model computation happens.

Examples

  • GPT-style tokenizers turn "ChatGPT is great" into a short sequence of token IDs, such as [16949, 318, 1049], the integers the model actually reads.

See also

token byte-pair encoding (BPE) vocabulary
Read the full section →
Graph settings

Calculator