Skip to content
Free public beta until December 4, 2026: every course is open, no card needed. After that, the first chapter of each course stays free and full access is a one-time $39 ($19 for up to 100 founding members chosen for helpful feedback). Your progress is kept. How the beta works

language-models

Characters, words, and subwords as tokenization choices

taught

Naming the trade-off any tokenizer makes: character-level tokens never go out-of-vocabulary but make sequences long; word-level tokens are short but can't handle new words; subword tokenization is the practical middle ground every modern LLM actually uses.

Taught in

Requires

  • nouns-pos helpful

    A tokenizer's "word-level" option is exactly the human notion of a word this skill already names -- knowing what a word IS (and that punctuation and word boundaries aren't the same thing) makes the character/word/subword trade-off concrete rather than abstract.

  • verbs-pos helpful

    Subword tokenization frequently splits a verb into a stem plus an inflectional ending (e.g. "walk" + "##ing") -- recognizing that split as a genuine word-formation pattern, not an arbitrary cut, is easier having already named verbs and their forms.

  • sentence-boundaries helpful

    A tokenizer's sequence-length trade-off is stated per sentence or document; knowing where a human reader would say a sentence starts and ends gives the character/word/subword comparison a real unit to measure against.

Required by

Nothing in the catalog needs this yet.

Local knowledge map

Local knowledge map

Click to expand

Solid arrows are required prerequisites; dashed are recommended or helpful. Lower rows are more foundational. Click any box to jump to that concept.