language-models

Next-token prediction and perplexity

taught

A language model as a distribution over the next token given the ones before it; the chain rule factors a sequence's probability into a product of these conditionals, and perplexity turns average per-token surprise into one comparable number.

Taught in

Language modeling as next-token prediction
Wiring Language Models

Requires

n-gram-language-models recommended
deep-learning ch.18 teaches the n-gram language model and perplexity as an RNN-motivating example; this course reuses the same definition and extends it to the LLM setting.

Required by

Human evaluation and why generic benchmarks aren't enough recommended
The pretraining objective, at scale required

Local knowledge map

Click to expand

Solid arrows are required prerequisites; dashed are recommended or helpful. Lower rows are more foundational. Click any box to jump to that concept.