language-models

The pretraining objective, at scale

taught

The same next-token-prediction objective this course already introduced, now run on a huge, broad corpus with a huge model -- what changes is scale, not the training signal itself.

Taught in

The pretraining objective, at scale
Wiring Language Models

Requires

next-token-prediction-and-perplexity required
Pretraining at scale is the identical objective this course already defined, applied to far more data and parameters.

Required by

Nothing in the catalog needs this yet.

Local knowledge map

Click to expand

Solid arrows are required prerequisites; dashed are recommended or helpful. Lower rows are more foundational. Click any box to jump to that concept.