pretraining
Training a language model on next-token prediction over a huge, broad corpus, before any task-specific or instruction-following training happens.
Pretraining is the stage of training a language model on the next-token-prediction objective over a large, broad corpus of text, self-supervised (the training signal comes from the text itself, not human-added labels). It is the stage that produces a base model, and every later stage this course covers -- fine-tuning, post-training, RAG, tool use -- starts from a pretrained model rather than training one from a blank slate.
Examples
Pretraining GPT-3 meant running next-token prediction over several hundred billion tokens of web text, books, and code, all before any instruction-following behavior was taught.