scaling laws
The empirical pattern that a language model's loss falls smoothly and predictably as data, parameters, and compute all grow together.
Scaling laws are empirical relationships describing how a language model's pretraining loss decreases as a smooth, roughly power-law function of model size, dataset size, and training compute, when the three are scaled together in the right proportion. They explain why "bigger model, more data" was a reliable recipe for years of LLM development, and why compute budget, not just model size alone, is the resource being allocated.
Examples
Scaling laws let researchers predict, before running a single expensive training job, roughly how much pretraining loss a 10x increase in compute (spent optimally on model size and data together) will buy.
See also