perplexity

A single number summarizing how surprised a language model is by held-out text on average -- lower is better.

Perplexity is the exponentiated average negative log-probability a model assigns to each token of a sequence: $\text{PPL} = \exp\left(-\frac{1}{n}\sum_{i=1}^n \log P(w_i \mid w_{<i}) \right)$ . A perplexity of $k$ means the model was, on average, as uncertain as if it were choosing uniformly among $k$ options at each step.

Examples

A model with perplexity 20 on a test set is, on average, as uncertain as a uniform guess among 20 tokens.