language-models

Human evaluation and why generic benchmarks aren't enough

taught

Direct human judgment of output quality as the evaluation method every automatic metric is ultimately trying to approximate, and why a generic published benchmark score often doesn't predict how well a model performs on your own specific task and data.

Taught in

Perplexity, recalled, and human evaluation
Wiring Language Models

Requires

next-token-prediction-and-perplexity recommended
Perplexity (chapter 0) measures how well a model predicts text overall; it doesn't measure whether a specific generated answer is actually correct or useful, which is why task-specific human and automatic evaluation still matters.

Required by

Nothing in the catalog needs this yet.

Local knowledge map

Click to expand

Solid arrows are required prerequisites; dashed are recommended or helpful. Lower rows are more foundational. Click any box to jump to that concept.