Recipe

Evaluate a classifier

Use when: you have a trained classifier and a labeled test set, and need to report how good it actually is

Goal: a confusion matrix, precision, recall, and F1, and an honest read on whether those numbers can be trusted as reported

Taught in Precision, recall, and F1

1. 1

   Run the classifier on every labeled example. Tally the four counts: TP, FP, FN, TN.

   Example

   This chapter's spam-filter tally on 100 emails: TP=50, FN=5, FP=10, TN=35.

   Why: every metric below is computed from these four counts alone. Get them right first, or nothing after this step can be trusted.

2. 2

   \text{precision}=\frac{TP}{TP+FP}, \text{recall}=\frac{TP}{TP+FN}.

   Example

   \text{precision}=\frac{50}{60}\approx0.833, \text{recall}=\frac{50}{55}\approx0.909.

   Why: precision and recall split apart the two different ways a classifier can be wrong (false alarms vs. misses), which one score collapses into one.

3. 3

   F_1=\frac{2\cdot\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}}.

   Example

   F_1=\frac{2(0.833)(0.909)}{0.833+0.909}\approx0.870.

   • precision and recall are already close together — F1 sits close to both; either one alone is nearly as useful.
   • precision and recall differ sharply — F1 is the more honest single number; reporting only the higher of the two would inflate the result.

   Why: F1's harmonic mean won't look good unless both precision and recall are decent, unlike a plain average.

4. 4

   Compare accuracy against precision, recall, and F1. Do they broadly agree, or does accuracy look much better?

   • classes are roughly balanced — accuracy, precision, recall, and F1 tell a similar story.
   • classes are heavily uneven — accuracy can look great while precision, recall, or F1 turn up real problems. Don't trust accuracy on its own. Section 10 covers this in full.

   Why: the next worked example shows accuracy and F1 picking different winners between two classifiers. Checking this catches that gap before it gets reported as one wrong number.

5. 5

   When comparing several candidate classifiers before any cutoff has been picked, precision, recall, and F1 at one threshold may not be the right comparison yet.

   • one exact cutoff is already set — precision, recall, and F1 at that cutoff are the right numbers to report.
   • comparing how good models are overall, before any cutoff is picked — sweep every threshold instead. That's the subject of Section 9.

   Why: precision, recall, and F1 all depend on one fixed cutoff. Comparing candidates fairly before that cutoff is chosen needs a metric built across every threshold at once.