Bayes' theorem plays one role in this course, again and again. It turns a starting guess about a data point's class into an updated guess. That happens once the point's features are seen. This new guess is called the posterior belief. Chapter 7 puts three named classifiers in this family: LDA, QDA, and Naive Bayes. Each one starts from this exact math, then adds one extra shortcut about the features, to make it fast and simple to compute.
Theorem
Bayes' Theorem
Recalled from College Algebra · Odds, Bayes' theorem, and binomial probability — first taught there.
For events A and B with P(B) \neq 0, P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}.
The Conditional Probability formula, applied to P(A\mid B) and to P(B\mid A) separately, gives two ways to write the same joint probability: P(A\text{ and }B) = P(A\mid B)\,P(B), \qquad P(A\text{ and }B) = P(B\mid A)\,P(A). Both right-hand sides equal the same quantity, so they equal each other: P(A\mid B)\,P(B) = P(B\mid A)\,P(A). Dividing both sides by P(B) (nonzero by hypothesis) isolates P(A\mid B): P(A\mid B) = \frac{P(B\mid A)\,P(A)}{P(B)}.
Picture a doctor guessing what's wrong with a patient. Before running any test, the doctor already has a rough sense of how common each condition is. That's the prior. Then a test comes back with a specific result. The doctor asks a question: how likely is this exact result under condition A? How likely is it under condition B, or condition C? That's the likelihood — one number per condition. Bayes' theorem is just the rule for combining those two things correctly. Multiply each condition's prior chance by how well it explains the test result. Whichever condition scores highest is the best guess. That best guess is the posterior. A classifier does exactly this for a data point instead of a patient. How common is each class in general? How well do this point's features fit each class? Multiply those two numbers together, for every class, and pick the winner.
In its classification reading, let A be "this point belongs to class k." Let B be "this point's features look like \mathbf{x}." Bayes' theorem turns one number into another. It starts from P(\text{class } k): how common that class is overall. This is the prior. It produces P(\text{class } k \mid \mathbf{x}): how likely that class is, given what was actually observed. This is the posterior: P(\text{class } k \mid \mathbf{x}) = \frac{P(\mathbf{x} \mid \text{class } k)\, P(\text{class } k)}{P(\mathbf{x})}. P(\mathbf{x} \mid \text{class } k) is the likelihood. This is where a specific distribution enters, most often the Gaussian, the normal distribution. It says how plausible these particular feature values are, assuming the point really is from class k. P(\mathbf{x}) is the denominator. It is the same total-probability sum across every class. That sum is what makes the posteriors across all classes add to 1. In practice, it's rarely computed directly. Comparing the unnormalized numerator P(\mathbf{x} \mid \text{class } k)\, P(\text{class } k) across classes is enough to pick the most likely one.
A tiny two-class Bayes update
Click both classes' feature-match leaves — the sum of their counts is the denominator a posterior calculation always divides by.
Loading interactive…
How the prior alone can dominate a classifier
Drag the class-A prevalence slider down toward 1%, keeping both classes' feature-match rates high — watch the posterior for class A stay small even when its features match well.
Loading interactive…
Worked example
A prior that overrides a strong likelihood
-
1
A dataset is 95% class B and 5% class A. A new point's features are 4 times more likely under class A than under class B (a likelihood ratio of 4). Is the point more likely to actually be class A?
-
2
The posterior odds are the prior odds times the likelihood ratio: \dfrac{P(A)}{P(B)}\times 4 = \dfrac{0.05}{0.95}\times 4 \approx 0.21.
-
3\boxed{\text{Still more likely class B} - \text{posterior odds} \approx 0.21 < 1}
A likelihood ratio of 4 sounds decisive, but a 19-to-1 prior against class A more than cancels it out. In other words, a big prior gap can beat a fairly strong signal from the data. Chapter 7 returns to this exact tension when comparing logistic regression (which doesn't use a prior this explicitly) against Naive Bayes (which does).
Loading interactive…
Worked example
Computing an actual posterior
-
1
A dataset is 40% class A and 60% class B. Given a new point's features \mathbf{x}: P(\mathbf{x} \mid A) = 0.5 and P(\mathbf{x} \mid B) = 0.2. What is P(A \mid \mathbf{x}), the actual posterior probability?
-
2
The unnormalized numerators are P(\mathbf{x}\mid A)P(A) = 0.5 \times 0.4 = 0.20 and P(\mathbf{x}\mid B)P(B) = 0.2 \times 0.6 = 0.12. These two raw numbers sum to P(\mathbf{x}) = 0.32. So P(A \mid \mathbf{x}) = 0.20 / 0.32.
-
3\boxed{P(A \mid \mathbf{x}) = 0.625}
Notice the two raw numbers, 0.20 and 0.12, don't add up to 1 on their own. This last step is simple: divide each by their total, 0.32. That one division is what turns them into a valid split between the two classes.
Loading interactive…
Where this goes next
Chapter 7's Naive Bayes model uses this section's rule: likelihood times prior. It adds one shortcut. Each feature is treated as if it acts on its own, separately from every other feature. Their separate effects just get multiplied together. Chapter 14's calibration content comes back to a related question. It asks whether a posterior number is any good in practice, not just correct in theory.
Check your understanding (counts toward your progress)
Which distribution does a generative classifier most commonly assume for a feature's likelihood?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
A uniform assumption would say every feature value is equally likely under every class — rarely a useful assumption for real features.
Poisson is used for count data specifically, not as the general-purpose default this section describes.
The Gaussian is the standard default likelihood assumption for continuous features in generative classifiers.
Why
LDA, QDA, and Gaussian Naive Bayes all assume a Gaussian likelihood for continuous features — the most common default in this course's classifiers.
A dataset is 99% class B and 1% class A. Why can a strong likelihood favoring class A still fail to make class A the more probable posterior?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
This is backwards — the worked example in Section 2 shows a 4x likelihood ratio still losing to a 19-to-1 prior.
The prior matters at every likelihood ratio, not just when the likelihood is uninformative.
Nothing forces a posterior toward 50/50; it can land anywhere depending on both the prior and the likelihood.
Why
Posterior odds equal prior odds times the likelihood ratio, so a sufficiently lopsided prior can dominate even a fairly strong likelihood.
A class's prior odds are 1-to-9 (against). The likelihood ratio in its favor is 6. What are the posterior odds, expressed as a single decimal (favor divided by against)?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Posterior odds = prior odds times likelihood ratio = (1/9)\times 6 = 6/9 \approx 0.667 — still under 1, so still more likely to be the other class.
Correct answer
Also accepted:
Why
Quick review
Practice from earlier coursework, pulled in for this topic.
From Probability
Using the widget, set prevalence to 0.05, sensitivity to 0.95, and specificity to 0.9, then read off the posterior P(disease | positive).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
What this task checks
Structure checked automatically. Whether your ideas are convincing isn't graded, so reread with the checklist.
The compiler says
A model answer
Why
P(+) = (0.95)(0.05)+(0.1)(0.95) = 0.0475+0.095 = 0.1425. P(D|+) = 0.0475/0.1425 ≈ 0.333.
Correct answer
Also accepted:
Why
Drag each slider to its target value, then read the live posterior readout.
From Probability
Bayes' theorem is derived from...
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The law of large numbers is about long-run frequencies, not about reversing a conditional probability.
The complement rule concerns 1 minus a probability, unrelated to Bayes' theorem's derivation.
Summing over a partition is how the DENOMINATOR is often computed, but the theorem itself comes from equating two expressions for the same joint probability.
Why
The conditional probability formula applied to P(A|B) and to P(B|A) both describe P(A∩B); setting them equal and solving for P(A|B) gives Bayes' theorem.
Think about the conditional probability formula written two different ways for the same joint probability.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Probability
X ~ Normal(100, 15). Find Var(X).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Var(X) = sigma^2 = 15^2 = 225.
Correct answer
Also accepted:
Why
Square the standard deviation to get the variance.
From Probability
At which x is a normal pdf f(x) maximized?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
sigma controls spread, not the peak's LOCATION.
mu+sigma is one standard deviation away from the peak, not the peak itself.
x=0 is only the peak if mu happens to be 0 -- not true in general.
Why
Every normal pdf peaks exactly at its own mean, by symmetry.