This course keeps coming back to two related ideas, again and again. The first: a single best-guess number, versus a range plus how sure you are. The second: a method that's wrong the same way every time, versus one that's shaky from sample to sample. Both ideas show up here first. Chapter 5 onward names them constantly, in ML's own words.
Guessing "the average height is 68 inches" is a point estimate. Saying "the average height is between 67 and 69 inches, and I'm 95% sure of that" is an interval estimate. The second one is more honest about how sure you actually are. Machine learning asks the exact same question about a model's guess. A model that just spits out "350,000" for a house's price is giving a point estimate. A model that says "somewhere between 320,000 and $380,000, most likely" is giving something closer to a range. Bias and variance work the same way. A biased model is like a scale that's always five pounds light, no matter what you put on it. It's wrong the same way, every time. A high-variance model is like a scale that gives a wildly different reading every time you step on and off it. It's not wrong the same way twice. It's just shaky. A good model tries to be neither. It's not off the same way every time, and it's not wildly shaky from one training run to the next.
A point estimate is a single best-guess number. An interval estimate states a range together with a confidence level. "Somewhere between 320{,}000 and 380{,}000, 95% confident" is honest about uncertainty in a way a bare number isn't. A model's parameters \theta are what a training procedure solves for. They are a linear regression's coefficients, or a decision tree's split points. A hyperparameter (\lambda, a regularization strength; k, a number of neighbors) is set before training. It is never adjusted by the fitting procedure itself. It's chosen separately, usually by trying several values and checking which generalizes best. The bias-variance tradeoff restates a two-sided estimation failure, in ML's own terms. A model with high bias systematically misses the true pattern, in the same direction every time. That's underfitting. A model with high variance fits the noise in whatever data it happened to see, as if that noise were signal. Fitting it again on a different random sample from the same population would then produce a wildly different fitted model. That's overfitting. Both bias and variance are properties of how a model is estimated, not of any one prediction. A single fitted model's prediction \hat{y} is a point estimate. Some methods — Chapter 6's Bayesian regression, Chapter 14's calibration — also produce an interval, or a full distribution, around \hat{y}. That's the same point-vs-interval idea, applied here to a model's own output instead of a population parameter.
Dragging a model from underfit to overfit
Drag the wiggliness slider from 0 up. Near 0 the curve barely bends (high bias, underfit); past about 4 it starts chasing every wiggle in the data (high variance, overfit) -- watch for the middle ground that tracks the trend without the noise.
Loading interactive…
Worked example
Naming bias vs. variance from a description
-
1
A model trained on many different random samples of the same population always predicts house prices about $20,000 too low, but the amount it's off by barely changes from sample to sample. Is this mostly a bias problem or a variance problem?
-
2
"Consistently off in the same direction, no matter which training sample was used" is the signature of bias, not variance. A high-variance model would instead give very different predictions, depending on which sample happened to be drawn.
-
3\boxed{\text{Bias}}
Chapters 6 and 11 return to this exact diagnosis with real tools. A learning curve and a regularization sweep both make bias and variance visible, not just describable in words.
Loading interactive…
Worked example
Point estimate or interval estimate?
-
1
A model outputs "350,000." A second model outputs "320,000 to $380,000, most likely." Which one gives a point estimate? Which gives an interval estimate?
-
2
A point estimate is one bare number. An interval estimate states a range, plus a confidence level. That's the same distinction, applied here to a model's own prediction instead of a population parameter.
-
3\boxed{\text{First: point estimate. Second: interval estimate.}}
Loading interactive…
Where this goes next
Chapter 5's cross-validation is a real-world tool. It checks how well a
model will generalize. The idea comes from statistics's
sampling-distribution reasoning. Here, it works a bit differently. It
re-splits one dataset, again and again, instead of drawing new samples
from a population. Chapter 11 tunes \lambda, k, and every other
hyperparameter this same way. That search differs from fitting a
trained parameter, which this section just covered.
Check your understanding (counts toward your progress)
Which of these is an interval estimate, in the sense this course uses the term?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
A single number with no stated range is a point estimate, not an interval estimate.
A feature count is a fixed fact about the dataset, not an estimate of any kind.
A hyperparameter is a training setting, not an estimate the model produces about an outcome.
Why
An interval estimate gives a range around a prediction rather than one bare number, mirroring statistics's confidence-interval idea applied to a model's output.
A hyperparameter, unlike a model's parameters, is
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That describes a parameter, not a hyperparameter — a hyperparameter is fixed before training starts.
Hyperparameters have no required default value of zero; each one has its own typical range.
Hyperparameters appear in supervised methods too — regularization strength and k in KNN are both supervised examples.
Why
A hyperparameter (like \lambda or k) is chosen before training and held fixed during it, typically selected via cross-validation rather than solved for.
A model has low bias but high variance. Describe, in your own words, what would happen if you trained it on several different random samples from the same population.
AI grading isn't configured on this deployment.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
First mistake
Correct answer
Also accepted:
Why
Why
Low bias means the model isn't systematically off in one direction; high variance means it's unstable, producing meaningfully different fits depending on which training sample it happened to see.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Statistics
A biased estimator's sampling distribution consistently misses the true parameter. Does collecting a much larger sample using the same biased method fix this?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
A larger sample shrinks the standard error, tightening the spread around whatever center the method already has -- it does nothing to move a biased center.
From Statistics
An estimator is unbiased if:
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Bias is about where the sampling distribution is centered, not about how spread out it is.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Statistics
A sample mean of 41.2 reported alone, with no range, is:
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
A single number offered as a best guess at a parameter is a point estimate.
From Statistics
Which of these is a point estimate rather than an interval estimate?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
25.4 alone, with no range attached, is a single number -- a point estimate.