Every method in this course does arithmetic on a table of numbers. That's true no matter how the method is described. This section covers what that table is, and how a vector and a matrix multiply. It also fixes the exact way this course uses both words from now on.
Think of a spreadsheet. Every row is one thing you're studying — a house, a customer, a patient. Every column is one fact you measured about it — square footage, age, blood pressure. That whole spreadsheet is what this course calls a design matrix. Pull out a single row on its own. That's a feature vector: one house's numbers, in order. A separate column off to the side holds the price each house actually sold for. That column is the target vector, the answer key. Multiplying that grid by a list of coefficients works one row at a time. For each row, multiply each number by its matching coefficient. Then add the results together. That sum is the row's prediction. Doing this for every row at once is the real meaning behind "the model predicts a price for every house." That phrase just hides the arithmetic underneath. What's new here isn't the arithmetic. It's the habit of reading "matrix" as "spreadsheet of examples," and "vector" as "one row of it." That's the only way this course uses either word from here on.
A vector is an ordered list of numbers. A matrix is a rectangular array of numbers with rows and columns. A single data sample — one house, one email, one patient — is recorded as a feature vector \mathbf{x}_i \in \mathbb{R}^p: a list of p numbers, one per measured quantity ("feature"). Stacking n samples as rows produces the design matrix \mathbf{X}, an n \times p matrix. Row i is sample i's feature vector. Column j is every sample's value for feature j. A separate target vector \mathbf{y} \in \mathbb{R}^n holds the known correct output for each sample — the price a house sold for, whether an email was spam. Multiplying a matrix by a vector, A\mathbf{x}, works one row at a time. The i-th entry of the result is row i of A dotted with \mathbf{x}: (A\mathbf{x})_i = \sum_{j=1}^p A_{ij}x_j. In words, multiply each entry of the row by the matching entry of \mathbf{x}, and add. Apply this to a design matrix and a coefficient vector \boldsymbol\theta. Then \mathbf{X}\boldsymbol\theta computes this same row-times-\boldsymbol\theta dot product for every sample at once, producing one predicted value \hat{y}_i per row. Every method in this course is doing the same underlying thing to the same underlying object, whatever it's called. It takes \mathbf{X} and \mathbf{y}, and produces a function. That function predicts a \hat{y} for a new \mathbf{x} it hasn't seen.
One dataset, as a matrix-vector product
Step through A\mathbf{x} entry by entry. Read A as a tiny 3-house design matrix (columns: square footage in hundreds, bedrooms, age in years) and \mathbf{x} as a candidate set of coefficients — the running answer in each row is that house's predicted price, in the same units.
Loading interactive…
Worked example
Reading a design matrix
-
1
A dataset has 4 houses, each with (square footage, bedrooms, age) recorded. Its design matrix is \mathbf{X} = \begin{bmatrix} 1400 & 3 & 10 \\ 2100 & 4 & 2 \\ 950 & 2 & 40 \\ 1800 & 3 & 15 \end{bmatrix}. What are n and p for this dataset, and what does the entry in row 3, column 1 mean?
-
2
\mathbf{X} has 4 rows and 3 columns, so n = 4 samples and p = 3 features. Row 3 is the third house's feature vector; column 1 is square footage across every house.
-
3\boxed{n = 4,\ p = 3,\ X_{3,1} = 950 \text{ sq ft (house 3's square footage)}}
Loading interactive…
Worked example
A prediction as a matrix-vector product
-
1
Using the same three features (square footage, bedrooms, age), a fitted model has coefficients \boldsymbol\theta = (120, 15000, -2000). What price does it predict for a 1400 sq ft, 3-bedroom, 10-year-old house?
-
2
The prediction is the dot product of the feature vector with \theta: 1400(120) + 3(15000) - 2000(10).
-
3\boxed{\hat{y} = 168000 + 45000 - 20000 = \$193{,}000}
Every regression method in Chapter 6 computes exactly this dot product — the only thing that changes chapter to chapter is how \theta gets fit.
Loading interactive…
Where this goes next
Chapter 6's linear regression fits a coefficient vector \boldsymbol\theta
by solving a matrix equation built directly from \mathbf{X} and
\mathbf{y}. This is the normal-equations machinery for A\mathbf{x} =
\mathbf{b}, taught in full in linear-algebra. Here it's applied to a
design matrix instead of an abstract A. Chapter 12's PCA reduces
\mathbf{X}'s column count a different way: it finds the directions of
greatest variance. That's an eigenvalue problem on \mathbf{X}'s own
covariance matrix.
Check your understanding (counts toward your progress)
A dataset's design matrix \mathbf{X} has shape 500 \times 8. How many samples and how many features does the dataset have?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The row count is the sample count, not the feature count — a design matrix's rows are samples, columns are features.
The two dimensions of a design matrix are not the same quantity repeated; rows count samples, columns count features.
Both numbers were swapped or dropped — reread the shape as (rows = samples, columns = features).
Why
A design matrix is n \times p with n rows (samples) and p columns (features), so a 500\times 8 matrix has 500 samples and 8 features.
In a design matrix \mathbf{X}, what does a single column represent?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That describes a row, not a column — a row is one sample's feature vector.
The target vector \mathbf{y} is stored separately, not as a column of \mathbf{X}.
A column's length equals the sample count, but the column itself is a feature's values, not a count.
Why
Each column of \mathbf{X} holds one feature's value for every sample — column j is feature j across the whole dataset.
Which quantity does every method in this course ultimately try to produce, given a new feature vector \mathbf{x}?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Producing a new design matrix isn't a model's job — a design matrix is the input a model is trained on, not its output.
Generating more training data is a data-collection step, not what a trained model does with a new input.
A p-value belongs to statistics's hypothesis-testing framework, not to what a predictive model outputs.
Why
Every method in this course, whatever it's called, takes a feature vector \mathbf{x} and produces a prediction \hat{y}.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
A dataset records 7 numeric features for every item. Each record corresponds to a vector in which space?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Off by one — the dimension equals the number of features, not one fewer.
That would be doubling the feature count. Each feature gets exactly one entry, so the dimension equals the number of features, not twice it.
49 = 7^2 — dimension doesn't square the feature count, it just counts the features.
Why
A record's vector has one entry per feature, so a record with 7 features lives in \mathbb R^7 — the dimension is just a count of the measurements.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
Record A lists features in the order (price, weight), and record B lists the same two numbers in the order (weight, price). What goes wrong if you compute A \cdot B directly?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The dot product's formula multiplies matching entries — if the entries don't mean the same thing in each vector, the sum it produces isn't a meaningful comparison anymore.
Sign isn't the issue here — the problem is which numbers get multiplied together, not whether the result is positive or negative.
The dimension stays 2 either way; the vectors just disagree about what each of the 2 entries represents.
Why
Dot product, distance, and every formula in this chapter assume entry i means the same feature in every vector being compared. Swap the order in only one record and entry-by-entry operations start pairing price with weight instead of price with price.
A dot product multiplies matching entries -- if B's entries are in a different order than A's, which quantities actually get multiplied together?
Quick review
Practice from earlier coursework, pulled in for this topic.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
For the product A\mathbf x to be defined, what must match?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That's related to the SIZE of the output, not to whether the product is defined at all — check which count of A has to match how many entries \mathbf x has.
A never has to be square for A\mathbf x to make sense — a 2\times 3 matrix times a vector with 3 entries works fine and produces a vector in \mathbb{R}^2.
That would only ever be true for a square matrix, and isn't the actual requirement for A\mathbf x to be defined.
Why
Each entry of A\mathbf x is a dot product of a row of A with \mathbf x, so every row needs exactly as many entries as \mathbf x has — the number of columns of A must equal the number of entries of \mathbf x.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
Let A = \begin{bmatrix} 2 & 5 \\ -3 & 1 \end{bmatrix} and \mathbf x = \begin{bmatrix} 4 \\ -2 \end{bmatrix}. Using the row picture, find (A\mathbf x)_1.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
(A\mathbf x)_1 is row 1 of A dotted with \mathbf x: 2(4) + 5(-2) = 8 - 10 = -2.
Correct answer
Also accepted:
Why