A norm turns a vector into a single non-negative number. It measures the vector's "size." This course leans on norms constantly. Norms measure how big an error is. The same idea, applied to matrices, measures how much a matrix can stretch that error.
Different norms are different honest ways to ask "how big is this list of numbers?" Add up the sizes. Or measure straight-line distance. Or just take the single largest entry. None is more "correct" than another. Each is more convenient in different situations. A matrix's own norm asks a related question, in a different shape: "what's the most this matrix can stretch any direction I feed it?" Stretch twice, by applying two matrices back to back. That can never stretch more than the product of what each one does alone. That is all "submultiplicativity" means. It matters for a real reason. Multiply a matrix's own stretch factor by its inverse's stretch factor. That gives an honest number for how much a tiny mistake already in the input can balloon into a much larger mistake in the answer, once a real computation has had its turn to make things worse.
The three standard vector norms on \mathbb{R}^n are \lVert v\rVert_1=\sum_i|v_i|, \lVert v\rVert_2=\sqrt{\sum_i v_i^2} (ordinary length), and \lVert v\rVert_\infty=\max_i|v_i|. All three satisfy the same three defining properties: non-negativity, scaling, and the triangle inequality. They get used interchangeably, whichever is most convenient for a given proof or computation.
A matrix earns its own norm from any of these. It's built as the induced norm \lVert A\rVert=\max_{\mathbf{x}\neq\mathbf{0}} \lVert A\mathbf{x}\rVert/\lVert\mathbf{x}\rVert — the most any unit vector can be stretched by A. The Frobenius norm \lVert A\rVert_F=\sqrt{\sum_{i,j}a_{ij}^2} takes a different route. It treats A's entries as one long vector, instead of asking about stretching at all. Every induced norm satisfies submultiplicativity: \lVert A\mathbf{x}\rVert\le\lVert A\rVert\,\lVert\mathbf{x}\rVert and \lVert AB\rVert\le\lVert A\rVert\,\lVert B\rVert. The first inequality is exactly what "\lVert A\rVert is the maximum of \lVert A\mathbf{x}\rVert/\lVert\mathbf{x}\rVert" already says. Apply that bound twice — first to B\mathbf{x}, then to A applied to that result — and the second inequality follows. This one property is what makes the condition number of A well defined: the product \lVert A\rVert\,\lVert A^{-1}\rVert\ge1. It bounds exactly how much a relative error already in \mathbf{b} can be amplified into a relative error in the solution of A\mathbf{x}=\mathbf{b}.
Read three norms off one vector
Drag the vector's tip and compare how the \ell_1, \ell_2, and \ell_\infty norms respond.
Loading interactive…
Worked example
Computing all three norms of one vector
-
1
Find \lVert v\rVert_1, \lVert v\rVert_2, and \lVert v\rVert_\infty for v=(3,-4).
-
2
\lVert v\rVert_1 = |3|+|-4| = 7.
-
3
\lVert v\rVert_2 = \sqrt{3^2+(-4)^2} = \sqrt{25} = 5.
-
4\boxed{\lVert v\rVert_1=7,\ \lVert v\rVert_2=5,\ \lVert v\rVert_\infty=4}
Every norm of the same vector is a different honest number — none is "the" size of v.
Loading interactive…
Worked example
Two vectors that swap places depending on the norm
-
1
Is it possible for \lVert u\rVert_1 > \lVert v\rVert_1 but \lVert u\rVert_2 < \lVert v\rVert_2?
-
2
Yes. Take u=(1,1,1,1) and v=(3.5,0,0,0): \lVert u\rVert_1=4 > 3.5=\lVert v\rVert_1, but \lVert u\rVert_2=2 < 3.5=\lVert v\rVert_2.
-
3\boxed{\text{"bigger" depends on which norm is asking}}
Concentrating a vector's total size into one coordinate (like v) inflates \lVert\cdot\rVert_2 relative to \lVert\cdot\rVert_1, compared to spreading the same total out evenly (like u). All norms on \mathbb{R}^n are still equivalent — each bounds the others by a constant depending only on n — but that guarantees comparable scale, never a fixed ranking between two specific vectors.
Loading interactive…
Worked example
A quick condition-number estimate
-
1
Estimate the condition number of the diagonal matrix A=\begin{bmatrix}10&0\\0&0.01\end{bmatrix} using the induced \ell_2 norm.
-
2
For a diagonal matrix, the induced \ell_2 norm is just the largest diagonal magnitude: \lVert A\rVert_2=10.
-
3
A^{-1}=\begin{bmatrix}0.1&0\\0&100\end{bmatrix}, so \lVert A^{-1}\rVert_2=100.
-
4\boxed{\lVert A\rVert_2\,\lVert A^{-1}\rVert_2 = 10\times100 = 1000}
A condition number this large is a warning sign, not a computational error — it says any tiny error already in the input data can come back roughly 1000\times larger in the solution.
Loading interactive…
Where this goes next
Measuring error and matrix conditioning
This course's own error-analysis vocabulary asks one question: "how big is the error between this approximation and the true answer?" That question is a norm, applied to a vector of errors. Now extend the same idea to matrices. The largest amount a matrix can stretch any unit vector becomes the condition number. That is the single most important quantity in this course's treatment of linear systems.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
Find \lVert \mathbf v \rVert_2 for \mathbf v = (5, 12).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
\lVert \mathbf v \rVert_2 = \sqrt{5^2 + 12^2} = \sqrt{25+144} = \sqrt{169} = 13, a Pythagorean-triple pair chosen so the square root lands exactly.
Correct answer
Also accepted:
Why
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
Ridge regression penalizes coefficients with the squared L2 norm; LASSO penalizes them with the L1 norm. Why does LASSO tend to zero out coefficients entirely while ridge only shrinks them?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Both norms measure a real relationship in size (L_\infty \le L_2 \le L_1 for the same vector), but 'smaller penalty value' isn't what drives coefficients to exactly zero -- the SHAPE of the unit ball (corners vs. smooth) is what does that.
Both LASSO and ridge typically pair the same prediction-error term with a different size penalty on the coefficients -- the loss function itself doesn't need to change.
It's a direct, provable geometric consequence of which norm's unit ball has corners on the axes -- not an implementation detail.
Why
The L1 norm's unit ball is a diamond with corners sitting exactly on the coordinate axes (where one coordinate is zero); an optimum constrained to touch that ball lands on a corner far more often than chance, zeroing out a coefficient. The L2 norm's unit ball is a smooth circle/sphere with no corners to land on, so ridge shrinks coefficients smoothly instead.
Picture each norm's 'unit ball' (the set of vectors at distance 1 from the origin) and ask which shape has sharp corners sitting on the axes.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
Which axiom does homogeneity, \lVert c\mathbf u \rVert = |c|\,\lVert \mathbf u \rVert, rule out?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That's nonnegativity (axiom 1), not homogeneity.
Homogeneity says nothing prevents two different vectors from sharing a norm value (e.g. (1,0) and (0,1) both have L2 norm 1) -- that's not what it rules out.
That's definiteness (axiom 2), the other half of the definition -- homogeneity is about scaling, not about which vector maps to zero.
Why
Homogeneity forces \lVert c\mathbf u \rVert = |c|\,\lVert \mathbf u \rVert, so scaling a vector by c=2 must double its norm (since |2|=2), not leave it unchanged -- a norm that let scaling by a nonzero constant leave the size reading identical would violate this axiom.