An inner product \langle \mathbf u, \mathbf v\rangle generalizes the dot product into a tool for measuring length and angle in any vector space that has one. Two vectors are orthogonal when their inner product is zero, and the Gram–Schmidt process turns any independent set into an orthogonal (then orthonormal) one spanning the same subspace, one projection at a time. Linear Algebra I built all of this for \mathbb R^n with the ordinary dot product. Chapter 05 keeps every word of this paragraph true while replacing \mathbb R^n with spaces where "vector" means polynomial or function — the inner product is what makes that generalization possible at all.
Gram–Schmidt is straightening out a leaning stack of books, one book at a time. Take the second book. Subtract however much of it was leaning against the first, and what's left stands up straight relative to the first. Move to the third book. Subtract its lean against both now-straightened books, and it stands up straight against both too. Repeat until every book stands independently. Orthogonality is exactly that "no leaning" condition, and once every book satisfies it, the whole set is orthogonal, built without changing what shelf space the whole stack occupies.
Orthogonality is the generalization of "perpendicular" to any inner product space: a whole set is orthogonal when its vectors are pairwise orthogonal to each other, none leaning on any other at all. Gram–Schmidt's core move — subtract off the projection onto everything already orthogonalized — never actually uses the fact that the vectors being processed are lists of real numbers. It only uses that an inner product exists to define "projection" and "orthogonal" in the first place. That's precisely why the same three lines of algebra that orthogonalize \{(1,1,0), (1,0,1)\} in \mathbb R^3 also orthogonalize \{1, x, x^2\} under \langle f,g\rangle = \int_{-1}^1 f(x)g(x)\,dx, producing the Legendre polynomials — a fact with no analogue at all until an inner product is defined abstractly.
Theorem
The Gram–Schmidt Process
Recalled from Linear Algebra — Vectors, Matrices, and the Geometry of Data · The Gram-Schmidt process — first taught there.
Given a basis \{\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_n\} for a subspace, an orthogonal basis \{\mathbf{u}_1, \mathbf{u}_2, \ldots, \mathbf{u}_n\} for the same subspace is built one vector at a time by
\mathbf{u}_1 = \mathbf{v}_1, \qquad \mathbf{u}_k = \mathbf{v}_k - \sum_{i=1}^{k-1} \operatorname{proj}_{\mathbf{u}_i}(\mathbf{v}_k) \quad \text{for } k = 2, \ldots, n,where \operatorname{proj}_{\mathbf{u}_i}(\mathbf{v}_k) = \dfrac{\mathbf{v}_k \cdot \mathbf{u}_i}{\mathbf{u}_i \cdot \mathbf{u}_i}\,\mathbf{u}_i. Dividing each \mathbf{u}_k by \lVert\mathbf{u}_k\rVert turns the result into an orthonormal basis.
The claim has two parts: each \mathbf{u}_k built this way is orthogonal to every \mathbf{u}_i that came before it, and the running set \{\mathbf{u}_1, \ldots, \mathbf{u}_k\} spans the same subspace as \{\mathbf{v}_1, \ldots, \mathbf{v}_k\} at every stage. Both are proved together by induction on k.
Base case. \mathbf{u}_1 = \mathbf{v}_1, so trivially \operatorname{span}\{\mathbf{u}_1\} = \operatorname{span}\{\mathbf{v}_1\}, and there is no earlier vector to be orthogonal to.
Inductive step. Suppose \{\mathbf{u}_1, \ldots, \mathbf{u}_{k-1}\} is already orthogonal and spans the same set as \{\mathbf{v}_1, \ldots, \mathbf{v}_{k-1}\}. Form \mathbf{u}_k = \mathbf{v}_k - \sum_{i=1}^{k-1} \operatorname{proj}_{\mathbf{u}_i}(\mathbf{v}_k) = \mathbf{v}_k - \sum_{i=1}^{k-1} \frac{\mathbf{v}_k \cdot \mathbf{u}_i}{\mathbf{u}_i \cdot \mathbf{u}_i}\,\mathbf{u}_i.
Orthogonality. Fix any j with 1 \le j \le k-1 and dot both sides with \mathbf{u}_j: \mathbf{u}_k \cdot \mathbf{u}_j = \mathbf{v}_k \cdot \mathbf{u}_j - \sum_{i=1}^{k-1} \frac{\mathbf{v}_k \cdot \mathbf{u}_i}{\mathbf{u}_i \cdot \mathbf{u}_i}\,(\mathbf{u}_i \cdot \mathbf{u}_j). By the inductive hypothesis \{\mathbf{u}_1, \ldots, \mathbf{u}_{k-1}\} is orthogonal, so \mathbf{u}_i \cdot \mathbf{u}_j = 0 for every i \ne j in that sum, and every term except i = j drops out. What's left is \mathbf{u}_k \cdot \mathbf{u}_j = \mathbf{v}_k \cdot \mathbf{u}_j - \frac{\mathbf{v}_k \cdot \mathbf{u}_j}{\mathbf{u}_j \cdot \mathbf{u}_j}\,(\mathbf{u}_j \cdot \mathbf{u}_j) = \mathbf{v}_k \cdot \mathbf{u}_j - \mathbf{v}_k \cdot \mathbf{u}_j = 0. Since j was arbitrary, \mathbf{u}_k is orthogonal to every earlier \mathbf{u}_i, and since \{\mathbf{u}_1, \ldots, \mathbf{u}_{k-1}\} is already pairwise orthogonal, \{\mathbf{u}_1, \ldots, \mathbf{u}_k\} is too. (This step needs \mathbf{u}_j \ne \mathbf{0} for each earlier j, which the spanning argument below guarantees: a zero \mathbf{u}_j would mean \mathbf{v}_j already lay in the span of the vectors before it, contradicting linear independence of the original basis.)
Same span. The formula for \mathbf{u}_k writes \mathbf{v}_k as \mathbf{u}_k plus a linear combination of \mathbf{u}_1, \ldots, \mathbf{u}_{k-1}, i.e. \mathbf{v}_k = \mathbf{u}_k + \sum_{i=1}^{k-1} \frac{\mathbf{v}_k \cdot \mathbf{u}_i}{\mathbf{u}_i \cdot \mathbf{u}_i}\,\mathbf{u}_i. By the inductive hypothesis each \mathbf{u}_i on the right is itself a linear combination of \mathbf{v}_1, \ldots, \mathbf{v}_{k-1}, so \mathbf{v}_k \in \operatorname{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_{k-1}, \mathbf{u}_k\}, and conversely \mathbf{u}_k \in \operatorname{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_k\} directly from its defining formula. Combined with the inductive hypothesis that \operatorname{span}\{\mathbf{u}_1, \ldots, \mathbf{u}_{k-1}\} = \operatorname{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_{k-1}\}, this gives \operatorname{span}\{\mathbf{u}_1, \ldots, \mathbf{u}_k\} = \operatorname{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}.
By induction, both properties hold for every k up to n, so \{\mathbf{u}_1, \ldots, \mathbf{u}_n\} is an orthogonal basis for the same subspace spanned by \{\mathbf{v}_1, \ldots, \mathbf{v}_n\}. None of the \mathbf{u}_k can be \mathbf{0}: that would force \mathbf{v}_k into the span of \mathbf{v}_1, \ldots, \mathbf{v}_{k-1}, contradicting linear independence, so every division by \mathbf{u}_i \cdot \mathbf{u}_i above is division by a positive number.
Normalizing. Once \{\mathbf{u}_1, \ldots, \mathbf{u}_n\} is orthogonal with no zero vector among them, setting \mathbf{q}_k = \mathbf{u}_k / \lVert\mathbf{u}_k\rVert scales each vector to unit length without disturbing the directions, so \{\mathbf{q}_1, \ldots, \mathbf{q}_n\} is orthonormal and spans the same subspace.
Orthogonalize three vectors, one projection at a time
Step forward — watch each vector get its lean against the earlier ones subtracted off, until all three stand mutually perpendicular.
Loading interactive…
A projection is the closest point in the subspace
Drag the point p — its shadow onto the plane is always the CLOSEST point in that plane, never a random point on it.
Loading interactive…
Worked example
Orthogonalizing two vectors by hand
-
1the first vector never needs adjusting — it has nothing to be orthogonal against yet
Orthogonalize \{\mathbf v_1, \mathbf v_2\} = \{(3,1,0), (2,2,0)\} using the standard dot product. \mathbf u_1 = \mathbf v_1 = (3, 1, 0)
the first vector never needs adjusting — it has nothing to be orthogonal against yet
-
2\operatorname{proj}_{\mathbf u_1}(\mathbf v_2) = \frac{\mathbf v_2 \cdot \mathbf u_1}{\mathbf u_1 \cdot \mathbf u_1}\mathbf u_1 = \frac{6+2}{9+1}(3,1,0) = \frac{8}{10}(3,1,0) = (2.4,\ 0.8,\ 0)
-
3\mathbf u_2 = \mathbf v_2 - \operatorname{proj}_{\mathbf u_1}(\mathbf v_2) = (2,2,0)-(2.4,0.8,0) = (-0.4,\ 1.2,\ 0)
-
4\boxed{\mathbf u_1 = (3,1,0), \quad \mathbf u_2 = (-0.4,\ 1.2,\ 0)}
Check: \mathbf u_1 \cdot \mathbf u_2 = (3)(-0.4)+(1)(1.2) = -1.2+1.2 = 0. ✓
Loading interactive…
Where this shows up: an inner product on any space at all
Chapter 05 states the axioms an inner product must satisfy — linearity, conjugate symmetry, and positive-definiteness — with no reference to \mathbb R^n anywhere in the definition. Any function satisfying those three axioms earns everything reviewed here for free: Cauchy–Schwarz, orthogonal projection, best approximation, and Gram–Schmidt all transfer without a single new proof technique.
Where this goes next
Signal processing and the Fourier basis
The Fourier series expresses a periodic signal as a combination of sines and cosines precisely because those functions form an orthogonal set under the inner product \langle f,g\rangle = \int f(x)g(x)\,dx — finding a Fourier coefficient is exactly the same projection formula this section uses to peel a vector's component off in a chosen direction, just run on functions instead of arrows. Chapter 05 makes this precise instead of asserting it by analogy.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
Which choice of weights makes \mathbf u\cdot\mathbf v = w_1u_1v_1 + w_2u_2v_2 fail to be an inner product on \mathbb R^2?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
This is the ordinary dot product (all weights equal 1) — it's a valid inner product.
Both weights here are positive, just very different in size — positive definiteness only needs each weight to be strictly greater than 0, not close in magnitude.
Both weights are positive (just less than 1), which is all positive definiteness requires — this is a valid inner product.
Why
With w_2 = -2, take \mathbf u = (0,1): \mathbf u\cdot\mathbf u = 3(0)^2 + (-2)(1)^2 = -2 < 0, violating positive definiteness. A weighted dot product stays a valid inner product only when every weight is strictly positive; a single negative weight breaks it.
Try plugging in a vector with a 0 in the coordinate whose weight stays positive, so only the negative-weight term survives \mathbf u\cdot\mathbf u.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
A weighted dot product on \mathbb R^3 uses weights w_1 = 1, w_2 = 1, w_3 = 1. How does it compare to the ordinary dot product?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Setting every weight to 1 doesn't zero anything out — it reduces w_ku_kv_k to plain u_kv_k in every term.
Positive definiteness only requires each weight to be strictly positive; w_k=1 satisfies that just as well as any other positive weight.
The ordinary dot product isn't separate from the weighted family — it's the specific member of that family where every weight equals 1.
Why
The weighted dot product w_1u_1v_1 + w_2u_2v_2 + w_3u_3v_3 becomes exactly u_1v_1+u_2v_2+u_3v_3, the ordinary dot product, once every weight is set to 1. The ordinary dot product is a special case of the weighted family, not a different operation.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
Find the value of k that makes \mathbf p = (k, 3, 5) orthogonal to \mathbf q = (2, 1, -1).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
\mathbf p \cdot \mathbf q = 2k + 3 - 5 = 2k - 2. Setting 2k - 2 = 0 gives k = 1. Check: 2(1) + 3 - 5 = 0. ✓
Correct answer
Also accepted:
Why
Compute \mathbf p\cdot\mathbf q in terms of k first, then set that expression equal to zero and solve.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
For W = \operatorname{span}\{(5, -2)\} in \mathbb R^2, a vector (x, y) lies in W^\perp exactly when a certain expression in x and y equals zero. Write that expression.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Any equivalent form counts — 2x+3, 3 + 2·x, … Use the keypad below the field for fractions, roots, and other symbols.
Why
(x,y) \in W^\perp means (x,y) \cdot (5,-2) = 0, i.e. 5x - 2y = 0. The expression that must equal zero is 5x - 2y.
Correct answer
Also accepted:
Why
Dot a general vector (x,y) against the spanning vector (5,-2) and set the result to zero.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
Let \mathbf v_1 = (1, 0, 0), \mathbf v_2 = (2, 0, 0). These are NOT independent (they're parallel). What does Gram-Schmidt produce as \mathbf u_2?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That's \mathbf v_2's projection onto \mathbf u_1, not the remainder after subtracting it off.
That's \mathbf v_1 itself — unrelated to this subtraction.
The subtraction \mathbf v_2 - \operatorname{proj}_{\mathbf u_1}(\mathbf v_2) is exactly zero here, not merely small — check the arithmetic again.
Why
\operatorname{proj}_{\mathbf u_1}(\mathbf v_2) = \frac{2}{1}(1,0,0) = (2,0,0) = \mathbf v_2 exactly, since \mathbf v_2 already lies entirely on the line through \mathbf u_1. So \mathbf u_2 = (2,0,0)-(2,0,0) = (0,0,0) — Gram-Schmidt's way of flagging that \mathbf v_1,\mathbf v_2 were never independent to begin with.
Compute \operatorname{proj}_{\mathbf u_1}(\mathbf v_2) and notice how it compares to \mathbf v_2 itself when the two original vectors are parallel.
From Linear Algebra — Vectors, Matrices, and the Geometry of Data
Let \mathbf v_1 = (3, 0) and \mathbf v_2 = (2, -6). Compute \lVert \mathbf u_2 \rVert.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
\operatorname{proj}_{\mathbf u_1}(\mathbf v_2) = \frac{6+0}{9+0}(3,0) = \frac{2}{3}(3,0) = (2,0). So \mathbf u_2 = (2,-6)-(2,0) = (0,-6), and \lVert \mathbf u_2 \rVert = 6.
Correct answer
Also accepted:
Why
Compute \operatorname{proj}_{\mathbf u_1}(\mathbf v_2) first, subtract it from \mathbf v_2 to get \mathbf u_2, then find that vector's length.