normal equations

$A^{\mathsf T}A\hat{\mathbf x}=A^{\mathsf T}\mathbf b$ — the algebraic form of least squares.

$\hat{\mathbf x}$ is a least-squares solution exactly when the residual $\mathbf r=\mathbf b-A\hat{\mathbf x}$ is orthogonal to every column of $A$ , which reduces to $A^{\mathsf T}\mathbf r=\mathbf 0$ , and substituting gives the normal equations: $A^{\mathsf T}A\hat{\mathbf x}=A^{\mathsf T}\mathbf b$ . When $A$ 's columns are independent, $A^{\mathsf T}A$ is invertible and $\hat{\mathbf x}=(A^{\mathsf T}A)^{-1}A^{\mathsf T}\mathbf b$ .