Measurements disagree, and a system built from them usually has no solution. Least squares asks for the next best thing: the choice of unknowns that makes the errors as small as possible, measured by the sum of their squares. Geometrically it is a projection, and algebraically one system of equations.

Several witnesses give slightly different times for an event. The fairest single time is the one that disagrees with all of them as little as possible overall, not one that agrees perfectly with any single witness.

No line passes through the three measurements $(-1,0)$, $(0,3)$ and $(1,3)$. The least-squares line $y=2+1.5t$ makes the sum of the squared vertical gaps, dashed, as small as any line can: $0.25+1+0.25=1.5$.

A least-squares solution of $A\mathbf{x}=\mathbf{b}$ is a vector $\hat{\mathbf{x}}$ that makes $\|\mathbf{b}-A\hat{\mathbf{x}}\|$ as small as possible. The vectors fill the column space, so must be the projection of onto , and the least-squares solutions are exactly the solutions of the normal equations below. When the columns of are independent, is invertible and the solution is unique.

• Fitting a line $y=c_0+c_1t$ to points $(t_i,y_i)$: has a column of ones and the column of the , lists the , and .

As $\mathbf{x}$ varies, $A\mathbf{x}$ runs over the column space, and the point of a subspace closest to $\mathbf{b}$ is the projection. So $\hat{\mathbf{x}}$ is best exactly when the error $\mathbf{b}-A\widehat{\mathbf{x}}$ is orthogonal to , that is, when it lies in . That is the equation , rearranged.

The vectors $A\mathbf{x}$ fill $\operatorname{Col}A$, and the one closest to $\mathbf{b}$ is the projection.

The error of a projection is orthogonal to the subspace.

The complement of the column space.

So $A^{T}$ sends the error to $\mathbf{0}$.

Rearrange. Every step reverses, so the solutions of the normal equations are exactly the least-squares solutions.

Multiply $A\mathbf{x}=\mathbf{b}$ by $A^{T}$ on the left. The new system always has a solution, and its solutions make $\|\mathbf{b}-A\mathbf{x}\|$ as small as it can be.

The system $x=b_1,\ldots,x=b_m$ has $$ a column of ones. The normal equation is : the least-squares answer is the average.

To fit $y=c_0+c_1t$, each point gives one equation $c_0+c_1{t}_i={}_{}$. With the matrix is diagonal, and the normal equations separate.

• Least squares makes $\|\mathbf{b}-A\mathbf{x}\|$ as small as possible.
• Its solutions are those of $A^{T}A\hat{\mathbf{x}}=A^{T}\mathbf{b}$; unique when the columns are independent.

Great work! You now know how to:

• set up and solve the normal equations;
• fit a line to data by least squares;
• explain least squares as a projection onto the column space;
• tell when the least-squares solution is unique.