Leastsquares problem  Orthogonality and Least Squares
Leastsquares problem
Lessons
Notes:
In linear algebra, we have dealt with questions in which $Ax=b$ does not have a solution. When a solution does not exist, the best thing we can do is to approximate $x$. In this section, we will learn how to find a $x$ such that it makes $Ax$ as close as possible to $b$.
If $A$ is an $m \times n$ matrix and $b$ is a vector in $\Bbb{R}^n$, then a leastsquares solution of $Ax=b$ is a $\hat{x}$ in $\Bbb{R}^n$ where
$\lVert bA \hat{x}\rVert \leq \lVert bAx\rVert$
For all $x$ in $\Bbb{R}^n$.
The smaller the distance, the smaller the error. Thus, the better the approximation. So the smallest distance gives the best approximation for $x$. So we call the best approximation for $x$ to be $\hat{x}$.
The LeastSquares Solution
The set of leastsquare solutions of $Ax=b$ matches with the nonempty set of solutions of the matrix equation $A^T A \hat{x}=A^T b$.
In other words,
$A^T A \hat{x}=A^T b$
→$\hat{x} = (A^TA)^{1}A^Tb$
Where $x$ is the least square solutions of $Ax=b$.
Keep in mind that $x$ is not always a unique solution. However, it is unique if one of the conditions hold:
1. The equation $Ax=b$ has unique leastsquares solution for each b in $\Bbb{R}^m$.
2. The columns of $A$ are linearly independent.
3. The matrix $A^T A$ is invertible.
The LeastSquares Error
To find the leastsquares error of the leastsquares solution of $Ax=b$, we compute
$\lVert b  A \hat{x} \rVert$
Alternative Calculations to LeastSquares Solutions
Let $A$ be a $m \times n$ matrix where $a_1,\cdots,a_n$ are the columns of $A$. If $Col(A)=${$a_1,\cdots,a_n$} form an orthogonal set, then we can find the leastsquares solutions using the equation
$A \hat{x}=\hat{b}$
where $\hat{b}=proj_{Col(A)}b.$
Let $A$ be a $m \times n$ matrix with linearly independent columns, and let $A=QR$ be the $QR$ factorization of $A$. Then for each $b$ in $\Bbb{R}^m$, the equation $Ax=b$ has a unique leastsquares solution where
$\hat{x}=R^{1} Q^T b$
→$R\hat{x}=Q^T b$
→$\hat{x} = (A^TA)^{1}A^Tb$
→$R\hat{x}=Q^T b$

Intro Lesson
Least Squares Problem Overview: