Applications to linear models - Orthogonality and Least Squares

Do better in math today
Get Started Now

Applications to linear models

Lessons

Notes:
In this section, we will apply least square problems to economics.

Instead of finding the least squares solution of Ax=bAx=b, we will be finding it for Xβ=yX\beta =y where

XX → design matrix
β\beta → parameter vector
yy → observation vector

Least-Squares Line
Suppose we are given data points, and we want to find a line that best fits the data points. Let the best fit line be the linear equation
y=β0+β1xy=\beta_0 + \beta_1 x

And let the data points be (x1,y1),(x2,y2),,,(xn,yn)(x_1,y_1 ),(x_2,y_2 ),\cdots,,(x_n,y_n ). The graph should look something like this:
best fit liney=beta_0 + beta_1 x

Our goal is to determine the parameters β0\beta_0 and β1\beta_1. Let's say that each data point is on the line. Then
best fit line data points beta_0 + beta_1 x

This is a linear system which we can write this as:
linear system

Then the least squares solution to Xβ=yX\beta=y will be XTXβ=XTyX^T X \beta =X^T y.

General Linear Model
Since the data points are not actually on the line, then there are residual values. Those are also known as errors. So we introduce a vector called the residual vector ϵ\epsilon, where
ϵ=yXβ\epsilon = y - X\beta
y=Xβ+ϵ y = X\beta + \epsilon

Our goal is to minimize the length of ϵ\epsilon (the error), so that XβX\beta is approximately equal to yy. This means we are finding a least-squares solution of y=Xβy=X\beta using XTXβ=XTyX^T X\beta=X^T y.

Least-Squares of Other Curves
Let the data points be (x1,y1),(x2,y2),,,(xn,yn)(x_1,y_1 ),(x_2,y_2 ),\cdots,,(x_n,y_n ) and we want to find the best fit using the function y=β0+β1x+β2x2y=\beta_0+\beta_1 x+\beta_2 x^2, where β0,β1,β2\beta_0,\beta_1,\beta_2 are parameters. Technically we are using a best fit quadratic function instead of a line now.
best fit quadratic function

Again, the data points don't actually lie on the function, so we add residue values ϵ1,ϵ2,,ϵn\epsilon_1,\epsilon_2,\cdots,\epsilon_n where

add residue values 
 to data points

Since we are minimizing the length of ϵ\epsilon, then we can find the least-squares solution β\beta using XTXβ=XTyX^T X\beta=X^T y. This can also be applied to other functions.

Multiple Regression
Let the data points be (u1,v1,y1),(u2,v2,y2),,,(un,vn,yn)(u_1,v_1,y_1 ),(u_2,v_2,y_2 ),\cdots,,(u_n,v_n,y_n ) and we want to use the best fit function y=β0+β1u+β2vy=\beta_0+\beta_1 u+\beta_2 v, where β0,β1,β2\beta_0,\beta_1,\beta_2 are parameters.
best fit function Multiple Regression

Again, the data points don't actually lie on the function, so we add residue values ϵ1,ϵ2,,ϵn\epsilon_1,\epsilon_2,\cdots,\epsilon_n where

add residue values to data points of the function

Since we are minimizing the length of ϵ\epsilon, then we can find the least-squares solution β\beta using XTXβ=XTyX^T X\beta=X^T y. This can also be applied to other multi-variable functions.
  • Intro Lesson
    Applications to Linear Models Overview:
Teacher pug

Applications to linear models

Don't just watch, practice makes perfect.

We have over 70 practice questions in Linear Algebra for you to master.