Linear Regression

If you ever had to fit a line to some data-points you quite possibly have come across linear regression and least squares. Most of the time (linear) regression is introduced as follows:

Assume we have some target data $y \in \mathbb{R}^N$ and some observations $X \in \mathbb{R}^{N \times p}$ and our task is to fit a line $f(X_i) = w^T X_i + w_0$, which minimizes the error.

Now at this point the mean squared error is often introduced:

The question one should always ask: Why exactly do we use this? If we ask ourselves which conditions the error function should satisfy, we will see that the (mean) squared error arises quite naturally.

• The error has to positive, i.e. $y_i - f(X_i) \ge 0$
• Small errors should have less influence than large errors
• Should be easy to optimize

All of these criterions are met for the squared error $(y_i - f(X_i))^2$. Squaring the error results in positive values and values between 0 and 1 have less weight, while values greater than 1 get further amplified. At the same time the least squares error has the solution $\hat{w} = (X^T X)^{-1} X^T y$.

Now even tough all of this sounds reasonable in my opinion there is a better way to introduce regression and least squares.

Probabilistic introduction to Regression¶

As before we want to fit a line $f(X_i) = w^T X_i + w_0$ as good as possible to our target data $y$. What this means, is we assume that the target values $y_i$ and the data have the following relationship:

Here $\epsilon_i$ is a random error, that will always be present in real observations, which we assume to be drawn from a normal distribution $\mathcal{N}(0, \sigma^2)$.

Knowing this we can also express the conditional probability of $y$ in terms of a normal distribution:

Now we still want to find the best possible $w$ and $w_0$ for our data. A simple way to fit a statistical model is to use maximum likelihood estimation which involves maximizing the following likelihood function:

Taking the logarithm of the above expression, multiplying with -1 and plugging in the definition of the normal distribution, we can equally minimize:

If you look closely the first term includes the squared error we introduced earlier. At the same time the second term is a constant that can be neglected, when minimizing.

The minimization problem at hand, $\mathop{\rm arg\,min}\limits_{w, w_0} \frac{1}{2\sigma^2}\sum_{i = 1}^N (y_n - w^T X_i - w_0)^2$ is actually proportional to the residual sum of squares and the mean squared error introduced above. Hence the solution to the maximum likelihood estimation is also:

To me this is quite a remarkable explanation of why the squared error is used.

One step further¶

We can take the above one step further, if instead of a maximum likelihood estimation we use maximum a posteriori estimation.

Here $P(\mathbf{w})$ is the probability density of the prior we choose for the weights $w, w_0$ of our model.

Using this framework we can easily derive many regression models such as lasso and ridge regression as explained by the following table:

A in depth explanation of this topic can also be found in Chapter 11 of Murphy (2022).