Linear Regression

todo We assume the model $y={\beta }^{t}X+ϵ$, where we make various different assumptions on $ϵ$.

Given $X_1,\dots ,X_n$, introduce the design matrix $\mathbb{X}$ which rows correspond to $X_1,\dots ,X_n$. Common practice is to add a 1 to each vector $X_i$ so that the first column of $\mathbb{X}$ is all 1s, which enables us to model non-zero offsets (i.e., $y={\beta }_0+{\beta }^{t}X+ϵ$).

A common loss is to minimize the “residual sum-of-squares (RSS)”, which is the ${\ell }_2$ distance between our predictions ($\hat{y}={\hat{\beta }}^{t}X$ ) and the true values $y$, i.e.,

$$\underset{\beta }{\min }={\left|Y-\mathbb{X}\beta \right|}_2^2.$$

The resulting $\hat{\beta }$ is the least squares estimate.

Once you’ve ft a model, you should run linear regression diagnostics.