squared error

Exactly what it sounds like. Worth a note just because it keeps coming up.

Actually, you know what, the bias-variance decomposition is possibly worth discussing. Given a predictor $\hat{f}:\mathcal{X}\to \mathbb{R}$ and a target $f^{\ast }:\mathcal{X}\to \mathbb{R}$, the squared error can be decomposed as

$$\mathbb{E}[(\hat{f}(X)-f^{\ast }(X){)}^2]={\text{Bias}}^2(\hat{f})+\mathbb{V}(\hat{f})+{\sigma }^2,$$

where $\text{Bias}(\hat{f})=\mathbb{E}[\hat{f}(X)-f^{\ast }(X)]$, $\mathbb{V}(\hat{f})=\mathbb{E}[(\hat{f}(X)-\mathbb{E}[\hat{f}(X)]{)}^2]$ and ${\sigma }^2$ is the irreducible noise in the data (i.e., given $X$, outcomes are drawn as $Y=f^{\ast }(X)+ϵ$ where $ϵ$ has variance ${\sigma }^2$.

As you increase model complexity, bias typically decreases and variance increases. This is known as the bias-variance tradeoff. So to minimize squared-error, you want to find a appropriate compromise between the two.