weighted least squares

OLS for linear regression makes the assumption that the variance for all covariates is the same. If this is not the case we use weighted least squares. In particular, the model is

$$\mathbf{Y}={\mathbf{\beta }}^{\mathbf{t}}\mathbf{X}+\mathbf{ϵ},$$

where $\mathbf{ϵ}={\sigma }^2{\mathbf{W}}^{-1}$ and $\mathbf{W}$ is a diagonal matrix with $w_i$ on the diagonal and $\mathbb{V}(Y|x=x_i)=\mathbb{V}({ϵ}_i)={\sigma }^2/w_i$. We solve for an estimate of $\mathbf{\beta }$ by minimizing the “weighted residual sum of squares”, which as we’d suspect, is

$$(\mathbf{Y}-{\hat{\mathbf{\beta }}}^t\mathbf{X}{)}^t\mathbf{W}(\mathbf{Y}-{\hat{\mathbf{\beta }}}^t\mathbf{X}),$$

which generalizes the usual residual sum of squares, in which $\mathbf{W}=\mathbf{I}$. The solution is $\hat{\mathbf{\beta }}=({\mathbf{X}}^t\mathbf{W}\mathbf{X}{)}^{-1}\mathbf{X}\mathbf{W}\mathbf{Y}$.