kernel density estimation

A specific approach to nonparametric density estimation, which can be viewed as a kind of “smoothed out” histogram. Let $K$ be a smoothing Kernel, i.e., $\int K(x)dx=1$, $\int xK(x)dx=0$ and $\int x^{2}K(x)dx>0$. That is, $K$ induces a distribution on the space $\mathcal{X}$ which is symmetric about the origin.

Our estimate for $p(x)$ is

$$\hat{p}(x)=\frac{1}{n}\sum _{i=1}^n\frac{1}{h^d}K\left(\frac{x-X_i}{h}\right).$$

Essentially, we’re placing small lumps of mass around the data points $X_1,\dots ,X_n$. The size of the lumps is controlled by the Kernel and $h$, called the bandwidth. As $h$ increases, $\hat{p}$ becomes more uniform.

We can generalize the above to allow for positive definite bandwidth matrices $H$ and write

$$\hat{p}(x)=\sum _{i=1}^n{K}_H(x-X_i),$$

where $K_H(z)=|H{|}^{-1/2}K(H^{-1/2}z)$. Taking $H=h^{2}I$ recovers the previous formula.

For kernels of the form $K(x)=c\cdot k(\Vert x\Vert )$ where $k$ is a one-dimensional kernel, or of the form $K(x)=k(x_1)\cdots k(x_n)$, the bias of KDE is $O(h^4)$ and the variance is $O(1/(n{h}^d))$. If we take $h=O(n^{-\frac{1}{4+d}})$ this gives an MSE of $O(n^{-\frac{4}{4+d}})$, which is better than histograms and is minimax optimal over many classes of densities (see statistical decision theory). Of course, this is worse than parametric rates such as the MLE.