nonparametric density estimation

Let $X_1,\dots ,X_n\sim P$. The goal of density estimation is to determine the density of $P$, call it $p$. Here we want to make as few assumptions about $p$ as possible (i.e., we don’t assume that $P$ comes from some parametric family; see parametric versus nonparametric statistics).

An obvious solution is to simply take the empirical distribution, in which case our estimator is

$$\hat{p}(x)=\frac{1}{n}\sum _{i}1(x=X_i).$$

But this solution obviously overfits the given data and has very few nice properties (continuity, smoothness, etc).

In terms of evaluation, typically we’re interested in $L_2$ loss, i.e.,

$$L(p,\hat{p})={\int }_x(\hat{p}(x)-p(x){)}^{2}dx.$$

Here $\hat{p}$ is treated as a fixed function of the training data. The risk is then the expectation of the loss over the training data:

$$R(p,\hat{p})=\mathbb{E}[L(p,\hat{p})].$$

As usual, the risk can be decomposed into a bias term and variance term.

Common methods to nonparametric density estimation include:

• histograms
• kernel density estimation

A solution to nonparametric density estimation also provides a solution to nonparametric regression as follows. Suppose $\hat{p}$ is an estimate of the distribution $(X_1,Y_1),\dots ,(X_n,Y_n)\sim P$. Then, for $m(x)=\mathbb{E}[Y|X=x]$, we can generate an estimate of $m$ with

$$\hat{m}(x)=\int y\hat{p}(y|x)dx=\int y\hat{p}(x,y)/\hat{p}(x)dy.$$

We can estimate both of $\hat{p}(x,y)$ and $\hat{p}(x)$ with nonparametric density estimation. Then we can plug this into the empirical distribution of the $Y_1,\dots ,Y_n$ to estimate the integral.