Gaussian process regression

A Bayesian (see Bayesian statistics) approach to nonparametric regression.

We say a stochastic process $\{m(x):x\in \mathcal{X}\}$ is a Gaussian process if, for any finite collection $x_1,\dots ,x_n$,

$$(m(x_1),\dots ,m(x_n))\sim N(\mu ,K),$$

where $K_{i,j}=K(x_i,x_j)$ is a Mercer kernel and $\mu =(\mu (x_1),\dots ,\mu (x_n))$. Typically we suppose that $\mu =0$ since we can always subtract the mean from the data.

Gaussian process regression places a prior $\pi$ on the regression function $m(x)=\mathbb{E}[Y|X=x]$ which assumes that $m$ is a Gaussian process. Thus, assuming $\mu =0$, the prior has density

$$\pi (m)=\frac{|K{|}^{-1/2}}{(2\pi {)}^{n/2}}\exp (-\frac{1}{2}{m}^T{K}^{-1}m),$$

where $m=(m(x_1),\dots ,m(x_n))$.

Predictive Distribution

Suppose we have observed training data $(X_1,Y_1),\dots ,(X_n,Y_n)$ and observe a new point $Y_{n+1}=m(X_{n+1})+{ϵ}_{n+1}$. We have

$$(Y_1,\dots ,Y_n)=(m(x_1),\dots ,m(x_n))+({ϵ}_1,\dots ,{ϵ}_n)\sim N(0,K+{\sigma }^{2}I),$$

if ${ϵ}_i\sim N(0,{\sigma }^2)$. Then $\text{Cov}(Y_i,Y_{n+1})=\text{Cov}(m(x_i),m(x_{n+1}))=K(x_i,x_{n+1})$ and $\mathbb{V}(Y_{n+1})=K(x_{n+1},x_{n+1}))+{\sigma }^2$, so

$$(Y_1,\dots ,Y_{n+1})\sim N(0,\left(\begin{array}{cc}{K}_n+{\sigma }^{2}I& k\\ k^t& K(x_{n+1},x_{n+1})+{\sigma }^2\end{array}\right)).$$