RKHS

A Reproducing Kernel Hilbert Space (RKHS) is a Hilbert space endowed with more structure that makes it amenable to statistical learning. Specifically, it is defined by a Mercer kernel $K(\cdot ,\cdot )$, and consists of functions $f$ that can be written as linear combinations of the Kernel, i.e.,

$${\mathcal{H}}_0=\{f:\exists x_1,\dots ,x_k,f(x)=\sum _i{\alpha }_{k}K(x_i,x)\},$$

Write $K_{x_i}(\cdot )$ for $K(x_i,\cdot )$. The inner product is defined as

$$⟨f,g⟩=\sum _{i,j}{\alpha }_i{\beta }_{j}K(x_i,y_j),$$

if $f={\sum }_i{\alpha }_i{K}_{x_i}$ and $g={\sum }_i{\beta }_i{K}_{y_i}$. This defines the norm

$$||f|{|}_K=\sqrt{⟨f,f⟩}=\sqrt{{\alpha }^T\mathbf{K}\alpha },$$

where ${\mathbf{K}}_{i,j}=K(x_i,x_j)$. To rigorously define the RKHS, we complete ${\mathcal{H}}_0$ with respect to $||\cdot |{|}_K$, i.e., we ensure it contains its limit points. This is then a well-defined Hilbert space. We label this ${\mathcal{H}}_K$.

RKHSs are highly useful for nonparametric regression (see RKHS regression specifically) via the representer theorem, which states that the regularized empirical risk minimizer can be represented as a function in a RKHS.

Representation in $L_2$ basis

Using the $L_2$ basis representation of the Mercer kernel, we can also write functions in ${\mathcal{H}}_K$ with respect to those orthonormal functions as

$$f(x)=\sum _i{\alpha }_{i}K(x_i,x)=\sum _i{a}_i{\psi }_i(x),$$

for some ${\alpha }_i$ and $a_i$ (probably distinct). If $g(x)={\sum }_i{b}_i{\psi }_i(x)$, then we can show that show that

$$⟨f,g⟩\sum _i\frac{{\alpha }_i{\beta }_i}{{\lambda }_i},$$

where ${\lambda }_i$ are the eigenvalues: $K(x,y)={\sum }_i{\lambda }_i{\psi }_i(x){\psi }_i(y)$.