rkhs regression

We are interested in solving the following regularized nonparametric regression problem under the squared loss:

$$\arg \underset{m}{\min }R(m)=\sum _i(Y_i-m(X_i){)}^2+\lambda ||m|{|}_K^2,$$

where $K$ is a Mercer kernel. This is optimization over a RKHS. The representer theorem tells us that the the minimizer $\hat{m}$ has the form

$$\hat{m}(x)=\sum _i{\alpha }_{i}K(x_i,x),$$

so the problem reduces to find the optimal $\alpha =({\alpha }_1,\dots ,{\alpha }_n).$ Thus we can write $R=||\mathbf{Y}-\mathbf{K}\alpha |{|}_2^2+\lambda {\alpha }^T\mathbf{K}\alpha ,$ where ${\mathbf{K}}_{i,j}=K(x_i,x_j)$. Taking the gradient w.r.t. to $\alpha$ and keeping in mind that $K$ is invertible and symmetric, the minimizer is

$$\hat{\alpha }=(\mathbf{K}+\lambda I{)}^{-1}\mathbf{Y}.$$

So $\hat{m}(x)={\sum }_j\widehat{{\alpha }_j}K(x_j,x)$, so

$$\hat{m}(x)=k_x^T(\mathbf{K}+\lambda I{)}^{-1}\mathbf{Y},$$

where $k_x=(K(x_1,x),\dots ,K(x_n,x))$. This demonstrates that RKHS regression is a member of the class of linear smoothers.