Fisher information

Intuitively, the Fisher information is the expected curvature around a parameter. The more curvature, the “easier” the parameter is to estimate.

Formally, the Fisher information is the expected outer-product of the score function, which is the gradient of the log-likelihood. If $s_n(\theta )={\nabla }_{\theta }{\ell }_n(\theta )$ is the score based on $n$ observations, then the Fisher information is:

$$I_n(\theta ):=\mathbb{E}[s(\theta )s(\theta {)}^{\top }].$$

Since $s(\theta )$ is a $d$-dimensional vector (if $\theta$ is d-dimensional), the Fisher information is a $d\times d$ matrix. For iid data, the Fisher information satisfies $I_n(\theta )=n{I}_1(\theta )$, which is extremely handy.

An equivalent way to define it is

$$I_1(\theta )=-\mathbb{E}[{\nabla }_{\theta }^2{\ell }_1(\theta )],$$

i.e. the second derivative of the negative log-likelihood.