Observed information

In

Suppose we observe random variables ${\displaystyle X_{1},\ldots ,X_{n}}$, independent and identically distributed with density f(X; θ), where θ is a (possibly unknown) vector. Then the log-likelihood of the parameters ${\displaystyle \theta }$ given the data ${\displaystyle X_{1},\ldots ,X_{n}}$ is

${\displaystyle \ell (\theta |X_{1},\ldots ,X_{n})=\sum _{i=1}^{n}\log f(X_{i}|\theta )}$.

We define the observed information matrix at ${\displaystyle \theta ^{*}}$ as

${\displaystyle {\mathcal {J}}(\theta ^{*})=-\left.\nabla \nabla ^{\top }\ell (\theta )\right|_{\theta =\theta ^{*}}}$

${\displaystyle =-\left.\left({\begin{array}{cccc}{\tfrac {\partial ^{2}}{\partial \theta _{1}^{2}}}&{\tfrac {\partial ^{2}}{\partial \theta _{1}\partial \theta _{2}}}&\cdots &{\tfrac {\partial ^{2}}{\partial \theta _{1}\partial \theta _{p}}}\\{\tfrac {\partial ^{2}}{\partial \theta _{2}\partial \theta _{1}}}&{\tfrac {\partial ^{2}}{\partial \theta _{2}^{2}}}&\cdots &{\tfrac {\partial ^{2}}{\partial \theta _{2}\partial \theta _{p}}}\\\vdots &\vdots &\ddots &\vdots \\{\tfrac {\partial ^{2}}{\partial \theta _{p}\partial \theta _{1}}}&{\tfrac {\partial ^{2}}{\partial \theta _{p}\partial \theta _{2}}}&\cdots &{\tfrac {\partial ^{2}}{\partial \theta _{p}^{2}}}\\\end{array}}\right)\ell (\theta )\right|_{\theta =\theta ^{*}}}$

Since the inverse of the information matrix is the

Alternative definition

Andrew Gelman, David Dunson and Donald Rubin^[2] define observed information instead in terms of the parameters' posterior probability, ${\displaystyle p(\theta |y)}$:

${\displaystyle I(\theta )=-{\frac {d^{2}}{d\theta ^{2}}}\log p(\theta |y)}$

Fisher information

The Fisher information ${\displaystyle {\mathcal {I}}(\theta )}$ is the expected value of the observed information given a single observation ${\displaystyle X}$ distributed according to the hypothetical model with parameter ${\displaystyle \theta }$:

${\displaystyle {\mathcal {I}}(\theta )=\mathrm {E} ({\mathcal {J}}(\theta ))}$.

Comparison with the expected information

The comparison between the observed information and the expected information remains an active and ongoing area of research and debate.

as an approximation of the true information if an error term of ${\displaystyle O(n^{-3/2})}$ is ignored.

However, when the construction of

• Fisher information matrix
• Fisher information metric

References