Empirical likelihood

In

Given a set of ${\displaystyle n}$ i.i.d. realizations ${\displaystyle y_{i}}$ of random variables ${\displaystyle Y_{i}}$, then the empirical distribution function is ${\displaystyle {\hat {F}}(y):=\sum _{i=1}^{n}\pi _{i}I(Y_{i}<y)}$, with the indicator function ${\displaystyle I}$ and the (normalized) weights ${\displaystyle \pi _{i}}$. Then, the empirical likelihood is:^[3]

${\displaystyle L:=\prod _{i=1}^{n}{\frac {{\hat {F}}(y_{i})-{\hat {F}}(y_{i}-\delta y)}{\delta y}},}$

where ${\displaystyle \delta y}$ is a small number (potentially the difference to the next smaller sample).

Empirical likelihood estimation can be augmented with side information by using further constraints (similar to the

${\displaystyle E[h(Y;\theta )]=\int _{-\infty }^{\infty }h(y;\theta )dF=0}$

${\displaystyle {\hat {E}}[h(y;\theta )]=\sum _{i=1}^{n}h(y_{i};\theta )\pi _{i}=0}$

With similar constraints, we could also model correlation.

Discrete random variables

The empirical-likelihood method can also be also employed for

Given ${\displaystyle \ p_{i}:={\hat {F}}(y_{i})-{\hat {F}}(y_{i}-\delta y),\ i=1,...,n}$ such that ${\displaystyle p_{i}\geq 0{\text{ and }}\sum _{i=1}^{n}\ p_{i}=1.}$

Then the empirical likelihood is again ${\displaystyle L(p_{1},...,p_{n})=\prod _{i=1}^{n}\ p_{i}}$.

Using the

Lagrangian multiplier

method to maximize the logarithm of the empirical likelihood subject to the trivial normalization constraint, we find ${\displaystyle p_{i}=1/n}$ as a maximum. Therefore, ${\displaystyle {\hat {F}}}$ is the

Estimation Procedure

EL estimates are calculated by maximizing the empirical

${\displaystyle s.t.\sum _{i=1}^{n}\pi _{i}=1,\sum _{i=1}^{n}\pi _{i}h(y_{i};\theta )=0,\forall i\in [1..n]\quad 0\leq \pi _{i}.}$[6]^{: Equation (73)}

The value of the theta parameter can be found by solving the Lagrangian function

${\displaystyle {\mathcal {L}}=\sum _{i=1}^{n}\ln \pi _{i}+\mu (1-\sum _{i=1}^{n}\pi _{i})-n\tau '\sum _{i=1}^{n}\pi _{i}h(y_{i};\theta ).}$^{[6]: Equation (74)}

There is a clear analogy between this maximization problem and the one solved for maximum entropy.

The parameters ${\displaystyle \pi _{i}}$ are nuisance parameters.

Empirical Likelihood Ratio (ELR)

An empirical likelihood ratio function is defined and used to obtain confidence intervals parameter of interest θ similar to parametric likelihood ratio confidence intervals.^[7][8] Let L(F) be the empirical likelihood of function ${\displaystyle F}$, then the ELR would be:

${\displaystyle R(F)=L(F)/L(F_{n})}$.

Consider sets of the form

${\displaystyle C=\{T(F)|R(F)\geq r\}}$.

Under such conditions a test of ${\displaystyle T(F)=t}$ rejects when t does not belong to ${\displaystyle C}$, that is, when no distribution F with ${\displaystyle T(F)=t}$ has likelihood ${\displaystyle L(F)\geq rL(F_{n})}$.

The central result is for the mean of X. Clearly, some restrictions on ${\displaystyle F}$ are needed, or else ${\displaystyle C=\mathbb {R} ^{p}}$ whenever ${\displaystyle r<1}$. To see this, let:

${\displaystyle F=\epsilon \delta _{x}+(1-\epsilon )F_{n}}$

If ${\displaystyle \epsilon }$ is small enough and ${\displaystyle \epsilon >0}$, then ${\displaystyle R(F)\geq r}$.

But then, as ${\displaystyle x}$ ranges through ${\displaystyle \mathbb {R} ^{p}}$, so does the mean of ${\displaystyle F}$, tracing out ${\displaystyle C=\mathbb {R} ^{p}}$. The problem can be solved by restricting to distributions F that are supported in a bounded set. It turns out to be possible to restrict attention t distributions with support in the sample, in other words, to distribution ${\displaystyle F\ll F_{n}}$. Such method is convenient since the statistician might not be willing to specify a bounded support for ${\displaystyle F}$, and since ${\displaystyle t}$ converts the construction of ${\displaystyle C}$ into a finite dimensional problem.

Other Applications

The use of empirical likelihood is not limited to confidence intervals. In efficient quantile regression, an EL-based categorization^[9] procedure helps determine the shape of the true discrete distribution at level p, and also provides a way of formulating a consistent estimator. In addition, EL can be used in place of parametric likelihood to form model selection criteria.^[10] Empirical likelihood can naturally be applied in survival analysis^[11] or regression problems^[12]

See also

• Bootstrapping (statistics)
• Jackknife (statistics)
• Nelson–Aalen estimator

Literature

• Nordman, Daniel J., and Soumendra N. Lahiri. "A review of empirical likelihood methods for time series." Journal of Statistical Planning and Inference 155 (2014): 1-18. https://doi.org/10.1016/j.jspi.2013.10.001

References