# Efficiency (statistics)

In statistics, **efficiency** is a measure of quality of an

^{[2]}Essentially, a more efficient estimator needs fewer input data or observations than a less efficient one to achieve the Cramér–Rao bound

*efficient estimator*is characterized by having the smallest possible

^{[1]}

The **relative efficiency** of two procedures is the ratio of their efficiencies, although often this concept is used where the comparison is made between a given procedure and a notional "best possible" procedure. The efficiencies and the relative efficiency of two procedures theoretically depend on the sample size available for the given procedure, but it is often possible to use the **asymptotic relative efficiency** (defined as the limit of the relative efficiencies as the sample size grows) as the principal comparison measure.

## Estimators

The efficiency of an unbiased estimator, *T*, of a parameter *θ* is defined as ^{[3]}

where is the Fisher information of the sample. Thus *e*(*T*) is the minimum possible variance for an unbiased estimator divided by its actual variance. The Cramér–Rao bound can be used to prove that *e*(*T*) ≤ 1.

### Efficient estimators

An **efficient estimator** is an

^{[4]}

In general, the spread of an estimator around the parameter θ is a measure of estimator efficiency and performance. This performance can be calculated by finding the mean squared error. More formally, let *T* be an estimator for the parameter *θ*. The mean squared error of *T* is the value , which can be decomposed as a sum of its variance and bias:

An estimator *T*_{1} performs better than an estimator *T*_{2} if .^{}[5] For a more specific case, if *T*_{1} and *T _{2} *are two unbiased estimators for the same parameter θ, then the variance can be compared to determine performance. In this case,

*T*

_{2}is

*more efficient*than

*T*

_{1}if the variance of

*T*

_{2}is

*smaller*than the variance of

*T*

_{1}, i.e. for all values of

*θ*. This relationship can be determined by simplifying the more general case above for mean squared error; since the expected value of an unbiased estimator is equal to the parameter value, . Therefore, for an unbiased estimator, , as the term drops out for being equal to 0.

^{[5]}

If an

*θ*attains for all values of the parameter, then the estimator is called efficient.

^{}[3]

Equivalently, the estimator achieves equality in the

*θ*. The Cramér–Rao lower bound

An efficient estimator is also the

Thus an efficient estimator need not exist, but if it does, it is the MVUE.

#### Finite-sample efficiency

Suppose { *P _{θ}* |

*θ*∈ Θ } is a

where is the

*θ*. Generally, the variance measures the degree of dispersion of a random variable around its mean. Thus estimators with small variances are more concentrated, they estimate the parameters more precisely. We say that the estimator is a

**finite-sample efficient estimator**(in the class of unbiased estimators) if it reaches the lower bound in the Cramér–Rao inequality above, for all

*θ*∈ Θ. Efficient estimators are always minimum variance unbiased estimators. However the converse is false: There exist point-estimation problems for which the minimum-variance mean-unbiased estimator is inefficient.

^{[6]}

Historically, finite-sample efficiency was an early optimality criterion. However this criterion has some limitations:

- Finite-sample efficient estimators are extremely rare. In fact, it was proved that efficient estimation is possible only in an exponential family, and only for the natural parameters of that family.
^{[7]} - This notion of efficiency is sometimes restricted to the class of inadmissible: Regardless of the outcome, its performance is worse than for example the James–Stein estimator.]
^{[citation needed} - Finite-sample efficiency is based on the variance, as a criterion according to which the estimators are judged. A more general approach is to use loss functions other than quadratic ones, in which case the finite-sample efficiency can no longer be formulated.
^{[citation needed]}^{[dubious – discuss]}

As an example, among the models encountered in practice, efficient estimators exist for: the mean *μ* of the normal distribution (but not the variance *σ*^{2}), parameter *λ* of the Poisson distribution, the probability *p* in the binomial or multinomial distribution.

Consider the model of a

This estimator has mean *θ* and variance of *σ*^{2} / *n*, which is equal to the reciprocal of the Fisher information from the sample. Thus, the sample mean is a finite-sample efficient estimator for the mean of the normal distribution.

### Asymptotic efficiency

Asymptotic efficiency requires Consistency (statistics), asymptotic normally distribution of estimator, and asymptotic variance-covariance matrix no worse than any other estimator.^{[9]}

#### Example: Median

Consider a sample of size drawn from a normal distribution of mean and unit variance, i.e.,

The

The variance of the mean, 1/*N* (the square of the

Now consider the

^{[10]}

The efficiency of the median for large is thus

In other words, the relative variance of the median will be , or 57% greater than the variance of the mean – the standard error of the median will be 25% greater than that of the mean.^{[11]}

Note that this is the asymptotic efficiency — that is, the efficiency in the limit as sample size tends to infinity. For finite values of the efficiency is higher than this (for example, a sample size of 3 gives an efficiency of about 74%).^{[citation needed]}

The sample mean is thus more efficient than the sample median in this example. However, there may be measures by which the median performs better. For example, the median is far more robust to outliers, so that if the Gaussian model is questionable or approximate, there may advantages to using the median (see Robust statistics).

### Dominant estimators

If and are estimators for the parameter , then is said to **dominate** if:

- its mean squared error (MSE) is smaller for at least some value of
- the MSE does not exceed that of for any value of θ.

Formally, dominates if

holds for all , with strict inequality holding somewhere.

### Relative efficiency

The relative efficiency of two unbiased estimators is defined as^{[12]}

Although is in general a function of , in many cases the dependence drops out; if this is so, being greater than one would indicate that is preferable, regardless of the true value of .

An alternative to relative efficiency for comparing estimators, is the Pitman closeness criterion. This replaces the comparison of mean-squared-errors with comparing how often one estimator produces estimates closer to the true value than another estimator.

#### Estimators of the mean of u.i.d. variables

In estimating the mean of uncorrelated, identically distributed variables we can take advantage of the fact that the variance of the sum is the sum of the variances. In this case efficiency can be defined as the square of the coefficient of variation, i.e.,^{[13]}

Relative efficiency of two such estimators can thus be interpreted as the relative sample size of one required to achieve the certainty of the other. Proof:

Now because we have , so the relative efficiency expresses the relative sample size of the first estimator needed to match the variance of the second.

### Robustness

Efficiency of an estimator may change significantly if the distribution changes, often dropping. This is one of the motivations of robust statistics – an estimator such as the sample mean is an efficient estimator of the population mean of a normal distribution, for example, but can be an inefficient estimator of a mixture distribution of two normal distributions with the same mean and different variances. For example, if a distribution is a combination of 98% *N*(*μ,* *σ*) and 2% *N*(*μ,* 10*σ*), the presence of extreme values from the latter distribution (often "contaminating outliers") significantly reduces the efficiency of the sample mean as an estimator of *μ.* By contrast, the trimmed mean is less efficient for a normal distribution, but is more robust (i.e., less affected) by changes in the distribution, and thus may be more efficient for a mixture distribution. Similarly, the shape of a distribution, such as skewness or heavy tails, can significantly reduce the efficiency of estimators that assume a symmetric distribution or thin tails.

### Uses of inefficient estimators

While efficiency is a desirable quality of an estimator, it must be weighed against other considerations, and an estimator that is efficient for certain distributions may well be inefficient for other distributions. Most significantly, estimators that are efficient for clean data from a simple distribution, such as the normal distribution (which is symmetric, unimodal, and has thin tails) may not be robust to contamination by outliers, and may be inefficient for more complicated distributions. In robust statistics, more importance is placed on robustness and applicability to a wide variety of distributions, rather than efficiency on a single distribution. M-estimators are a general class of estimators motivated by these concerns. They can be designed to yield both robustness and high relative efficiency, though possibly lower efficiency than traditional estimators for some cases. They can be very computationally complicated, however.

A more traditional alternative are L-estimators, which are very simple statistics that are easy to compute and interpret, in many cases robust, and often sufficiently efficient for initial estimates. See applications of L-estimators for further discussion.

### Efficiency in statistics

Efficiency in statistics is important because they allow one to compare the performance of various estimators. Although an unbiased estimator is usually favored over a biased one, a more efficient biased estimator can sometimes be more valuable than a less efficient unbiased estimator. For example, this can occur when the values of the biased estimator gathers around a number closer to the true value. Thus, estimator performance can be predicted easily by comparing their mean squared errors or variances.

## Hypothesis tests

For comparing

^{[14]}

## Experimental design

For experimental designs, efficiency relates to the ability of a design to achieve the objective of the study with minimal expenditure of resources such as time and money. In simple cases, the relative efficiency of designs can be expressed as the ratio of the sample sizes required to achieve a given objective.^{}[19]

## See also

## Notes

- ^
^{a}^{b}Everitt 2002, p. 128. **^**Nikulin, M.S. (2001) [1994], "Efficiency of a statistical procedure",*Encyclopedia of Mathematics*, EMS Press- ^ JSTOR 91208.
**^**Everitt 2002, p. 128.- ^ ISBN 978-1852338961.
**^**Romano, Joseph P.; Siegel, Andrew F. (1986).*Counterexamples in Probability and Statistics*. Chapman and Hall. p. 194.**)****^**DeGroot; Schervish (2002).*Probability and Statistics*(3rd ed.). pp. 440–441.- OCLC 726074601.
**.****.****.****^**Grubbs, Frank (1965).*Statistical Measures of Accuracy for Riflemen and Missile Engineers*. pp. 26–27.**^**Everitt 2002, p. 321.**^**Nikitin, Ya.Yu. (2001) [1994], "Efficiency, asymptotic",*Encyclopedia of Mathematics*, EMS Press**^**"Bahadur efficiency - Encyclopedia of Mathematics".**^**Arcones M. A. "Bahadur efficiency of the likelihood ratio test" preprint**^**Canay I. A. & Otsu, T. "Hodges–Lehmann Optimality for Testing Moment Condition Models"- ISBN 0-19-920613-9.

**
**## References

## Further reading