Likelihood function
Part of a series on |
Bayesian statistics |
---|
Posterior = Likelihood × Prior ÷ Evidence |
Background |
Model building |
Posterior approximation |
Estimators |
|
Evidence approximation |
Model evaluation |
|
The likelihood function (often simply called the likelihood) is the
Intuitively, the likelihood function is the probability of observing data assuming is the actual parameter.In maximum likelihood estimation, the arg max (over the parameter ) of the likelihood function serves as a point estimate for , while the Fisher information (often approximated by the likelihood's Hessian matrix) indicates the estimate's precision.
In contrast, in Bayesian statistics, parameter estimates are derived from the converse of the likelihood, the so-called posterior probability, which is calculated via Bayes' rule.[4]
Definition
The likelihood function, parameterized by a (possibly multivariate) parameter , is usually defined differently for discrete and continuous probability distributions (a more general definition is discussed below). Given a probability density or mass function
where is a realization of the random variable , the likelihood function is
In other words, when is viewed as a function of with fixed, it is a probability density function, and when viewed as a function of with fixed, it is a likelihood function. In the frequentist paradigm, the notation is often avoided and instead or are used to indicate that is regarded as a fixed unknown quantity rather than as a random variable being conditioned on.
The likelihood function does not specify the probability that is the truth, given the observed sample . Such an interpretation is a common error, with potentially disastrous consequences (see
Discrete probability distribution
Let be a discrete random variable with probability mass function depending on a parameter . Then the function
considered as a function of , is the likelihood function, given the outcome of the random variable . Sometimes the probability of "the value of for the parameter value " is written as P(X = x | θ) or P(X = x; θ). The likelihood is the probability that a particular outcome is observed when the true value of the parameter is , equivalent to the probability mass on ; it is not a probability density over the parameter . The likelihood, , should not be confused with , which is the posterior probability of given the data .
Given no event (no data), the likelihood is 1;[citation needed] any non-trivial event will have a lower likelihood.
Example
Consider a simple statistical model of a coin flip: a single parameter that expresses the "fairness" of the coin. The parameter is the probability that a coin lands heads up ("H") when tossed. can take on any value within the range 0.0 to 1.0. For a perfectly fair coin, .
Imagine flipping a fair coin twice, and observing two heads in two tosses ("HH"). Assuming that each successive coin flip is i.i.d., then the probability of observing HH is
Equivalently, the likelihood of observing "HH" assuming is
This is not the same as saying that , a conclusion which could only be reached via Bayes' theorem given knowledge about the marginal probabilities and .
Now suppose that the coin is not a fair coin, but instead that . Then the probability of two heads on two flips is
Hence
More generally, for each value of , we can calculate the corresponding likelihood. The result of such calculations is displayed in Figure 1. The integral of over [0, 1] is 1/3; likelihoods need not integrate or sum to one over the parameter space.
Continuous probability distribution
Let be a random variable following an absolutely continuous probability distribution with density function (a function of ) which depends on a parameter . Then the function
considered as a function of , is the likelihood function (of , given the outcome ). Again, is not a probability density or mass function over , despite being a function of given the observation .
Relationship between the likelihood and probability density functions
The use of the probability density in specifying the likelihood function above is justified as follows. Given an observation , the likelihood for the interval , where is a constant, is given by . Observe that
where is the probability density function, it follows that
The first fundamental theorem of calculus provides that
Then
Therefore,
In general
In measure-theoretic probability theory, the density function is defined as the Radon–Nikodym derivative of the probability distribution relative to a common dominating measure.[5] The likelihood function is this density interpreted as a function of the parameter, rather than the random variable.[6] Thus, we can construct a likelihood function for any distribution, whether discrete, continuous, a mixture, or otherwise. (Likelihoods are comparable, e.g. for parameter estimation, only if they are Radon–Nikodym derivatives with respect to the same dominating measure.)
The above discussion of the likelihood for discrete random variables uses the counting measure, under which the probability density at any outcome equals the probability of that outcome.
Likelihoods for mixed continuous–discrete distributions
The above can be extended in a simple way to allow consideration of distributions which contain both discrete and continuous components. Suppose that the distribution consists of a number of discrete probability masses and a density , where the sum of all the 's added to the integral of is always one. Assuming that it is possible to distinguish an observation corresponding to one of the discrete probability masses from one which corresponds to the density component, the likelihood function for an observation from the continuous component can be dealt with in the manner shown above. For an observation from the discrete component, the likelihood function for an observation from the discrete component is simply
The fact that the likelihood function can be defined in a way that includes contributions that are not commensurate (the density and the probability mass) arises from the way in which the likelihood function is defined up to a constant of proportionality, where this "constant" can change with the observation , but not with the parameter .
Regularity conditions
In the context of parameter estimation, the likelihood function is usually assumed to obey certain conditions, known as regularity conditions. These conditions are assumed in various proofs involving likelihood functions, and need to be verified in each particular application. For maximum likelihood estimation, the existence of a global maximum of the likelihood function is of the utmost importance. By the
More specifically, if the likelihood function is twice continuously differentiable on the k-dimensional parameter space assumed to be an open connected subset of there exists a unique maximum if the matrix of second partials
In the proofs of consistency and asymptotic normality of the maximum likelihood estimator, additional assumptions are made about the probability densities that form the basis of a particular likelihood function. These conditions were first established by Chanda.[10] In particular, for almost all , and for all
The above conditions are sufficient, but not necessary. That is, a model that does not meet these regularity conditions may or may not have a maximum likelihood estimator of the properties mentioned above. Further, in case of non-independently or non-identically distributed observations additional properties may need to be assumed.
In Bayesian statistics, almost identical regularity conditions are imposed on the likelihood function in order to proof asymptotic normality of the
Likelihood ratio and relative likelihood
Likelihood ratio
A likelihood ratio is the ratio of any two specified likelihoods, frequently written as:
The likelihood ratio is central to
In
The likelihood ratio is also of central importance in
The likelihood ratio is not directly used in AIC-based statistics. Instead, what is used is the relative likelihood of models (see below).
In
Relative likelihood function
Since the actual value of the likelihood function depends on the sample, it is often convenient to work with a standardized measure. Suppose that the
Likelihood region
A likelihood region is the set of all values of θ whose relative likelihood is greater than or equal to a given threshold. In terms of percentages, a p% likelihood region for θ is defined to be[16][18][21]
If θ is a single real parameter, a p% likelihood region will usually comprise an interval of real values. If the region does comprise an interval, then it is called a likelihood interval.[16][18][22]
Likelihood intervals, and more generally likelihood regions, are used for interval estimation within likelihoodist statistics: they are similar to confidence intervals in frequentist statistics and credible intervals in Bayesian statistics. Likelihood intervals are interpreted directly in terms of relative likelihood, not in terms of coverage probability (frequentism) or posterior probability (Bayesianism).
Given a model, likelihood intervals can be compared to confidence intervals. If θ is a single real parameter, then under certain conditions, a 14.65% likelihood interval (about 1:7 likelihood) for θ will be the same as a 95% confidence interval (19/20 coverage probability).[16][21] In a slightly different formulation suited to the use of log-likelihoods (see Wilks' theorem), the test statistic is twice the difference in log-likelihoods and the probability distribution of the test statistic is approximately a chi-squared distribution with degrees-of-freedom (df) equal to the difference in df's between the two models (therefore, the e−2 likelihood interval is the same as the 0.954 confidence interval; assuming difference in df's to be 1).[21][22]
Likelihoods that eliminate nuisance parameters
In many cases, the likelihood is a function of more than one parameter but interest focuses on the estimation of only one, or at most a few of them, with the others being considered as nuisance parameters. Several alternative approaches have been developed to eliminate such nuisance parameters, so that a likelihood can be written as a function of only the parameter (or parameters) of interest: the main approaches are profile, conditional, and marginal likelihoods.[23][24] These approaches are also useful when a high-dimensional likelihood surface needs to be reduced to one or two parameters of interest in order to allow a graph.
Profile likelihood
It is possible to reduce the dimensions by concentrating the likelihood function for a subset of parameters by expressing the nuisance parameters as functions of the parameters of interest and replacing them in the likelihood function.[25][26] In general, for a likelihood function depending on the parameter vector that can be partitioned into , and where a correspondence can be determined explicitly, concentration reduces computational burden of the original maximization problem.[27]
For instance, in a linear regression with normally distributed errors, , the coefficient vector could be partitioned into (and consequently the design matrix ). Maximizing with respect to yields an optimal value function . Using this result, the maximum likelihood estimator for can then be derived as
Since graphically the procedure of concentration is equivalent to slicing the likelihood surface along the ridge of values of the nuisance parameter that maximizes the likelihood function, creating an isometric profile of the likelihood function for a given , the result of this procedure is also known as profile likelihood.
Conditional likelihood
Sometimes it is possible to find a sufficient statistic for the nuisance parameters, and conditioning on this statistic results in a likelihood which does not depend on the nuisance parameters.[32]
One example occurs in 2×2 tables, where conditioning on all four marginal totals leads to a conditional likelihood based on the non-central hypergeometric distribution. This form of conditioning is also the basis for Fisher's exact test.
Marginal likelihood
Sometimes we can remove the nuisance parameters by considering a likelihood based on only part of the information in the data, for example by using the set of ranks rather than the numerical values. Another example occurs in linear
Partial likelihood
A partial likelihood is an adaption of the full likelihood such that only a part of the parameters (the parameters of interest) occur in it.[33] It is a key component of the proportional hazards model: using a restriction on the hazard function, the likelihood does not contain the shape of the hazard over time.
Products of likelihoods
The likelihood, given two or more independent events, is the product of the likelihoods of each of the individual events:
This is particularly important when the events are from
The empty product has value 1, which corresponds to the likelihood, given no event, being 1: before any data, the likelihood is always 1. This is similar to a
Log-likelihood
Log-likelihood function is the logarithm of the likelihood function, often denoted by a lowercase l or , to contrast with the uppercase L or for the likelihood. Because logarithms are
Given the independence of each event, the overall log-likelihood of intersection equals the sum of the log-likelihoods of the individual events. This is analogous to the fact that the overall
A logarithm of a likelihood ratio is equal to the difference of the log-likelihoods:
Just as the likelihood, given no event, being 1, the log-likelihood, given no event, is 0, which corresponds to the value of the empty sum: without any data, there is no support for any models.
Graph
The graph of the log-likelihood is called the support curve (in the univariate case).[36] In the multivariate case, the concept generalizes into a support surface over the parameter space. It has a relation to, but is distinct from, the support of a distribution.
The term was coined by
The log-likelihood function being plotted is used in the computation of the
Likelihood equations
If the log-likelihood function is
The equations defined by the stationary point of the score function serve as estimating equations for the maximum likelihood estimator.
The second derivative evaluated at , known as Fisher information, determines the curvature of the likelihood surface,[40] and thus indicates the precision of the estimate.[41]
Exponential families
The log-likelihood is also particularly useful for
An exponential family is one whose probability density function is of the form (for some functions, writing for the
Each of these terms has an interpretation,[a] but simply switching from probability to likelihood and taking logarithms yields the sum:
The and each correspond to a
In words, the log-likelihood of an exponential family is inner product of the natural parameter and the sufficient statistic , minus the normalization factor (
Example: the gamma distribution
The gamma distribution is an exponential family with two parameters, and . The likelihood function is
Finding the maximum likelihood estimate of for a single observed value looks rather daunting. Its logarithm is much simpler to work with:
To maximize the log-likelihood, we first take the partial derivative with respect to :
If there are a number of independent observations , then the joint log-likelihood will be the sum of individual log-likelihoods, and the derivative of this sum will be a sum of derivatives of each individual log-likelihood:
To complete the maximization procedure for the joint log-likelihood, the equation is set to zero and solved for :
Here denotes the maximum-likelihood estimate, and is the
Background and interpretation
Historical remarks
The term "likelihood" has been in use in English since at least late
[I]n 1922, I proposed the term 'likelihood,' in view of the fact that, with respect to [the parameter], it is not a probability, and does not obey the laws of probability, while at the same time it bears to the problem of rational choice among the possible values of [the parameter] a relation similar to that which probability bears to the problem of predicting events in games of chance. . . . Whereas, however, in relation to psychological judgment, likelihood has some resemblance to probability, the two concepts are wholly distinct. . . ."[46]
The concept of likelihood should not be confused with probability as mentioned by Sir Ronald Fisher
I stress this because in spite of the emphasis that I have always laid upon the difference between probability and likelihood there is still a tendency to treat likelihood as though it were a sort of probability. The first result is thus that there are two different measures of rational belief appropriate to different cases. Knowing the population we can express our incomplete knowledge of, or expectation of, the sample in terms of probability; knowing the sample we can express our incomplete knowledge of the population in terms of likelihood.[47]
Fisher's invention of statistical likelihood was in reaction against an earlier form of reasoning called inverse probability.[48] His use of the term "likelihood" fixed the meaning of the term within mathematical statistics.
A. W. F. Edwards (1972) established the axiomatic basis for use of the log-likelihood ratio as a measure of relative support for one hypothesis against another. The support function is then the natural logarithm of the likelihood function. Both terms are used in phylogenetics, but were not adopted in a general treatment of the topic of statistical evidence.[49]
Interpretations under different foundations
Among statisticians, there is no consensus about what the
For each of the proposed foundations, the interpretation of likelihood is different. The four interpretations are described in the subsections below.Frequentist interpretation
This section is empty. You can help by adding to it. (March 2019) |
Bayesian interpretation
In
Likelihoodist interpretation
This article includes a list of general references, but it lacks sufficient corresponding inline citations. (April 2019) |
In frequentist statistics, the likelihood function is itself a statistic that summarizes a single sample from a population, whose calculated value depends on a choice of several parameters θ1 ... θp, where p is the count of parameters in some already-selected statistical model. The value of the likelihood serves as a figure of merit for the choice used for the parameters, and the parameter set with maximum likelihood is the best choice, given the data available.
The specific calculation of the likelihood is the probability that the observed sample would be assigned, assuming that the model chosen and the values of the several parameters θ give an accurate approximation of the
Each independent sample's maximum likelihood estimate is a separate estimate of the "true" parameter set describing the population sampled. Successive estimates from many independent samples will cluster together with the population's "true" set of parameter values hidden somewhere in their midst. The difference in the logarithms of the maximum likelihood and adjacent parameter sets' likelihoods may be used to draw a confidence region on a plot whose co-ordinates are the parameters θ1 ... θp. The region surrounds the maximum-likelihood estimate, and all points (parameter sets) within that region differ at most in log-likelihood by some fixed value. The χ2 distribution given by Wilks' theorem converts the region's log-likelihood differences into the "confidence" that the population's "true" parameter set lies inside. The art of choosing the fixed log-likelihood difference is to make the confidence acceptably high while keeping the region acceptably small (narrow range of estimates).
As more data are observed, instead of being used to make independent estimates, they can be combined with the previous samples to make a single combined sample, and that large sample may be used for a new maximum likelihood estimate. As the size of the combined sample increases, the size of the likelihood region with the same confidence shrinks. Eventually, either the size of the confidence region is very nearly a single point, or the entire population has been sampled; in both cases, the estimated parameter set is essentially the same as the population parameter set.
AIC-based interpretation
This section needs expansion. You can help by adding to it. (March 2019) |
Under the AIC paradigm, likelihood is interpreted within the context of information theory.[57][58][59]
See also
- Bayes factor
- Conditional entropy
- Conditional probability
- Empirical likelihood
- Likelihood principle
- Likelihood-ratio test
- Likelihoodist statistics
- Maximum likelihood
- Principle of maximum entropy
- Pseudolikelihood
- Score (statistics)
Notes
References
- ISBN 0-534-24312-6.
- ISBN 978-1-4419-0925-1.
- ISBN 0-387-98502-6.
- ISBN 0-471-98165-6.
- John Wiley & Sons. pp. 422–423.
- ^ Shao, Jun (2003). Mathematical Statistics (2nd ed.). Springer. §4.4.1.
- ISBN 0-521-40551-3.
- JSTOR 2240844.
- S2CID 15896597.
- JSTOR 2333005.
- ISBN 0-471-09077-8.
- .
- .
- ISBN 0-444-88376-2.
- .
- ^ a b c d Kalbfleisch, J. G. (1985), Probability and Statistical Inference, Springer (§9.3).
- ISBN 9780412606502(§1.4.2).
- ^ a b c Sprott, D. A. (2000), Statistical Inference in Science, Springer (chap. 2).
- ^ Davison, A. C. (2008), Statistical Models, Cambridge University Press (§4.1.2).
- ^ Held, L.; Sabanés Bové, D. S. (2014), Applied Statistical Inference—Likelihood and Bayes, Springer (§2.1).
- ^ a b c Rossi, R. J. (2018), Mathematical Statistics, Wiley, p. 267.
- ^ Journal of the Royal Statistical Society, Series B, 33 (2): 256–262.
- ^ Pawitan, Yudi (2001). In All Likelihood: Statistical Modelling and Inference Using Likelihood. Oxford University Press.
- ^ Wen Hsiang Wei. "Generalized Linear Model - course notes". Taichung, Taiwan: Tunghai University. pp. Chapter 5. Retrieved 2017-10-01.
- ISBN 978-0-674-00560-0.
- ISBN 978-0-19-506011-9.
- ISBN 978-0-521-40551-5.
- ISBN 0-86094-190-6.
- ISBN 978-0-691-12522-0.
- ISBN 0-387-90777-7.
- JSTOR 2347496.
- JSTOR 25049882.
- ^
MR 0400509.
- ISBN 0-471-82668-5.
- ^ Papadopoulos, Alecos (September 25, 2013). "Why we always put log() before the joint pdf when we use MLE (Maximum likelihood Estimation)?". Stack Exchange.
- ^ ISBN 0-8018-4443-6.
- .
- .
- .
- .
- ^ Ward, Michael D.; Ahlquist, John S. (2018). Maximum Likelihood for Social Science : Strategies for Analysis. Cambridge University Press. pp. 25–27.
- ^ "likelihood", Shorter Oxford English Dictionary (2007).
- JSTOR 2676741.
- ^ Fisher, R.A. (1921). "On the "probable error" of a coefficient of correlation deduced from a small sample". Metron. 1: 3–32.
- JSTOR 91208.
- ^ Klemens, Ben (2008). Modeling with Data: Tools and Techniques for Scientific Computing. Princeton University Press. p. 329.
- .
- .
- ^ Royall, R. (1997). Statistical Evidence. Chapman & Hall.
- North-Holland Publishing.
- ^ a b c d I. J. Good: Probability and the Weighing of Evidence (Griffin 1950), §6.1
- ^ a b c d H. Jeffreys: Theory of Probability (3rd ed., Oxford University Press 1983), §1.22
- ^ a b c d e E. T. Jaynes: Probability Theory: The Logic of Science (Cambridge University Press 2003), §4.1
- ^ a b c d D. V. Lindley: Introduction to Probability and Statistics from a Bayesian Viewpoint. Part 1: Probability (Cambridge University Press 1980), §1.6
- ^ a b c d A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, D. B. Rubin: Bayesian Data Analysis (3rd ed., Chapman & Hall/CRC 2014), §1.3
- ISBN 9781118341544
- ^ Akaike, H. (1985). "Prediction and entropy". In Atkinson, A. C.; Fienberg, S. E. (eds.). A Celebration of Statistics. Springer. pp. 1–24.
- ^ Sakamoto, Y.; Ishiguro, M.; Kitagawa, G. (1986). Akaike Information Criterion Statistics. D. Reidel. Part I.
- Springer-Verlag. chap. 7.
Further reading
- Azzalini, Adelchi (1996). "Likelihood". Statistical Inference Based on the Likelihood. Chapman and Hall. pp. 17–50. ISBN 0-412-60650-X.
- Boos, Dennis D.; Stefanski, L. A. (2013). "Likelihood Construction and Estimation". Essential Statistical Inference : Theory and Methods. New York: Springer. pp. 27–124. ISBN 978-1-4614-4817-4.
- ISBN 0-8018-4443-6.
- ISBN 0-521-36697-6.
- Richard, Mark; Vecer, Jan (1 February 2021). "Efficiency Testing of Prediction Markets: Martingale Approach, Likelihood Ratio and Bayes Factor Analysis". Risks. 9 (2): 31. .
- Lindsey, J. K. (1996). "Likelihood". Parametric Statistical Inference. Oxford University Press. pp. 69–139. ISBN 0-19-852359-9.
- Rohde, Charles A. (2014). Introductory Statistical Inference with the Likelihood Function. Berlin: Springer. ISBN 978-3-319-10460-7.
- Royall, Richard (1997). Statistical Evidence : A Likelihood Paradigm. London: Chapman & Hall. ISBN 0-412-04411-0.
- ISBN 978-1-316-63682-4.