Maximum a posteriori estimation
This article needs additional citations for verification. (September 2011) |
Part of a series on |
Bayesian statistics |
---|
Posterior = Likelihood × Prior ÷ Evidence |
Background |
Model building |
Posterior approximation |
Estimators |
|
Evidence approximation |
Model evaluation |
|
In
Description
Assume that we want to estimate an unobserved population parameter on the basis of observations . Let be the sampling distribution of , so that is the probability of when the underlying population parameter is . Then the function:
is known as the likelihood function and the estimate:
is the maximum likelihood estimate of .
Now assume that a
where is density function of , is the domain of .
The method of maximum a posteriori estimation then estimates as the mode of the posterior distribution of this random variable:
The denominator of the posterior distribution (so-called marginal likelihood) is always positive and does not depend on and therefore plays no role in the optimization. Observe that the MAP estimate of coincides with the ML estimate when the prior is uniform (i.e., is a constant function).
When the loss function is of the form
as goes to 0, the Bayes estimator approaches the MAP estimator, provided that the distribution of is quasi-concave.[1] But generally a MAP estimator is not a Bayes estimator unless is
Computation
MAP estimates can be computed in several ways:
- Analytically, when the mode(s) of the posterior distribution can be given in closed form. This is the case when conjugate priorsare used.
- Via optimization such as the conjugate gradient method or Newton's method. This usually requires first or second derivatives, which have to be evaluated analytically or numerically.
- Via a modification of an expectation-maximization algorithm. This does not require derivatives of the posterior density.
- Via a Monte Carlo method using simulated annealing
Limitations
While only mild conditions are required for MAP estimation to be a limiting case of Bayes estimation (under the 0–1 loss function),[1] it is not very representative of Bayesian methods in general. This is because MAP estimates are point estimates, whereas Bayesian methods are characterized by the use of distributions to summarize data and draw inferences: thus, Bayesian methods tend to report the posterior mean or median instead, together with credible intervals. This is both because these estimators are optimal under squared-error and linear-error loss respectively—which are more representative of typical loss functions—and for a continuous posterior distribution there is no loss function which suggests the MAP is the optimal point estimator. In addition, the posterior distribution may often not have a simple analytic form: in this case, the distribution can be simulated using Markov chain Monte Carlo techniques, while optimization to find its mode(s) may be difficult or impossible.[citation needed]
In many types of models, such as
Finally, unlike ML estimators, the MAP estimate is not invariant under reparameterization. Switching from one parameterization to another involves introducing a Jacobian that impacts on the location of the maximum.[2]
As an example of the difference between Bayes estimators mentioned above (mean and median estimators) and using a MAP estimate, consider the case where there is a need to classify inputs as either positive or negative (for example, loans as risky or safe). Suppose there are just three possible hypotheses about the correct method of classification , and with posteriors 0.4, 0.3 and 0.3 respectively. Suppose given a new instance, , classifies it as positive, whereas the other two classify it as negative. Using the MAP estimate for the correct classifier , is classified as positive, whereas the Bayes estimators would average over all hypotheses and classify as negative.
Example
Suppose that we are given a sequence of
The function to be maximized is then given by
which is equivalent to minimizing the following function of :
Thus, we see that the MAP estimator for μ is given by
which turns out to be a linear interpolation between the prior mean and the sample mean weighted by their respective covariances.
The case of is called a non-informative prior and leads to an ill-defined a priori probability distribution; in this case
References
- ^ ISSN 0025-5610.
- ISBN 978-0-262-01802-9.
- DeGroot, M. (1970). Optimal Statistical Decisions. McGraw-Hill. ISBN 0-07-016242-5.
- Sorenson, Harold W. (1980). Parameter Estimation: Principles and Problems. Marcel Dekker. ISBN 0-8247-6987-2.
- Hald, Anders (2007). "Gauss's Derivation of the Normal Distribution and the Method of Least Squares, 1809". A History of Parametric Statistical Inference from Bernoulli to Fisher, 1713–1935. New York: Springer. pp. 55–61. ISBN 978-0-387-46409-1.