Bayes factor

Part of a series on
Bayesian statistics
Posterior = Likelihood × Prior ÷ Evidence
Background
Bayesian inference Bayesian probability Bayes' theorem Bernstein–von Mises theorem Coherence Cox's theorem Cromwell's rule Principle of indifference Principle of maximum entropy
Model building
Strong prior Conjugate prior Linear regression Empirical Bayes Hierarchical model
Posterior approximation
Markov chain Monte Carlo Laplace's approximation Integrated nested Laplace approximations Variational inference Approximate Bayesian computation
Estimators
Bayesian estimator Credible interval Maximum a posteriori estimation
Evidence approximation
Evidence lower bound Nested sampling
Model evaluation
Bayes factor Model averaging Posterior predictive
Mathematics portal
v t e

The Bayes factor is a ratio of two competing

The Bayes factor is the ratio of two marginal likelihoods; that is, the likelihoods of two statistical models integrated over the prior probabilities of their parameters.^[9]

The posterior probability ${\displaystyle \Pr(M|D)}$ of a model M given data D is given by Bayes' theorem:

${\displaystyle \Pr(M|D)={\frac {\Pr(D|M)\Pr(M)}{\Pr(D)}}.}$

The key data-dependent term ${\displaystyle \Pr(D|M)}$ represents the probability that some data are produced under the assumption of the model M; evaluating it correctly is the key to Bayesian model comparison.

Given a model selection problem in which one wishes to choose between two models on the basis of observed data D, the plausibility of the two different models M₁ and M₂, parametrised by model parameter vectors ${\displaystyle \theta _{1}}$ and ${\displaystyle \theta _{2}}$, is assessed by the Bayes factor K given by

${\displaystyle K={\frac {\Pr(D|M_{1})}{\Pr(D|M_{2})}}={\frac {\int \Pr(\theta _{1}|M_{1})\Pr(D|\theta _{1},M_{1})\,d\theta _{1}}{\int \Pr(\theta _{2}|M_{2})\Pr(D|\theta _{2},M_{2})\,d\theta _{2}}}={\frac {\frac {\Pr(M_{1}|D)\Pr(D)}{\Pr(M_{1})}}{\frac {\Pr(M_{2}|D)\Pr(D)}{\Pr(M_{2})}}}={\frac {\Pr(M_{1}|D)}{\Pr(M_{2}|D)}}{\frac {\Pr(M_{2})}{\Pr(M_{1})}}.}$

When the two models have equal prior probability, so that ${\displaystyle \Pr(M_{1})=\Pr(M_{2})}$, the Bayes factor is equal to the ratio of the posterior probabilities of M₁ and M₂. If instead of the Bayes factor integral, the likelihood corresponding to the

Other approaches are:

• to treat model comparison as a decision problem, computing the expected value or cost of each model choice;
• to use minimum message length (MML).
• to use minimum description length (MDL).

Interpretation

A value of K > 1 means that M₁ is more strongly supported by the data under consideration than M₂. Note that classical

An alternative table, widely cited, is provided by Kass and Raftery (1995):[10]

Example

Suppose we have a

at the 5% significance level, the Bayes factor hardly considers this to be an extreme result. Note, however, that a non-uniform prior (for example one that reflects the fact that you expect the number of success and failures to be of the same order of magnitude) could result in a Bayes factor that is more in agreement with the frequentist hypothesis test.

A classical

maximum likelihood

estimate for q, namely ${\displaystyle {\hat {q}}={\frac {115}{200}}=0.575}$, whence

${\displaystyle \textstyle P(X=115\mid M_{2})={{200 \choose 115}{\hat {q}}^{115}(1-{\hat {q}})^{85}}\approx 0.06}$

(rather than averaging over all possible q). That gives a likelihood ratio of 0.1 and points towards M₂.

M₂ is a more complex model than M₁ because it has a free parameter which allows it to model the data more closely. The ability of Bayes factors to take this into account is a reason why

On the other hand, the modern method of relative likelihood takes into account the number of free parameters in the models, unlike the classical likelihood ratio. The relative likelihood method could be applied as follows. Model M₁ has 0 parameters, and so its Akaike information criterion (AIC) value is ${\displaystyle 2\cdot 0-2\cdot \ln(0.005956)\approx 10.2467}$. Model M₂ has 1 parameter, and so its AIC value is ${\displaystyle 2\cdot 1-2\cdot \ln(0.056991)\approx 7.7297}$. Hence M₁ is about ${\displaystyle \exp \left({\frac {7.7297-10.2467}{2}}\right)\approx 0.284}$ times as probable as M₂ to minimize the information loss. Thus M₂ is slightly preferred, but M₁ cannot be excluded.

See also

• Akaike information criterion
• Approximate Bayesian computation
• Bayesian information criterion
• Deviance information criterion
• Lindley's paradox
• Minimum message length
• Model selection
• E-Value

Statistical ratios

• Odds ratio
• Relative risk

References