Bayesian probability
Part of a series on |
Bayesian statistics |
---|
Posterior = Likelihood × Prior ÷ Evidence |
Background |
Model building |
Posterior approximation |
Estimators |
|
Evidence approximation |
Model evaluation |
|
Bayesian probability (/ˈbeɪziən/ BAY-zee-ən or /ˈbeɪʒən/ BAY-zhən)[1] is an interpretation of the concept of probability, in which, instead of frequency or propensity of some phenomenon, probability is interpreted as reasonable expectation[2] representing a state of knowledge[3] or as quantification of a personal belief.[4]
The Bayesian interpretation of probability can be seen as an extension of
Bayesian probability belongs to the category of evidential probabilities; to evaluate the probability of a hypothesis, the Bayesian probabilist specifies a prior probability. This, in turn, is then updated to a posterior probability in the light of new, relevant data (evidence).[7] The Bayesian interpretation provides a standard set of procedures and formulae to perform this calculation.
The term Bayesian derives from the 18th-century mathematician and theologian Thomas Bayes, who provided the first mathematical treatment of a non-trivial problem of statistical data analysis using what is now known as Bayesian inference.[8]: 131 Mathematician Pierre-Simon Laplace pioneered and popularized what is now called Bayesian probability.[8]: 97–98
Bayesian methodology
Bayesian methods are characterized by concepts and procedures as follows:
- The use of random variables, or more generally unknown quantities,[9] to model all sources of uncertainty in statistical models including uncertainty resulting from lack of information (see also aleatoric and epistemic uncertainty).
- The need to determine the prior probability distribution taking into account the available (prior) information.
- The sequential use of Bayes' theorem: as more data become available, calculate the posterior distribution using Bayes' theorem; subsequently, the posterior distribution becomes the next prior.
- While for the frequentist, a hypothesis is a proposition (which must be either true or false) so that the frequentist probability of a hypothesis is either 0 or 1, in Bayesian statistics, the probability that can be assigned to a hypothesis can also be in a range from 0 to 1 if the truth value is uncertain.
Objective and subjective Bayesian probabilities
Broadly speaking, there are two interpretations of Bayesian probability. For objectivists, who interpret probability as an extension of logic, probability quantifies the reasonable expectation that everyone (even a "robot") who shares the same knowledge should share in accordance with the rules of Bayesian statistics, which can be justified by Cox's theorem.[3][10] For subjectivists, probability corresponds to a personal belief.[4] Rationality and coherence allow for substantial variation within the constraints they pose; the constraints are justified by the Dutch book argument or by decision theory and de Finetti's theorem.[4] The objective and subjective variants of Bayesian probability differ mainly in their interpretation and construction of the prior probability.
History
The term Bayesian derives from
In the 20th century, the ideas of Laplace developed in two directions, giving rise to objective and subjective currents in Bayesian practice.
In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the discovery of Markov chain Monte Carlo methods and the consequent removal of many of the computational problems, and to an increasing interest in nonstandard, complex applications.[18] While frequentist statistics remains strong (as demonstrated by the fact that much of undergraduate teaching is based on it [19]), Bayesian methods are widely accepted and used, e.g., in the field of machine learning.[20]
Justification
The use of Bayesian probabilities as the basis of Bayesian inference has been supported by several arguments, such as Cox axioms, the Dutch book argument, arguments based on decision theory and de Finetti's theorem.
Axiomatic approach
Dutch book approach
Bruno de Finetti proposed the Dutch book argument based on betting. A clever bookmaker makes a Dutch book by setting the odds and bets to ensure that the bookmaker profits—at the expense of the gamblers—regardless of the outcome of the event (a horse race, for example) on which the gamblers bet. It is associated with probabilities implied by the odds not being coherent.
However, Ian Hacking noted that traditional Dutch book arguments did not specify Bayesian updating: they left open the possibility that non-Bayesian updating rules could avoid Dutch books. For example, Hacking writes[23][24] "And neither the Dutch book argument, nor any other in the personalist arsenal of proofs of the probability axioms, entails the dynamic assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption to be Bayesian. It is true that in consistency a personalist could abandon the Bayesian model of learning from experience. Salt could lose its savour."
In fact, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on "
Decision theory approach
A
Personal probabilities and objective methods for constructing priors
Following the work on
Ramsey and
The "Ramsey test" for evaluating probability distributions is implementable in theory, and has kept experimental psychologists occupied for a half century.[36] This work demonstrates that Bayesian-probability propositions can be falsified, and so meet an empirical criterion of Charles S. Peirce, whose work inspired Ramsey. (This falsifiability-criterion was popularized by Karl Popper.[37][38])
Modern work on the experimental evaluation of personal probabilities uses the randomization,
Personal probabilities are problematic for science and for some applications where decision-makers lack the knowledge or time to specify an informed probability-distribution (on which they are prepared to act). To meet the needs of science and of human limitations, Bayesian statisticians have developed "objective" methods for specifying prior probabilities.
Indeed, some Bayesians have argued the prior state of knowledge defines the (unique) prior probability-distribution for "regular" statistical problems; cf. well-posed problems. Finding the right method for constructing such "objective" priors (for appropriate classes of regular problems) has been the quest of statistical theorists from Laplace to John Maynard Keynes, Harold Jeffreys, and Edwin Thompson Jaynes. These theorists and their successors have suggested several methods for constructing "objective" priors (Unfortunately, it is not always clear how to assess the relative "objectivity" of the priors proposed under these methods):
Each of these methods contributes useful priors for "regular" one-parameter problems, and each prior can handle some challenging
Thus, the Bayesian statistician needs either to use informed priors (using relevant expertise or previous data) or to choose among the competing methods for constructing "objective" priors.
See also
- An Essay towards solving a Problem in the Doctrine of Chances
- Bayesian epistemology
- Bertrand paradox—a paradox in classical probability
- Credal network
- Credence (statistics)
- De Finetti's game—a procedure for evaluating someone's subjective probability
- Evidence under Bayes' theorem
- Monty Hall problem
- interpretation of quantum mechanicsbased on subjective Bayesian probability
- Reference class problem
References
- ^ "Bayesian". Merriam-Webster.com Dictionary.
- .
- ^ CiteSeerX 10.1.1.41.1055.
- ^ ISBN 9781119286370.
- ISBN 0934223459.
- ISBN 1-4020-0223-8.
- ^ Paulos, John Allen (5 August 2011). "The Mathematics of Changing Your Mind [by Sharon Bertsch McGrayne]". Book Review. New York Times. Archived from the original on 2022-01-01. Retrieved 2011-08-06.
- ^ ISBN 9780674403413.
- ^ doi:10.1214/09-BA422.
- ^ ISBN 9780801869822.
- ^ McGrayne, Sharon Bertsch (2011). The Theory that Would not Die. [https://archive.org/details/theorythatwouldn0000mcgr/page/10 10 ], p. 10, at Google Books.
- ISBN 9780674403406.
- ^ doi:10.1214/06-BA101. Archived from the original(PDF) on 10 September 2014.
- Savage, The Foundation of Statistics (1954) are commonly regarded starting points for current Bayesian approaches
- ^ Annals of the Computation Laboratory of Harvard University. Vol. 31. 1962. p. 180.
This revolution, which may or may not succeed, is neo-Bayesianism. Jeffreys tried to introduce this approach, but did not succeed at the time in giving it general appeal.
- ^ Kempthorne, Oscar (1967). The Classical Problem of Inference—Goodness of Fit. Fifth Berkeley Symposium on Mathematical Statistics and Probability. p. 235.
It is curious that even in its activities unrelated to ethics, humanity searches for a religion. At the present time, the religion being 'pushed' the hardest is Bayesianism.
- ISBN 9780444515391.
- .
- ^ Bernardo, José M. (2006). A Bayesian mathematical statistics primer (PDF). ICOTS-7. Bern. Archived (PDF) from the original on 2022-10-09.
- ^ Bishop, C.M. (2007). Pattern Recognition and Machine Learning. Springer.
- ISBN 0-7923-0224-9.
- (PDF) from the original on 2022-10-09.
- ^ Hacking (1967), Section 3, page 316
- ^ Hacking (1988, page 124)
- S2CID 120881078.
- ^ Joyce, James (30 September 2003). "Bayes' Theorem". The Stanford Encyclopedia of Philosophy. stanford.edu.
- S2CID 119215115.
- ISBN 0-19-824860-1.
- ^ Wald, Abraham (1950). Statistical Decision Functions. Wiley.
- ISBN 0-471-92416-4.
- ^ Pfanzagl (1967, 1968)
- ^ Morgenstern (1976, page 65)
- S2CID 170802937.
- ^ ISSN 1755-2567.
- ^ ISBN 9781134445936.
- ^ Davidson et al. (1957)
- ^ Thornton, Stephen (7 August 2018). "Karl Popper". Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University.
- ISBN 0-415-27843-0– via Google Books. (translation of 1935 original, in German).
- ^ Peirce & Jastrow (1885)
- ^ a b Bernardo, J. M. (2005). "Reference Analysis". In Dey, D.K.; Rao, C. R. (eds.). Handbook of Statistics (PDF). Vol. 25. Amsterdam: Elsevier. pp. 17–90. Archived (PDF) from the original on 2022-10-09.
Bibliography
- ISBN 978-0-387-96098-2.
- Bessière, Pierre; Mazer, E.; Ahuacatzin, J.-M.; Mekhnacha, K. (2013). Bayesian Programming. CRC Press. ISBN 9781439880326.
- ISBN 978-0-471-49464-5.
- MR 0443141.
(updated printing, 2007, of Holden-Day, 1976)
- Davidson, Donald; Suppes, Patrick; Siegel, Sidney (1957). Decision-Making: an Experimental Approach. Stanford University Press.
- de Finetti, Bruno (1937). "La Prévision: ses lois logiques, ses sources subjectives" [Foresight: Its logical laws, its subjective sources]. Annales de l'Institut Henri Poincaré (in French). 7 (1): 1–68.
- de Finetti, Bruno (September 1989) [1931]. "Probabilism: A critical essay on the theory of probability and on the value of science". Erkenntnis. 31. (translation of de Finetti, 1931)
- de Finetti, Bruno (1964) [1937]. "Foresight: Its logical laws, its subjective sources". In Kyburg, H.E.; Smokler, H.E. (eds.). Studies in Subjective Probability. New York, NY: Wiley. (translation of de Finetti, 1937, above)
- de Finetti, Bruno (1974–1975) [1970]. Theory of Probability: A critical introductory treatment. Translated by Machi, A.; ISBN 0-471-20142-1, two volumes.
- Goertz, Gary and James Mahoney. 2012. A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences. Princeton University Press.
- ISBN 0-471-68029-X..
- S2CID 14344339.
- (Partly reprinted in ISBN 0-521-33658-9.)
- Hajek, A.; Hartmann, S. (2010) [2001]. "Bayesian Epistemology". In Dancy, J.; Sosa, E.; Steup, M. (eds.). A Companion to Epistemology (PDF). Wiley. ISBN 978-1-4051-3900-7. Archived from the original(PDF) on 2011-07-28.
- ISBN 978-0-471-17912-2.
- Hartmann, S.; Sprenger, J. (2011). "Bayesian Epistemology". In Bernecker, S.; Pritchard, D. (eds.). Routledge Companion to Epistemology (PDF). Routledge. ISBN 978-0-415-96219-3. Archived from the original(PDF) on 2011-07-28.
- "Bayesian approach to statistical problems", Encyclopedia of Mathematics, EMS Press, 2001 [1994]
- ISBN 978-0-8126-9578-6.
- .
- McGrayne, S.B. (2011). The Theory that would not Die: How Bayes' rule cracked the Enigma code, hunted down Russian submarines, and emerged triumphant from two centuries of controversy. New Haven, CT: Yale University Press. OCLC 670481486.
- ISBN 978-0-8147-7771-8.
- Peirce, C.S. & Jastrow J. (1885). "On Small Differences in Sensation". Memoirs of the National Academy of Sciences. 3: 73–83.
- Pfanzagl, J (1967). "Subjective Probability Derived from the Morgenstern-von Neumann Utility Theory". In Martin Shubik (ed.). Essays in Mathematical Economics In Honor of Oskar Morgenstern. Princeton University Press. pp. 237–251.
- Pfanzagl, J.; Baumann, V. & Huber, H. (1968). "Events, Utility and Subjective Probability". Theory of Measurement. Wiley. pp. 195–220.
- (PDF) on 2008-02-27.
- ISBN 978-0-674-40341-3.
- Stigler, S.M. (1999). Statistics on the Table: The history of statistical concepts and methods. Harvard University Press. ISBN 0-674-83601-4.
- Stone, J.V. (2013). Bayes' Rule: A tutorial introduction to Bayesian analysis. England: Sebtel Press. "Chapter 1 of Bayes' Rule".
- Winkler, R.L. (2003). Introduction to Bayesian Inference and Decision (2nd ed.). Probabilistic. ISBN 978-0-9647938-4-2.
Updated classic textbook. Bayesian theory clearly presented