Statistical inference

Whatever level of assumption is made, correctly calibrated inference, in general, requires these assumptions to be correct; i.e. that the data-generating mechanisms really have been correctly specified.

Incorrect assumptions of 'simple' random sampling can invalidate statistical inference.^[10] More complex semi- and fully parametric assumptions are also cause for concern. For example, incorrectly assuming the Cox model can in some cases lead to faulty conclusions.^[11] Incorrect assumptions of Normality in the population also invalidates some forms of regression-based inference.^[12] The use of any parametric model is viewed skeptically by most experts in sampling human populations: "most sampling statisticians, when they deal with confidence intervals at all, limit themselves to statements about [estimators] based on very large samples, where the central limit theorem ensures that these [estimators] will have distributions that are nearly normal."^[13] In particular, a normal distribution "would be a totally unrealistic and catastrophically unwise assumption to make if we were dealing with any kind of economic population."^[13] Here, the central limit theorem states that the distribution of the sample mean "for very large samples" is approximately normally distributed, if the distribution is not heavy-tailed.

Approximate distributions

Given the difficulty in specifying exact distributions of sample statistics, many methods have been developed for approximating these.

With finite samples,

Objective randomization allows properly inductive procedures.[28]^{[29][30][31][32]} Many statisticians prefer randomization-based analysis of data that was generated by well-defined randomization procedures.^[33] (However, it is true that in fields of science with developed theoretical knowledge and experimental control, randomized experiments may increase the costs of experimentation without improving the quality of inferences.^[34][35]) Similarly, results from randomized experiments are recommended by leading statistical authorities as allowing inferences with greater reliability than do observational studies of the same phenomena.^[36] However, a good observational study may be better than a bad randomized experiment.

The statistical analysis of a randomized experiment may be based on the randomization scheme stated in the experimental protocol and does not need a subjective model.^[37][38]

However, at any time, some hypotheses cannot be tested using objective statistical models, which accurately describe randomized experiments or random samples. In some cases, such randomized studies are uneconomical or unethical.

Model-based analysis of randomized experiments

It is standard practice to refer to a statistical model, e.g., a linear or logistic models, when analyzing data from randomized experiments.^[39] However, the randomization scheme guides the choice of a statistical model. It is not possible to choose an appropriate model without knowing the randomization scheme.^[23] Seriously misleading results can be obtained analyzing data from randomized experiments while ignoring the experimental protocol; common mistakes include forgetting the blocking used in an experiment and confusing repeated measurements on the same experimental unit with independent replicates of the treatment applied to different experimental units.^[40]

Model-free randomization inference

Model-free techniques provide a complement to model-based methods, which employ reductionist strategies of reality-simplification. The former combine, evolve, ensemble and train algorithms dynamically adapting to the contextual affinities of a process and learning the intrinsic characteristics of the observations.^[39][41]

For example, model-free simple linear regression is based either on

• a random design, where the pairs of observations ${\displaystyle (X_{1},Y_{1}),(X_{2},Y_{2}),\cdots ,(X_{n},Y_{n})}$ are independent and identically distributed (iid), or
• a deterministic design, where the variables ${\displaystyle X_{1},X_{2},\cdots ,X_{n}}$ are deterministic, but the corresponding response variables ${\displaystyle Y_{1},Y_{2},\cdots ,Y_{n}}$ are random and independent with a common conditional distribution, i.e., ${\displaystyle P\left(Y_{j}\leq y|X_{j}=x\right)=D_{x}(y)}$, which is independent of the index ${\displaystyle j}$.

In either case, the model-free randomization inference for features of the common conditional distribution ${\displaystyle D_{x}(.)}$ relies on some regularity conditions, e.g. functional smoothness. For instance, model-free randomization inference for the population feature conditional mean, ${\displaystyle \mu (x)=E(Y|X=x)}$, can be consistently estimated via local averaging or local polynomial fitting, under the assumption that ${\displaystyle \mu (x)}$ is smooth. Also, relying on asymptotic normality or resampling, we can construct confidence intervals for the population feature, in this case, the conditional mean, ${\displaystyle \mu (x)}$.^[42]

Paradigms for inference

Different schools of statistical inference have become established. These schools—or "paradigms"—are not mutually exclusive, and methods that work well under one paradigm often have attractive interpretations under other paradigms.

Bandyopadhyay & Forster describe four paradigms: The classical (or

This paradigm calibrates the plausibility of propositions by considering (notional) repeated sampling of a population distribution to produce datasets similar to the one at hand. By considering the dataset's characteristics under repeated sampling, the frequentist properties of a statistical proposition can be quantified—although in practice this quantification may be challenging.

Examples of frequentist inference

• p-value
• Confidence interval
• Null hypothesis significance testing

Frequentist inference, objectivity, and decision theory

One interpretation of

The Bayesian calculus describes degrees of belief using the 'language' of probability; beliefs are positive, integrate into one, and obey probability axioms. Bayesian inference uses the available posterior beliefs as the basis for making statistical propositions.^[47] There are several different justifications for using the Bayesian approach.

Examples of Bayesian inference

• Credible interval for interval estimation
• Bayes factors for model comparison

Bayesian inference, subjectivity and decision theory

Many informal Bayesian inferences are based on "intuitively reasonable" summaries of the posterior. For example, the posterior mean, median and mode, highest posterior density intervals, and Bayes Factors can all be motivated in this way. While a user's

Formally, Bayesian inference is calibrated with reference to an explicitly stated utility, or loss function; the 'Bayes rule' is the one which maximizes expected utility, averaged over the posterior uncertainty. Formal Bayesian inference therefore automatically provides optimal decisions in a decision theoretic sense. Given assumptions, data and utility, Bayesian inference can be made for essentially any problem, although not every statistical inference need have a Bayesian interpretation. Analyses which are not formally Bayesian can be (logically) incoherent; a feature of Bayesian procedures which use proper priors (i.e. those integrable to one) is that they are guaranteed to be coherent. Some advocates of Bayesian inference assert that inference must take place in this decision-theoretic framework, and that Bayesian inference should not conclude with the evaluation and summarization of posterior beliefs.

Likelihood-based inference

Main article:

Likelihoodism approaches statistics by using the likelihood function

, denoted as ${\displaystyle L(\theta |x)}$, quantifies the probability of observing the given data ${\displaystyle x}$, assuming a specific set of parameter values ${\displaystyle \theta }$. In likelihood-based inference, the goal is to find the set of parameter values that maximizes the likelihood function, or equivalently, maximizes the probability of observing the given data.

The process of likelihood-based inference usually involves the following steps:

1. Formulating the statistical model: A statistical model is defined based on the problem at hand, specifying the distributional assumptions and the relationship between the observed data and the unknown parameters. The model can be simple, such as a normal distribution with known variance, or complex, such as a hierarchical model with multiple levels of random effects.
2. Constructing the likelihood function: Given the statistical model, the likelihood function is constructed by evaluating the joint probability density or mass function of the observed data as a function of the unknown parameters. This function represents the probability of observing the data for different values of the parameters.
3. Maximizing the likelihood function: The next step is to find the set of parameter values that maximizes the likelihood function. This can be achieved using optimization techniques such as numerical optimization algorithms. The estimated parameter values, often denoted as ${\displaystyle {\bar {y}}}$, are the maximum likelihood estimates (MLEs).
4. Assessing uncertainty: Once the MLEs are obtained, it is crucial to quantify the uncertainty associated with the parameter estimates. This can be done by calculating
   hypothesis tests based on asymptotic theory or simulation techniques such as bootstrapping
   .
5. Model checking: After obtaining the parameter estimates and assessing their uncertainty, it is important to assess the adequacy of the statistical model. This involves checking the assumptions made in the model and evaluating the fit of the model to the data using goodness-of-fit tests, residual analysis, or graphical diagnostics.
6. Inference and interpretation: Finally, based on the estimated parameters and model assessment, statistical inference can be performed. This involves drawing conclusions about the population parameters, making predictions, or testing hypotheses based on the estimated model.

AIC-based inference

Main article: Akaike information criterion

This section needs expansion. You can help by adding to it. (November 2017)

The Akaike information criterion (AIC) is an estimator of the relative quality of statistical models for a given set of data. Given a collection of models for the data, AIC estimates the quality of each model, relative to each of the other models. Thus, AIC provides a means for model selection.

AIC is founded on information theory: it offers an estimate of the relative information lost when a given model is used to represent the process that generated the data. (In doing so, it deals with the trade-off between the goodness of fit of the model and the simplicity of the model.)

Other paradigms for inference

Minimum description length

Main article: Minimum description length

The minimum description length (MDL) principle has been developed from ideas in

probability models

for the data, as might be done in frequentist or Bayesian approaches.

However, if a "data generating mechanism" does exist in reality, then according to

source coding theorem it provides the MDL description of the data, on average and asymptotically.^[50] In minimizing description length (or descriptive complexity), MDL estimation is similar to maximum likelihood estimation and maximum a posteriori estimation (using maximum-entropy Bayesian priors). However, MDL avoids assuming that the underlying probability model is known; the MDL principle can also be applied without assumptions that e.g. the data arose from independent sampling.^[50][51]

The MDL principle has been applied in communication-coding theory in information theory, in linear regression,^[51] and in data mining.^[49]

The evaluation of MDL-based inferential procedures often uses techniques or criteria from computational complexity theory.^[52]

Fiducial inference

Main article: Fiducial inference

fiducial argument as a special case of an inference theory using upper and lower probabilities.^[56]

Structural inference

Developing ideas of Fisher and of Pitman from 1938 to 1939,

George A. Barnard developed "structural inference" or "pivotal inference",^[58] an approach using invariant probabilities on group families. Barnard reformulated the arguments behind fiducial inference on a restricted class of models on which "fiducial" procedures would be well-defined and useful. Donald A. S. Fraser developed a general theory for structural inference^[59] based on group theory and applied this to linear models.^[60] The theory formulated by Fraser has close links to decision theory and Bayesian statistics and can provide optimal frequentist decision rules if they exist.^[61]

Inference topics

The topics below are usually included in the area of statistical inference.

1. Statistical assumptions
2. Statistical decision theory
3. Estimation theory
4. Statistical hypothesis testing
5. Revising opinions in statistics
6. Design of experiments, the analysis of variance, and regression
7. Survey sampling
8. Summarizing statistical data

Predictive inference

Predictive inference is an approach to statistical inference that emphasizes the prediction of future observations based on past observations.

Initially, predictive inference was based on observable parameters and it was the main purpose of studying

exchangeability—that future observations should behave like past observations—came to the attention of the English-speaking world with the 1974 translation from French of his 1937 paper,^[62] and has since been propounded by such statisticians as Seymour Geisser.^[63]

See also

• Algorithmic inference
• Induction (philosophy)
• Informal inferential reasoning
• Information field theory
• Population proportion
• Philosophy of statistics
• Prediction interval
• Predictive analytics
• Predictive modelling
• Stylometry

Notes

1. ^ According to Peirce, acceptance means that inquiry on this question ceases for the time being. In science, all scientific theories are revisable.

References

Citations

1. ISBN 978-0-19-954145-4
   .
2. ^ "TensorFlow Lite inference". The term inference refers to the process of executing a TensorFlow Lite model on-device in order to make predictions based on input data.
3. ^ Johnson, Richard (12 March 2016). "Statistical Inference". Encyclopedia of Mathematics. Springer: The European Mathematical Society. Retrieved 26 October 2022.
4. ^ Konishi & Kitagawa (2008), p. 75.
5. ^ Cox (2006), p. 197.
6. ^ "Statistical inference - Encyclopedia of Mathematics". www.encyclopediaofmath.org. Retrieved 2019-01-23.
7. ^ ^{a b} Cox (2006) page 2
8. ISBN 9780716747420
   .
9. ISBN 0-521-78450-6
   (page 341)
10. ^ Kruskal 1988
11. ^ Freedman, D.A. (2008) "Survival analysis: An Epidemiological hazard?". The American Statistician (2008) 62: 110-119. (Reprinted as Chapter 11 (pages 169–192) of Freedman (2010)).
12. ISBN 0-7619-2904-5
13. ^
    ISBN 978-0340692295
    .
14. ^ ^{a b} Jörgen Hoffman-Jörgensen's Probability With a View Towards Statistics, Volume I. Page 399 ^{[full citation needed]}
15. ^ Le Cam (1986) ^{[page needed]}
16. ^ Erik Torgerson (1991) Comparison of Statistical Experiments, volume 36 of Encyclopedia of Mathematics. Cambridge University Press. ^{[full citation needed]}
17. ISBN 978-0-387-73193-3
    .
18. ^ Kolmogorov (1963, p.369): "The frequency concept, based on the notion of limiting frequency as the number of trials increases to infinity, does not contribute anything to substantiate the applicability of the results of probability theory to real practical problems where we have always to deal with a finite number of trials".
19. ^ "Indeed, limit theorems 'as ${\displaystyle n}$ tends to infinity' are logically devoid of content about what happens at any particular ${\displaystyle n}$. All they can do is suggest certain approaches whose performance must then be checked on the case at hand." — Le Cam (1986) (page xiv)
20. ^ Pfanzagl (1994): "The crucial drawback of asymptotic theory: What we expect from asymptotic theory are results which hold approximately . . . . What asymptotic theory has to offer are limit theorems."(page ix) "What counts for applications are approximations, not limits." (page 188)
21. ^ Pfanzagl (1994) : "By taking a limit theorem as being approximately true for large sample sizes, we commit an error the size of which is unknown. [. . .] Realistic information about the remaining errors may be obtained by simulations." (page ix)
22. JSTOR 2342192
23. ^ ^{a b} Hinkelmann and Kempthorne(2008) ^{[page needed]}
24. ^ ASA Guidelines for the first course in statistics for non-statisticians. (available at the ASA website)
25. ^ David A. Freedman et alia's Statistics.
26. ^ Moore et al. (2015).
27. ^ Gelman A. et al. (2013). Bayesian Data Analysis (Chapman & Hall).
28. ^ Peirce (1877-1878)
29. ^ Peirce (1883)
30. ^ Freedman, Pisani & Purves 1978.
31. ^ David A. Freedman Statistical Models.
32. ^
    ISBN 981-02-3111-3
33. ^ Peirce; Freedman; Moore et al. (2015).^{[citation needed]}
34. ISBN 978-0-471-72755-2
35. ^ Cox (2006), p. 196.
36. ^ ASA Guidelines for the first course in statistics for non-statisticians. (available at the ASA website)
    • David A. Freedman et alias Statistics.
    • Moore et al. (2015).
37. ^ Neyman, Jerzy. 1923 [1990]. "On the Application of Probability Theory to AgriculturalExperiments. Essay on Principles. Section 9." Statistical Science 5 (4): 465–472. Trans. Dorota M. Dabrowska and Terence P. Speed.
38. ^ Hinkelmann & Kempthorne (2008) ^{[page needed]}
39. ^ ^{a b} Dinov, Ivo; Palanimalai, Selvam; Khare, Ashwini; Christou, Nicolas (2018). "Randomization-based statistical inference: A resampling and simulation infrastructure". Teaching Statistics. 40 (2): 64–73.
    PMID 30270947
    .
40. ^ Hinkelmann and Kempthorne (2008) Chapter 6.
41. ^ Tang, Ming; Gao, Chao; Goutman, Stephen; Kalinin, Alexandr; Mukherjee, Bhramar; Guan, Yuanfang; Dinov, Ivo (2019). "Model-Based and Model-Free Techniques for Amyotrophic Lateral Sclerosis Diagnostic Prediction and Patient Clustering". Neuroinformatics. 17 (3): 407–421.
    PMID 30460455
    .
42. ^ Politis, D.N. (2019). "Model-free inference in statistics: how and why". IMS Bulletin. 48.
43. ^ Bandyopadhyay & Forster (2011). See the book's Introduction (p.3) and "Section III: Four Paradigms of Statistics".
44. JSTOR 91337
    .
45. ^ Preface to Pfanzagl.
46. S2CID 53505632
    .
47. S2CID 220935477
    .
48. ^ Soofi (2000)
49. ^ ^{a b} Hansen & Yu (2001)
50. ^ ^{a b} Hansen and Yu (2001), page 747.
51. ^ ^{a b} Rissanen (1989), page 84
52. ^ Joseph F. Traub, G. W. Wasilkowski, and H. Wozniakowski. (1988) ^{[page needed]}
53. ^ Neyman (1956)
54. ^ Zabell (1992)
55. ^ Cox (2006) page 66
56. ^ Hampel 2003.
57. ^ Davison, page 12. ^{[full citation needed]}
58. JSTOR 1403482
59. OCLC 440926
    .
60. OCLC 3559629
    .
61. S2CID 88520957
    .
62. ISBN 978-0-387-94037-3
    .
63. ISBN 0-412-03471-9

Sources

• Bandyopadhyay, P. S.; Forster, M. R., eds. (2011), Philosophy of Statistics, Elsevier.
• Bickel, Peter J.; Doksum, Kjell A. (2001). Mathematical statistics: Basic and selected topics. Vol. 1 (Second (updated printing 2007) ed.).
  MR 0443141
  .
• ISBN 0-521-68567-2
  .
• Fisher, R. A. (1955), "Statistical methods and scientific induction", Journal of the Royal Statistical Society, Series B, 17, 69–78. (criticism of statistical theories of Jerzy Neyman and Abraham Wald
  )
• MR 2489600
  .
• Freedman, D. A. (2010). Statistical Models and Causal Inferences: A Dialogue with the Social Sciences (Edited by David Collier, Jasjeet Sekhon, and Philip B. Stark), Cambridge University Press.
• Hampel, Frank R. (February 2003). "The proper fiducial argument". Seminar für Statistik, Eidgenössische Technische Hochschule (ETH). 114.
  doi:10.3929/ethz-a-004526011
  .
• Hansen, Mark H.;
  S2CID 14460386. Archived from the original
  on 2004-11-16.
• Hinkelmann, Klaus;
  ISBN 978-0-471-72756-9
  .
• MR 1643414
  .
• Konishi S., Kitagawa G. (2008), Information Criteria and Statistical Modeling, Springer.
• JSTOR 2290117
  .
• ISBN 0-387-96307-3
• Moore, D. S.; McCabe, G. P.; Craig, B. A. (2015), Introduction to the Practice of Statistics, Eighth Edition, Macmillan.
• JSTOR 2983716
  . (reply to Fisher 1955)
• Popular Science Monthly
  , vols. 12–13. Relevant individual papers:
  • (1878 March), "The Doctrine of Chances", Popular Science Monthly, v. 12, March issue, pp. 604–615. Internet Archive Eprint.
  • (1878 April), "The Probability of Induction", Popular Science Monthly, v. 12, pp. 705–718. Internet Archive Eprint.
  • (1878 June), "The Order of Nature", Popular Science Monthly, v. 13, pp. 203–217.Internet Archive Eprint.
  • (1878 August), "Deduction, Induction, and Hypothesis", Popular Science Monthly, v. 13, pp. 470–482. Internet Archive Eprint.
• ISBN 90-272-3271-7
  )
• Freedman, D.A; Pisani, R.; Purves, R.A. (1978). Statistics. New York: W. W. Norton & Company.
• Pfanzagl, Johann; with the assistance of R. Hamböker (1994). Parametric Statistical Theory. Berlin:
  MR 1291393
  .
• MR 1082556
  .
• Soofi, Ehsan S. (December 2000). "Principal information-theoretic approaches (Vignettes for the Year 2000: Theory and Methods, ed. by George Casella)".
  S2CID 120143121
  .
• ISBN 978-0-12-697545-1
  .
• Zabell, S. L. (Aug 1992). "R. A. Fisher and Fiducial Argument". Statistical Science. 7 (3): 369–387.
  JSTOR 2246073
  .

Further reading

• ISBN 0-534-24312-6
• JSTOR 270939
  .
• Held L., Bové D.S. (2014). Applied Statistical Inference—Likelihood and Bayes (Springer).
• Lenhard, Johannes (2006). "Models and Statistical Inference: the controversy between Fisher and Neyman–Pearson" (PDF).
  S2CID 14136146
  .
• Lindley, D (1958). "Fiducial distribution and Bayes' theorem". Journal of the Royal Statistical Society, Series B. 20: 102–7.
• Rahlf, Thomas (2014). "Statistical Inference", in Claude Diebolt, and Michael Haupert (eds.), "Handbook of Cliometrics ( Springer Reference Series)", Berlin/Heidelberg: Springer.
• Reid, N.; Cox, D. R. (2014). "On Some Principles of Statistical Inference". International Statistical Review. 83 (2): 293–308.
  S2CID 17410547
  .
• Sagitov, Serik (2022). "Statistical Inference". Wikibooks. http://upload.wikimedia.org/wikipedia/commons/f/f9/Statistical_Inference.pdf
• Young, G.A., Smith, R.L. (2005). Essentials of Statistical Inference, CUP.
  ISBN 0-521-83971-8

External links

Wikimedia Commons has media related to Statistical inference.

Wikiversity has learning resources about Statistical inference

• Statistical Inference – lecture on the MIT OpenCourseWare platform
• Statistical Inference – lecture by the National Programme on Technology Enhanced Learning
• An online, Bayesian (MCMC) demo/calculator is available at causaScientia