Errors and residuals

This article includes a list of general references, but it lacks sufficient corresponding inline citations. Please help to improve this article by introducing more precise citations. (September 2016) (Learn how and when to remove this template message)

In

Introduction

Suppose there is a series of observations from a

A statistical error (or disturbance) is the amount by which an observation differs from its expected value, the latter being based on the whole population from which the statistical unit was chosen randomly. For example, if the mean height in a population of 21-year-old men is 1.75 meters, and one randomly chosen man is 1.80 meters tall, then the "error" is 0.05 meters; if the randomly chosen man is 1.70 meters tall, then the "error" is −0.05 meters. The expected value, being the mean of the entire population, is typically unobservable, and hence the statistical error cannot be observed either.

A residual (or fitting deviation), on the other hand, is an observable estimate of the unobservable statistical error. Consider the previous example with men's heights and suppose we have a random sample of n people. The

• The difference between the height of each man in the sample and the unobservable population mean is a statistical error, whereas
• The difference between the height of each man in the sample and the observable sample mean is a residual.

Note that, because of the definition of the sample mean, the sum of the residuals within a random sample is necessarily zero, and thus the residuals are necessarily not

If we assume a normally distributed population with mean μ and standard deviation σ, and choose individuals independently, then we have

${\displaystyle X_{1},\dots ,X_{n}\sim N\left(\mu ,\sigma ^{2}\right)\,}$

and the sample mean

${\displaystyle {\overline {X}}={X_{1}+\cdots +X_{n} \over n}}$

is a random variable distributed such that:

${\displaystyle {\overline {X}}\sim N\left(\mu ,{\frac {\sigma ^{2}}{n}}\right).}$

The statistical errors are then

${\displaystyle e_{i}=X_{i}-\mu ,\,}$

with

The sum of squares of the statistical errors, divided by σ², has a chi-squared distribution with n degrees of freedom:

${\displaystyle {\frac {1}{\sigma ^{2}}}\sum _{i=1}^{n}e_{i}^{2}\sim \chi _{n}^{2}.}$

However, this quantity is not observable as the population mean is unknown. The sum of squares of the residuals, on the other hand, is observable. The quotient of that sum by σ² has a chi-squared distribution with only n − 1 degrees of freedom:

${\displaystyle {\frac {1}{\sigma ^{2}}}\sum _{i=1}^{n}r_{i}^{2}\sim \chi _{n-1}^{2}.}$

This difference between n and n − 1 degrees of freedom results in

Remark

It is remarkable that the

where ${\displaystyle {\overline {X}}_{n}-\mu _{0}}$ represents the errors, ${\displaystyle S_{n}}$ represents the sample standard deviation for a sample of size n, and unknown σ, and the denominator term ${\displaystyle S_{n}/{\sqrt {n}}}$ accounts for the standard deviation of the errors according to:[5]

$${\displaystyle \operatorname {Var} \left({\overline {X}}_{n}\right)={\frac {\sigma ^{2}}{n}}}$$

The probability distributions of the numerator and the denominator separately depend on the value of the unobservable population standard deviation σ, but σ appears in both the numerator and the denominator and cancels. That is fortunate because it means that even though we do not know σ, we know the probability distribution of this quotient: it has a Student's t-distribution with n − 1 degrees of freedom. We can therefore use this quotient to find a confidence interval for μ. This t-statistic can be interpreted as "the number of standard errors away from the regression line."^[6]

Regressions

In

However, a terminological difference arises in the expression mean squared error (MSE). The mean squared error of a regression is a number computed from the sum of squares of the computed residuals, and not of the unobservable errors. If that sum of squares is divided by n, the number of observations, the result is the mean of the squared residuals. Since this is a biased estimate of the variance of the unobserved errors, the bias is removed by dividing the sum of the squared residuals by df = n − p − 1, instead of n, where df is the number of degrees of freedom (n minus the number of parameters (excluding the intercept) p being estimated - 1). This forms an unbiased estimate of the variance of the unobserved errors, and is called the mean squared error.^[7]

Another method to calculate the mean square of error when analyzing the variance of linear regression using a technique like that used in

The use of the term "error" as discussed in the sections above is in the sense of a deviation of a value from a hypothetical unobserved value. At least two other uses also occur in statistics, both referring to observable

The

References