Pearson correlation coefficient
In
Naming and history
It was developed by
Definition
Pearson's correlation coefficient is the covariance of the two variables divided by the product of their standard deviations. The form of the definition involves a "product moment", that is, the mean (the first moment about the origin) of the product of the mean-adjusted random variables; hence the modifier product-moment in the name.[verification needed]
For a population
Pearson's correlation coefficient, when applied to a population, is commonly represented by the Greek letter ρ (rho) and may be referred to as the population correlation coefficient or the population Pearson correlation coefficient. Given a pair of random variables (for example, Height and Weight), the formula for ρ[10] is[11]
where
- is the covariance
- is the standard deviation of
- is the standard deviation of .
The formula for can be expressed in terms of
the formula for can also be written as
where
- and are defined as above
- is the mean of
- is the mean of
- is the expectation.
The formula for can be expressed in terms of uncentered moments. Since
the formula for can also be written as
For a sample
Pearson's correlation coefficient, when applied to a
where
- is sample size
- are the individual sample points indexed with i
- (the sample mean); and analogously for .
Rearranging gives us this formula for :
where are defined as above.
This formula suggests a convenient single-pass algorithm for calculating sample correlations, though depending on the numbers involved, it can sometimes be numerically unstable.
Rearranging again gives us this[10] formula for :
where are defined as above.
An equivalent expression gives the formula for as the mean of the products of the standard scores as follows:
where
- are defined as above, and are defined below
- is the standard score (and analogously for the standard score of ).
Alternative formulae for are also available. For example, one can use the following formula for :
where
- are defined as above and:
- (the sample standard deviation); and analogously for .
For jointly gaussian distributions
If is
Practical issues
Under heavy noise conditions, extracting the correlation coefficient between two sets of
In case of missing data, Garren derived the
Some distributions (e.g., stable distributions other than a normal distribution) do not have a defined variance.
Mathematical properties
The values of both the sample and population Pearson correlation coefficients are on or between −1 and 1. Correlations equal to +1 or −1 correspond to data points lying exactly on a line (in the case of the sample correlation), or to a bivariate distribution entirely supported on a line (in the case of the population correlation). The Pearson correlation coefficient is symmetric: corr(X,Y) = corr(Y,X).
A key mathematical property of the Pearson correlation coefficient is that it is invariant under separate changes in location and scale in the two variables. That is, we may transform X to a + bX and transform Y to c + dY, where a, b, c, and d are constants with b, d > 0, without changing the correlation coefficient. (This holds for both the population and sample Pearson correlation coefficients.) More general linear transformations do change the correlation: see § Decorrelation of n random variables for an application of this.
Interpretation
The correlation coefficient ranges from −1 to 1. An absolute value of exactly 1 implies that a linear equation describes the relationship between X and Y perfectly, with all data points lying on a
More generally, (Xi − X)(Yi − Y) is positive if and only if Xi and Yi lie on the same side of their respective means. Thus the correlation coefficient is positive if Xi and Yi tend to be simultaneously greater than, or simultaneously less than, their respective means. The correlation coefficient is negative (
Rodgers and Nicewander[16] cataloged thirteen ways of interpreting correlation or simple functions of it:
- Function of raw scores and means
- Standardized covariance
- Standardized slope of the regression line
- Geometric mean of the two regression slopes
- Square root of the ratio of two variances
- Mean cross-product of standardized variables
- Function of the angle between two standardized regression lines
- Function of the angle between two variable vectors
- Rescaled variance of the difference between standardized scores
- Estimated from the balloon rule
- Related to the bivariate ellipses of isoconcentration
- Function of test statistics from designed experiments
- Ratio of two means
Geometric interpretation
For uncentered data, there is a relation between the correlation coefficient and the angle φ between the two regression lines, y = gX(x) and x = gY(y), obtained by regressing y on x and x on y respectively. (Here, φ is measured counterclockwise within the first quadrant formed around the lines' intersection point if r > 0, or counterclockwise from the fourth to the second quadrant if r < 0.) One can show[17] that if the standard deviations are equal, then r = sec φ − tan φ, where sec and tan are trigonometric functions.
For centered data (i.e., data which have been shifted by the sample means of their respective variables so as to have an average of zero for each variable), the correlation coefficient can also be viewed as the
Both the uncentered (non-Pearson-compliant) and centered correlation coefficients can be determined for a dataset. As an example, suppose five countries are found to have gross national products of 1, 2, 3, 5, and 8 billion dollars, respectively. Suppose these same five countries (in the same order) are found to have 11%, 12%, 13%, 15%, and 18% poverty. Then let x and y be ordered 5-element vectors containing the above data: x = (1, 2, 3, 5, 8) and y = (0.11, 0.12, 0.13, 0.15, 0.18).
By the usual procedure for finding the angle θ between two vectors (see dot product), the uncentered correlation coefficient is
This uncentered correlation coefficient is identical with the cosine similarity. The above data were deliberately chosen to be perfectly correlated: y = 0.10 + 0.01 x. The Pearson correlation coefficient must therefore be exactly one. Centering the data (shifting x by ℰ(x) = 3.8 and y by ℰ(y) = 0.138) yields x = (−2.8, −1.8, −0.8, 1.2, 4.2) and y = (−0.028, −0.018, −0.008, 0.012, 0.042), from which
as expected.
Interpretation of the size of a correlation
Several authors have offered guidelines for the interpretation of a correlation coefficient.[19][20] However, all such criteria are in some ways arbitrary.[20] The interpretation of a correlation coefficient depends on the context and purposes. A correlation of 0.8 may be very low if one is verifying a physical law using high-quality instruments, but may be regarded as very high in the social sciences, where there may be a greater contribution from complicating factors.
Inference
Statistical inference based on Pearson's correlation coefficient often focuses on one of the following two aims:
- One aim is to test the null hypothesis that the true correlation coefficient ρ is equal to 0, based on the value of the sample correlation coefficient r.
- The other aim is to derive a confidence interval that, on repeated sampling, has a given probability of containing ρ.
Methods of achieving one or both of these aims are discussed below.
Using a permutation test
Permutation tests provide a direct approach to performing hypothesis tests and constructing confidence intervals. A permutation test for Pearson's correlation coefficient involves the following two steps:
- Using the original paired data (xi, yi), randomly redefine the pairs to create a new data set (xi, yi′), where the i′ are a permutation of the set {1,...,n}. The permutation i′ is selected randomly, with equal probabilities placed on all n! possible permutations. This is equivalent to drawing the i′ randomly without replacement from the set {1, ..., n}. In bootstrapping, a closely related approach, the i and the i′ are equal and drawn with replacement from {1, ..., n};
- Construct a correlation coefficient r from the randomized data.
To perform the permutation test, repeat steps (1) and (2) a large number of times. The
Using a bootstrap
The bootstrap can be used to construct confidence intervals for Pearson's correlation coefficient. In the "non-parametric" bootstrap, n pairs (xi, yi) are resampled "with replacement" from the observed set of n pairs, and the correlation coefficient r is calculated based on the resampled data. This process is repeated a large number of times, and the empirical distribution of the resampled r values are used to approximate the sampling distribution of the statistic. A 95% confidence interval for ρ can be defined as the interval spanning from the 2.5th to the 97.5th percentile of the resampled r values.
Standard error
If and are random variables, a standard error associated to the correlation in the null case is
where is the correlation (assumed r≈0) and the sample size.[21][22]
Testing using Student's t-distribution
has a student's t-distribution in the null case (zero correlation).[23] This holds approximately in case of non-normal observed values if sample sizes are large enough.[24] For determining the critical values for r the inverse function is needed:
Alternatively, large sample, asymptotic approaches can be used.
Another early paper[25] provides graphs and tables for general values of ρ, for small sample sizes, and discusses computational approaches.
In the case where the underlying variables are not normal, the sampling distribution of Pearson's correlation coefficient follows a Student's t-distribution, but the degrees of freedom are reduced.[26]
Using the exact distribution
For data that follow a
where is the gamma function and is the Gaussian hypergeometric function.
In the special case when (zero population correlation), the exact density function f(r) can be written as
where is the
Using the exact confidence distribution
Confidence intervals and tests can be calculated from a confidence distribution. An exact confidence density for ρ is[30]
where is the Gaussian hypergeometric function and .
Using the Fisher transformation
In practice,
F(r) approximately follows a normal distribution with
- and standard error
where n is the sample size. The approximation error is lowest for a large sample size and small and and increases otherwise.
Using the approximation, a z-score is
under the null hypothesis that , given the assumption that the sample pairs are
To obtain a confidence interval for ρ, we first compute a confidence interval for F():
The inverse Fisher transformation brings the interval back to the correlation scale.
For example, suppose we observe r = 0.7 with a sample size of n=50, and we wish to obtain a 95% confidence interval for ρ. The transformed value is , so the confidence interval on the transformed scale is , or (0.5814, 1.1532). Converting back to the correlation scale yields (0.5237, 0.8188).
In least squares regression analysis
The square of the sample correlation coefficient is typically denoted r2 and is a special case of the coefficient of determination. In this case, it estimates the fraction of the variance in Y that is explained by X in a simple linear regression. So if we have the observed dataset and the fitted dataset then as a starting point the total variation in the Yi around their average value can be decomposed as follows
where the are the fitted values from the regression analysis. This can be rearranged to give
The two summands above are the fraction of variance in Y that is explained by X (right) and that is unexplained by X (left).
Next, we apply a property of least square regression models, that the sample covariance between and is zero. Thus, the sample correlation coefficient between the observed and fitted response values in the regression can be written (calculation is under expectation, assumes Gaussian statistics)
Thus
where is the proportion of variance in Y explained by a linear function of X.
In the derivation above, the fact that
can be proved by noticing that the partial derivatives of the residual sum of squares (RSS) over β0 and β1 are equal to 0 in the least squares model, where
- .
In the end, the equation can be written as
where
- .
The symbol is called the regression sum of squares, also called the explained sum of squares, and is the total sum of squares (proportional to the variance of the data).
Sensitivity to the data distribution
Existence
The population Pearson correlation coefficient is defined in terms of
Sample size
- If the sample size is moderate or large and the population is normal, then, in the case of the bivariate maximum likelihood estimate of the population correlation coefficient, and is asymptotically unbiased and efficient, which roughly means that it is impossible to construct a more accurate estimate than the sample correlation coefficient.
- If the sample size is large and the population is not normal, then the sample correlation coefficient remains approximately unbiased, but may not be efficient.
- If the sample size is large, then the sample correlation coefficient is a consistent estimator of the population correlation coefficient as long as the sample means, variances, and covariance are consistent (which is guaranteed when the law of large numbers can be applied).
- If the sample size is small, then the sample correlation coefficient r is not an unbiased estimate of ρ.[10] The adjusted correlation coefficient must be used instead: see elsewhere in this article for the definition.
- Correlations can be different for imbalanced dichotomous data when there is variance error in sample.[31]
Robustness
Like many commonly used statistics, the sample
Statistical inference for Pearson's correlation coefficient is sensitive to the data distribution. Exact tests, and asymptotic tests based on the
A stratified analysis is one way to either accommodate a lack of bivariate normality, or to isolate the correlation resulting from one factor while controlling for another. If W represents cluster membership or another factor that it is desirable to control, we can stratify the data based on the value of W, then calculate a correlation coefficient within each stratum. The stratum-level estimates can then be combined to estimate the overall correlation while controlling for W.[36]
Variants
Variations of the correlation coefficient can be calculated for different purposes. Here are some examples.
Adjusted correlation coefficient
The sample correlation coefficient r is not an unbiased estimate of ρ. For data that follows a
- therefore r is a biased estimator of
The unique minimum variance unbiased estimator radj is given by[38]
-
(1)
where:
- are defined as above,
- is the Gaussian hypergeometric function.
An approximately unbiased estimator radj can be obtained[citation needed] by truncating E[r] and solving this truncated equation:
-
(2)
An approximate solution[citation needed] to equation (2) is
-
(3)
where in (3)
- are defined as above,
- radj is a suboptimal estimator,[citation needed][clarification needed]
- radj can also be obtained by maximizing log(f(r)),
- radj has minimum variance for large values of n,
- radj has a bias of order 1⁄(n − 1).
Another proposed[10] adjusted correlation coefficient is[citation needed]
radj ≈ r for large values of n.
Weighted correlation coefficient
Suppose observations to be correlated have differing degrees of importance that can be expressed with a weight vector w. To calculate the correlation between vectors x and y with the weight vector w (all of length n),[39][40]
- Weighted mean:
- Weighted covariance
- Weighted correlation
Reflective correlation coefficient
The reflective correlation is a variant of Pearson's correlation in which the data are not centered around their mean values.[citation needed] The population reflective correlation is
The reflective correlation is symmetric, but it is not invariant under translation:
The sample reflective correlation is equivalent to cosine similarity:
The weighted version of the sample reflective correlation is
Scaled correlation coefficient
Scaled correlation is a variant of Pearson's correlation in which the range of the data is restricted intentionally and in a controlled manner to reveal correlations between fast components in time series.[41] Scaled correlation is defined as average correlation across short segments of data.
Let be the number of segments that can fit into the total length of the signal for a given scale :
The scaled correlation across the entire signals is then computed as
where is Pearson's coefficient of correlation for segment .
By choosing the parameter , the range of values is reduced and the correlations on long time scale are filtered out, only the correlations on short time scales being revealed. Thus, the contributions of slow components are removed and those of fast components are retained.
Pearson's distance
A distance metric for two variables X and Y known as Pearson's distance can be defined from their correlation coefficient as[42]
Considering that the Pearson correlation coefficient falls between [−1, +1], the Pearson distance lies in [0, 2]. The Pearson distance has been used in cluster analysis and data detection for communications and storage with unknown gain and offset.[43]
The Pearson "distance" defined this way assigns distance greater than 1 to negative correlations. In reality, both strong positive correlation and negative correlations are meaningful, so care must be taken when Pearson "distance" is used for nearest neighbor algorithm as such algorithm will only include neighbors with positive correlation and exclude neighbors with negative correlation. Alternatively, an absolute valued distance, , can be applied, which will take both positive and negative correlations into consideration. The information on positive and negative association can be extracted separately, later.
Circular correlation coefficient
For variables X = {x1,...,xn} and Y = {y1,...,yn} that are defined on the unit circle [0, 2π), it is possible to define a circular analog of Pearson's coefficient.
where and are the
Partial correlation
If a population or data-set is characterized by more than two variables, a partial correlation coefficient measures the strength of dependence between a pair of variables that is not accounted for by the way in which they both change in response to variations in a selected subset of the other variables.
Decorrelation of n random variables
It is always possible to remove the correlations between all pairs of an arbitrary number of random variables by using a data transformation, even if the relationship between the variables is nonlinear. A presentation of this result for population distributions is given by Cox & Hinkley.[45]
A corresponding result exists for reducing the sample correlations to zero. Suppose a vector of n random variables is observed m times. Let X be a matrix where is the jth variable of observation i. Let be an m by m square matrix with every element 1. Then D is the data transformed so every random variable has zero mean, and T is the data transformed so all variables have zero mean and zero correlation with all other variables – the sample
where an exponent of −+1⁄2 represents the
This decorrelation is related to
Software implementations
- R's statistics base-package implements the correlation coefficient with
cor(x, y)
, or (with the P value also) withcor.test(x, y)
. - The SciPy Python library via
pearsonr(x, y)
. - The Pandas Python library implements Pearson correlation coefficient calculation as the default option for the method
pandas.DataFrame.corr
- Wolfram Mathematica via the
Correlation
function, or (with the P value) withCorrelationTest
. - The Boost C++ library via the
correlation_coefficient
function. - Excel has an in-builtfunction for calculating the pearson's correlation coefficient.
correl(array1, array2)
See also
- Anscombe's quartet
- Association (statistics)
- Coefficient of colligation
- Yule's Q
- Yule's Y
- Coefficient of multiple correlation
- Concordance correlation coefficient
- Correlation and dependence
- Correlation ratio
- Disattenuation
- Distance correlation
- Maximal information coefficient
- Multiple correlation
- Normally distributed and uncorrelated does not imply independent
- Odds ratio
- Partial correlation
- Polychoric correlation
- Quadrant count ratio
- RV coefficient
- Spearman's rank correlation coefficient
Footnotes
References
- ^ "SPSS Tutorials: Pearson Correlation".
- ^ "Correlation Coefficient: Simple Definition, Formula, Easy Steps". Statistics How To.
- S2CID 4136393. In the "Appendix" on page 532, Galton uses the term "reversion" and the symbol r.
- ^ Galton, F. (24 September 1885). "The British Association: Section II, Anthropology: Opening address by Francis Galton, F.R.S., etc., President of the Anthropological Institute, President of the Section". Nature. 32 (830): 507–510.
- JSTOR 2841583.
- Bibcode:1895RSPS...58..240P.
- JSTOR 2245329.
- ^ "Analyse mathematique sur les probabilités des erreurs de situation d'un point". Mem. Acad. Roy. Sci. Inst. France. Sci. Math, et Phys. (in French). 9: 255–332. 1844 – via Google Books.
- ^ Wright, S. (1921). "Correlation and causation". Journal of Agricultural Research. 20 (7): 557–585.
- ^ a b c d e Real Statistics Using Excel, "Basic Concepts of Correlation", retrieved 22 February 2015.
- ^ Weisstein, Eric W. "Statistical Correlation". Wolfram MathWorld. Retrieved 22 August 2020.
- ISBN 978-1-60021-976-4.
- .
- ^ "2.6 - (Pearson) Correlation Coefficient r". STAT 462. Retrieved 10 July 2021.
- ^ "Introductory Business Statistics: The Correlation Coefficient r". opentextbc.ca. Retrieved 21 August 2020.
- JSTOR 2685263.
- JSTOR 27528906.
- ^ Rummel, R.J. (1976). "Understanding Correlation". ch. 5 (as illustrated for a special case in the next paragraph).
- ISBN 9788391527290.
- ^ a b Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
- JSTOR 2277400.
- ^ "Derivation of the standard error for Pearson's correlation coefficient". Cross Validated. Retrieved 30 July 2021.
- ^ Rahman, N. A. (1968) A Course in Theoretical Statistics, Charles Griffin and Company, 1968
- ISBN 0-85264-215-6(Section 31.19)
- .
- S2CID 207184701.
- JSTOR 2983768.
- ^ Kenney, J.F.; Keeping, E.S. (1951). Mathematics of Statistics. Vol. Part 2 (2nd ed.). Princeton, NJ: Van Nostrand.
- ^ Weisstein, Eric W. "Correlation Coefficient—Bivariate Normal Distribution". Wolfram MathWorld.
- .
- S2CID 52878443.
- ^ a b Wilcox, Rand R. (2005). Introduction to robust estimation and hypothesis testing. Academic Press.
- JSTOR 2335508.
- ^ Huber, Peter. J. (2004). Robust Statistics. Wiley.[page needed]
- ISBN 978-0-511-80225-6.
- ISBN 0-521-54985-X
- JSTOR 2983768.
- JSTOR 2237306..
- ^ "Re: Compute a weighted correlation". sci.tech-archive.net.
- ^ "Weighted Correlation Matrix – File Exchange – MATLAB Central".
- S2CID 4694570.
- ISBN 1-4020-8879-5
- S2CID 1027502. Retrieved 11 February 2018.
- ISBN 978-981-02-3778-3. Retrieved 21 September 2016.
- ISBN 0-412-12420-3.
External links
- "cocor". comparingcorrelations.org. – A free web interface and R package for the statistical comparison of two dependent or independent correlations with overlapping or non-overlapping variables.
- "Correlation". nagysandor.eu. – an interactive Flash simulation on the correlation of two normally distributed variables.
- "Correlation coefficient calculator". hackmath.net. Linear regression.
- "Critical values for Pearson's correlation coefficient" (PDF). frank.mtsu.edu/~dkfuller. – large table.
- "Guess the Correlation". – A game where players guess how correlated two variables in a scatter plot are, in order to gain a better understanding of the concept of correlation.