Statistical distance

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these template messages) This article includes a list of general references, but it lacks sufficient corresponding inline citations. Please help to improve this article by introducing more precise citations. (February 2012) (Learn how and when to remove this template message) This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "Statistical distance" – news · newspapers · books · scholar · JSTOR (December 2020) (Learn how and when to remove this template message) (Learn how and when to remove this template message)

In

, or the distance can be between an individual sample point and a population or a wider sample of points.

A distance between populations can be interpreted as measuring the distance between two

and hence these distances are not directly related to measures of distances between probability measures. Again, a measure of distance between random variables may relate to the extent of dependence between them, rather than to their individual values.

Many statistical distance measures are not

.

Terminology

Many terms are used to refer to various notions of distance; these are often confusingly similar, and may be used inconsistently between authors and over time, either loosely or with precise technical meaning. In addition to "distance", similar terms include

information gain

.

Distances as metrics

Metrics

A metric on a set X is a function (called the distance function or simply distance) d : X × X → R⁺ (where R⁺ is the set of non-negative real numbers). For all x, y, z in X, this function is required to satisfy the following conditions:

1. d(x, y) ≥ 0     (
   non-negativity
   )
2. d(x, y) = 0   if and only if   x = y     (identity of indiscernibles. Note that condition 1 and 2 together produce positive definiteness)
3. d(x, y) = d(y, x)     (symmetry)
4. d(x, z) ≤ d(x, y) + d(y, z)     (subadditivity / triangle inequality).

Generalized metrics

Many statistical distances are not

.

Statistically close

The total variation distance of two distributions ${\displaystyle X}$ and ${\displaystyle Y}$ over a finite domain ${\displaystyle D}$, (often referred to as statistical difference^[2] or statistical distance^[3] in cryptography) is defined as

${\displaystyle \Delta (X,Y)={\frac {1}{2}}\sum _{\alpha \in D}|\Pr[X=\alpha ]-\Pr[Y=\alpha ]|}$.

We say that two

${\displaystyle \{X_{k}\}_{k\in \mathbb {N} }}$ and ${\displaystyle \{Y_{k}\}_{k\in \mathbb {N} }}$ are statistically close if ${\displaystyle \Delta (X_{k},Y_{k})}$ is a negligible function in ${\displaystyle k}$.

Examples

Metrics

• Total variation distance
  (sometimes just called "the" statistical distance)
• Hellinger distance
• Lévy–Prokhorov metric
• Wasserstein metric: also known as the Kantorovich metric, or earth mover's distance
• Mahalanobis distance
• Amari distance
• Integral probability metrics generalize several metrics or pseudometrics on distributions

Divergences

• Kullback–Leibler divergence
• Rényi divergence
• Jensen–Shannon divergence
• Bhattacharyya distance (despite its name it is not a distance, as it violates the triangle inequality)
• f-divergence: generalizes several distances and divergences
• Bayes discriminability index
  , is a positive-definite symmetric measure of the overlap of two distributions.

See also

• Probabilistic metric space
• Randomness extractor
• Similarity measure
• Zero-knowledge proof

Notes

External links

• Distance and Similarity Measures (Wolfram Alpha)

References

• Dodge, Y. (2003) Oxford Dictionary of Statistical Terms, OUP.
  ISBN 0-19-920613-9