Matrix decomposition

In the mathematical discipline of linear algebra, a matrix decomposition or matrix factorization is a factorization of a matrix into a product of matrices. There are many different matrix decompositions; each finds use among a particular class of problems.

Example

In numerical analysis, different decompositions are used to implement efficient matrix algorithms.

For instance, when solving a system of linear equations ${\displaystyle A\mathbf {x} =\mathbf {b} }$, the matrix A can be decomposed via the

U. The systems ${\displaystyle L(U\mathbf {x} )=\mathbf {b} }$ and ${\displaystyle U\mathbf {x} =L^{-1}\mathbf {b} }$ require fewer additions and multiplications to solve, compared with the original system ${\displaystyle A\mathbf {x} =\mathbf {b} }$, though one might require significantly more digits in inexact arithmetic such as .

Similarly, the

.

Decompositions related to solving systems of linear equations

LU decomposition

• Traditionally applicable to: square matrix A, although rectangular matrices can be applicable.^{[1][nb 1]}
• Decomposition: ${\displaystyle A=LU}$, where L is lower triangular and U is upper triangular.
• Related: the
  LDU decomposition
  is ${\displaystyle A=LDU}$, where L is lower triangular with ones on the diagonal, U is upper triangular with ones on the diagonal, and D is a diagonal matrix.
• Related: the
  LUP decomposition
  is ${\displaystyle PA=LU}$, where L is lower triangular, U is upper triangular, and P is a permutation matrix.
• Existence: An LUP decomposition exists for any square matrix A. When P is an identity matrix, the LUP decomposition reduces to the LU decomposition.
• Comments: The LUP and LU decompositions are useful in solving an n-by-n system of linear equations ${\displaystyle A\mathbf {x} =\mathbf {b} }$. These decompositions summarize the process of Gaussian elimination in matrix form. Matrix P represents any row interchanges carried out in the process of Gaussian elimination. If Gaussian elimination produces the row echelon form without requiring any row interchanges, then P = I, so an LU decomposition exists.

LU reduction

Block LU decomposition

Rank factorization

• Applicable to: m-by-n matrix A of rank r
• Decomposition: ${\displaystyle A=CF}$ where C is an m-by-r full column rank matrix and F is an r-by-n full row rank matrix
• Comment: The rank factorization can be used to
  obtain all solutions of the linear system
  ${\displaystyle A\mathbf {x} =\mathbf {b} }$.

Cholesky decomposition

• Applicable to:
  positive definite
  matrix ${\displaystyle A}$
• Decomposition: ${\displaystyle A=U^{*}U}$, where ${\displaystyle U}$ is upper triangular with real positive diagonal entries
• Comment: if the matrix ${\displaystyle A}$ is Hermitian and positive semi-definite, then it has a decomposition of the form ${\displaystyle A=U^{*}U}$ if the diagonal entries of ${\displaystyle U}$ are allowed to be zero
• Uniqueness: for positive definite matrices Cholesky decomposition is unique. However, it is not unique in the positive semi-definite case.
• Comment: if ${\displaystyle A}$ is real and symmetric, ${\displaystyle U}$ has all real elements
• Comment: An alternative is the
  LDL decomposition
  , which can avoid extracting square roots.

QR decomposition

• Applicable to: m-by-n matrix A with linearly independent columns
• Decomposition: ${\displaystyle A=QR}$ where ${\displaystyle Q}$ is a unitary matrix of size m-by-m, and ${\displaystyle R}$ is an upper triangular matrix of size m-by-n
• Uniqueness: In general it is not unique, but if ${\displaystyle A}$ is of full
  rank
  , then there exists a single ${\displaystyle R}$ that has all positive diagonal elements. If ${\displaystyle A}$ is square, also ${\displaystyle Q}$ is unique.
• Comment: The QR decomposition provides an effective way to solve the system of equations ${\displaystyle A\mathbf {x} =\mathbf {b} }$. The fact that ${\displaystyle Q}$ is orthogonal means that ${\displaystyle Q^{\mathrm {T} }Q=I}$, so that ${\displaystyle A\mathbf {x} =\mathbf {b} }$ is equivalent to ${\displaystyle R\mathbf {x} =Q^{\mathsf {T}}\mathbf {b} }$, which is very easy to solve since ${\displaystyle R}$ is triangular.

RRQR factorization

Interpolative decomposition

Decompositions based on eigenvalues and related concepts

Eigendecomposition

• Also called
  spectral decomposition
  .
• Applicable to: square matrix A with linearly independent eigenvectors (not necessarily distinct eigenvalues).
• Decomposition: ${\displaystyle A=VDV^{-1}}$, where D is a
  eigenvectors
  of A.
• Existence: An n-by-n matrix A always has n (complex) eigenvalues, which can be ordered (in more than one way) to form an n-by-n diagonal matrix D and a corresponding matrix of nonzero columns V that satisfies the
  eigenvalue equation
  ${\displaystyle AV=VD}$. ${\displaystyle V}$ is invertible if and only if the n eigenvectors are
  algebraic multiplicity
  ). A sufficient (but not necessary) condition for this to happen is that all the eigenvalues are different (in this case geometric and algebraic multiplicity are equal to 1)
• Comment: One can always normalize the eigenvectors to have length one (see the definition of the eigenvalue equation)
• Comment: Every normal matrix A (that is, matrix for which ${\displaystyle AA^{*}=A^{*}A}$, where ${\displaystyle A^{*}}$ is a conjugate transpose) can be eigendecomposed. For a normal matrix A (and only for a normal matrix), the eigenvectors can also be made orthonormal (${\displaystyle VV^{*}=I}$) and the eigendecomposition reads as ${\displaystyle A=VDV^{*}}$. In particular all unitary, Hermitian, or skew-Hermitian (in the real-valued case, all orthogonal, symmetric, or skew-symmetric, respectively) matrices are normal and therefore possess this property.
• Comment: For any real symmetric matrix A, the eigendecomposition always exists and can be written as ${\displaystyle A=VDV^{\mathsf {T}}}$, where both D and V are real-valued.
• Comment: The eigendecomposition is useful for understanding the solution of a system of linear ordinary differential equations or linear difference equations. For example, the difference equation ${\displaystyle x_{t+1}=Ax_{t}}$ starting from the initial condition ${\displaystyle x_{0}=c}$ is solved by ${\displaystyle x_{t}=A^{t}c}$, which is equivalent to ${\displaystyle x_{t}=VD^{t}V^{-1}c}$, where V and D are the matrices formed from the eigenvectors and eigenvalues of A. Since D is diagonal, raising it to power ${\displaystyle D^{t}}$, just involves raising each element on the diagonal to the power t. This is much easier to do and understand than raising A to power t, since A is usually not diagonal.

Jordan decomposition

The Jordan normal form and the Jordan–Chevalley decomposition

• Applicable to: square matrix A
• Comment: the Jordan normal form generalizes the eigendecomposition to cases where there are repeated eigenvalues and cannot be diagonalized, the Jordan–Chevalley decomposition does this without choosing a basis.

Schur decomposition

• Applicable to: square matrix A
• Decomposition (complex version): ${\displaystyle A=UTU^{*}}$, where U is a unitary matrix, ${\displaystyle U^{*}}$ is the
  eigenvalues
  of A along its diagonal.
• Comment: if A is a normal matrix, then T is diagonal and the Schur decomposition coincides with the spectral decomposition.

Real Schur decomposition

• Applicable to: square matrix A
• Decomposition: This is a version of Schur decomposition where ${\displaystyle V}$ and ${\displaystyle S}$ only contain real numbers. One can always write ${\displaystyle A=VSV^{\mathsf {T}}}$ where V is a real orthogonal matrix, ${\displaystyle V^{\mathsf {T}}}$ is the
  Schur form. The blocks on the diagonal of S are of size 1×1 (in which case they represent real eigenvalues) or 2×2 (in which case they are derived from complex conjugate
  eigenvalue pairs).

QZ decomposition

Main article:

QZ decomposition

• Also called: generalized Schur decomposition
• Applicable to: square matrices A and B
• Comment: there are two versions of this decomposition: complex and real.
• Decomposition (complex version): ${\displaystyle A=QSZ^{*}}$ and ${\displaystyle B=QTZ^{*}}$ where Q and Z are
  upper triangular
  matrices.
• Comment: in the complex QZ decomposition, the ratios of the diagonal elements of S to the corresponding diagonal elements of T, ${\displaystyle \lambda _{i}=S_{ii}/T_{ii}}$, are the generalized
  eigenvalues that solve the generalized eigenvalue problem
  ${\displaystyle A\mathbf {v} =\lambda B\mathbf {v} }$ (where ${\displaystyle \lambda }$ is an unknown scalar and v is an unknown nonzero vector).
• Decomposition (real version): ${\displaystyle A=QSZ^{\mathsf {T}}}$ and ${\displaystyle B=QTZ^{\mathsf {T}}}$ where A, B, Q, Z, S, and T are matrices containing real numbers only. In this case Q and Z are
  transposition, and S and T are block upper triangular
  matrices. The blocks on the diagonal of S and T are of size 1×1 or 2×2.

Takagi's factorization

• Applicable to: square, complex, symmetric matrix A.
• Decomposition: ${\displaystyle A=VDV^{\mathsf {T}}}$, where D is a real nonnegative diagonal matrix, and V is unitary. ${\displaystyle V^{\mathsf {T}}}$ denotes the
  matrix transpose
  of V.
• Comment: The diagonal elements of D are the nonnegative square roots of the eigenvalues of ${\displaystyle AA^{*}=VD^{2}V^{-1}}$.
• Comment: V may be complex even if A is real.
• Comment: This is not a special case of the eigendecomposition (see above), which uses ${\displaystyle V^{-1}}$ instead of ${\displaystyle V^{\mathsf {T}}}$. Moreover, if A is not real, it is not Hermitian and the form using ${\displaystyle V^{*}}$ also does not apply.

Singular value decomposition

• Applicable to: m-by-n matrix A.
• Decomposition: ${\displaystyle A=UDV^{*}}$, where D is a nonnegative diagonal matrix, and U and V satisfy ${\displaystyle U^{*}U=I,V^{*}V=I}$. Here ${\displaystyle V^{*}}$ is the
  transpose
  , if V contains real numbers only), and I denotes the identity matrix (of some dimension).
• Comment: The diagonal elements of D are called the singular values of A.
• Comment: Like the eigendecomposition above, the singular value decomposition involves finding basis directions along which matrix multiplication is equivalent to scalar multiplication, but it has greater generality since the matrix under consideration need not be square.
• Uniqueness: the singular values of ${\displaystyle A}$ are always uniquely determined. ${\displaystyle U}$ and ${\displaystyle V}$ need not to be unique in general.

Scale-invariant decompositions

Refers to variants of existing matrix decompositions, such as the SVD, that are invariant with respect to diagonal scaling.

• Applicable to: m-by-n matrix A.
• Unit-Scale-Invariant Singular-Value Decomposition: ${\displaystyle A=DUSV^{*}E}$, where S is a unique nonnegative diagonal matrix of scale-invariant singular values, U and V are unitary matrices, ${\displaystyle V^{*}}$ is the conjugate transpose of V, and positive diagonal matrices D and E.
• Comment: Is analogous to the SVD except that the diagonal elements of S are invariant with respect to left and/or right multiplication of A by arbitrary nonsingular diagonal matrices, as opposed to the standard SVD for which the singular values are invariant with respect to left and/or right multiplication of A by arbitrary unitary matrices.
• Comment: Is an alternative to the standard SVD when invariance is required with respect to diagonal rather than unitary transformations of A.
• Uniqueness: The scale-invariant singular values of ${\displaystyle A}$ (given by the diagonal elements of S) are always uniquely determined. Diagonal matrices D and E, and unitary U and V, are not necessarily unique in general.
• Comment: U and V matrices are not the same as those from the SVD.

Analogous scale-invariant decompositions can be derived from other matrix decompositions; for example, to obtain scale-invariant eigenvalues.^[3][4]

Hessenberg decomposition

• Applicable to: square matrix A.
• Decomposition: ${\displaystyle A=PHP^{*}}$ where ${\displaystyle H}$ is the Hessenberg matrix and ${\displaystyle P}$ is a unitary matrix.
• Comment: often the first step in the Schur decomposition.

Complete orthogonal decomposition

• Also known as: UTV decomposition, ULV decomposition, URV decomposition.
• Applicable to: m-by-n matrix A.
• Decomposition: ${\displaystyle A=UTV^{*}}$, where T is a triangular matrix, and U and V are unitary matrices.
• Comment: Similar to the singular value decomposition and to the Schur decomposition.

Other decompositions

Polar decomposition

• Applicable to: any square complex matrix A.
• Decomposition: ${\displaystyle A=UP}$ (right polar decomposition) or ${\displaystyle A=P'U}$ (left polar decomposition), where U is a
  Hermitian matrices
  .
• Uniqueness: ${\displaystyle P}$ is always unique and equal to ${\displaystyle {\sqrt {A^{*}A}}}$ (which is always hermitian and positive semidefinite). If ${\displaystyle A}$ is invertible, then ${\displaystyle U}$ is unique.
• Comment: Since any Hermitian matrix admits a spectral decomposition with a unitary matrix, ${\displaystyle P}$ can be written as ${\displaystyle P=VDV^{*}}$. Since ${\displaystyle P}$ is positive semidefinite, all elements in ${\displaystyle D}$ are non-negative. Since the product of two unitary matrices is unitary, taking ${\displaystyle W=UV}$one can write ${\displaystyle A=U(VDV^{*})=WDV^{*}}$ which is the singular value decomposition. Hence, the existence of the polar decomposition is equivalent to the existence of the singular value decomposition.

Algebraic polar decomposition

• Applicable to: square, complex, non-singular matrix A.^[5]
• Decomposition: ${\displaystyle A=QS}$, where Q is a complex orthogonal matrix and S is complex symmetric matrix.
• Uniqueness: If ${\displaystyle A^{\mathsf {T}}A}$ has no negative real eigenvalues, then the decomposition is unique.^[6]
• Comment: The existence of this decomposition is equivalent to ${\displaystyle AA^{\mathsf {T}}}$ being similar to ${\displaystyle A^{\mathsf {T}}A}$.^[7]
• Comment: A variant of this decomposition is ${\displaystyle A=RC}$, where R is a real matrix and C is a circular matrix.^[6]

Mostow's decomposition

• Applicable to: square, complex, non-singular matrix A.^[8][9]
• Decomposition: ${\displaystyle A=Ue^{iM}e^{S}}$, where U is unitary, M is real anti-symmetric and S is real symmetric.
• Comment: The matrix A can also be decomposed as ${\displaystyle A=U_{2}e^{S_{2}}e^{iM_{2}}}$, where U₂ is unitary, M₂ is real anti-symmetric and S₂ is real symmetric.^[6]

Sinkhorn normal form

• Applicable to: square real matrix A with strictly positive elements.
• Decomposition: ${\displaystyle A=D_{1}SD_{2}}$, where S is doubly stochastic and D₁ and D₂ are real diagonal matrices with strictly positive elements.

Sectoral decomposition

• Applicable to: square, complex matrix A with numerical range contained in the sector ${\displaystyle S_{\alpha }=\left\{re^{i\theta }\in \mathbb {C} \mid r>0,|\theta |\leq \alpha <{\frac {\pi }{2}}\right\}}$.
• Decomposition: ${\displaystyle A=CZC^{*}}$, where C is an invertible complex matrix and ${\displaystyle Z=\operatorname {diag} \left(e^{i\theta _{1}},\ldots ,e^{i\theta _{n}}\right)}$ with all ${\displaystyle \left|\theta _{j}\right|\leq \alpha }$.^[10][11]

Williamson's normal form

• Applicable to: square,
  positive-definite
  real matrix A with order 2n×2n.
• Decomposition: ${\displaystyle A=S^{\mathsf {T}}\operatorname {diag} (D,D)S}$, where ${\displaystyle S\in {\text{Sp}}(2n)}$ is a symplectic matrix and D is a nonnegative n-by-n diagonal matrix.^[12]

Matrix square root

• Decomposition: ${\displaystyle A=BB}$, not unique in general.
• In the case of positive semidefinite ${\displaystyle A}$, there is a unique positive semidefinite ${\displaystyle B}$ such that ${\displaystyle A=B^{*}B=BB}$.

Generalizations

This section needs expansion with: examples and additional citations. You can help by adding to it. (December 2014)

There exist analogues of the SVD, QR, LU and Cholesky factorizations for quasimatrices and cmatrices or continuous matrices.^[13] A ‘quasimatrix’ is, like a matrix, a rectangular scheme whose elements are indexed, but one discrete index is replaced by a continuous index. Likewise, a ‘cmatrix’, is continuous in both indices. As an example of a cmatrix, one can think of the kernel of an integral operator.

These factorizations are based on early work by Fredholm (1903), Hilbert (1904) and Schmidt (1907). For an account, and a translation to English of the seminal papers, see Stewart (2011).

See also

• Matrix splitting
• Non-negative matrix factorization
• Principal component analysis

References

Notes

Citations

Bibliography

• Choudhury, Dipa; Horn, Roger A. (April 1987). "A Complex Orthogonal-Symmetric Analog of the Polar Decomposition". SIAM Journal on Algebraic and Discrete Methods. 8 (2): 219–225.
  doi:10.1137/0608019
  .
• doi:10.1007/bf02421317
• Hilbert, D. (1904), "Grundzüge einer allgemeinen Theorie der linearen Integralgleichungen", Nachr. Königl. Ges. Gött (in German), 1904: 49–91
• Horn, Roger A.; Merino, Dennis I. (January 1995). "Contragredient equivalence: A canonical form and some applications". Linear Algebra and Its Applications. 214: 43–92.
  doi:10.1016/0024-3795(93)00056-6
  .
• Meyer, C. D. (2000), Matrix Analysis and Applied Linear Algebra,
  ISBN 978-0-89871-454-8
• doi:10.1007/bf01449770
• Simon, C.; Blume, L. (1994). Mathematics for Economists. Norton.
  ISBN 978-0-393-95733-4
  .
• Stewart, G. W. (2011), Fredholm, Hilbert, Schmidt: three fundamental papers on integral equations (PDF), retrieved 2015-01-06
• Townsend, A.; Trefethen, L. N. (2015), "Continuous analogues of matrix factorizations",
  PMID 25568618
• Jun, Lu (2021), Numerical matrix decomposition and its modern applications: A rigorous first course,
  arXiv:2107.02579

External links

• Online Matrix Calculator
• Wolfram Alpha Matrix Decomposition Computation » LU and QR Decomposition
• Springer Encyclopaedia of Mathematics » Matrix factorization
• GraphLab GraphLab collaborative filtering library, large scale parallel implementation of matrix decomposition methods (in C++) for multicore.