User:Bruguiea/sandbox

Playground

${\displaystyle y=X\beta +\epsilon }$

${\displaystyle (n;1)}$

${\displaystyle (n;p)}$

${\displaystyle (p;1)}$

${\displaystyle {\hat {\beta }}=(X'X)^{-1}Xy}$

${\displaystyle {\hat {\epsilon }}=\left(I-(X'X)^{-1}X\right)y}$

${\displaystyle n-r}$

${\displaystyle t_{n-r}={\frac {\hat {\beta _{i}}}{{\hat {\epsilon }}'{\hat {\epsilon }}}}={\frac {\hat {\beta _{i}}}{y'\left(I-(X'X)^{-1}X\right)'\left(I-(X'X)^{-1}X\right)y}}}$

${\displaystyle t_{n-r}\rightarrow p}$

${\displaystyle U=\operatorname {ER} -\alpha \operatorname {Risk} }$

Special case of covariance matrices

A covariance matrix ${\displaystyle \Sigma }$ can be represented as the product ${\displaystyle A'A}$. Its

are positive:

${\displaystyle \Sigma v_{i}=\lambda _{i}v_{i}}$

${\displaystyle A'Av_{i}=\lambda _{i}v_{i}}$

${\displaystyle v_{i}'A'Av_{i}=v_{i}'\lambda _{i}v_{i}}$

${\displaystyle \left\|Av_{i}\right\|=\lambda _{i}\left\|v_{i}\right\|}$

${\displaystyle \lambda _{i}={\frac {\left\|v_{i}\right\|}{\left\|Av_{i}\right\|}}\geq 0}$

The

are orthogonal to one another:

${\displaystyle A'Av_{i}=\lambda _{i}v_{i}}$

${\displaystyle v_{j}'A'Av_{i}=\lambda _{i}v_{j}'v_{i}}$

${\displaystyle (A'Av_{j})'v_{i}=\lambda _{i}v_{j}'v_{i}}$

${\displaystyle \lambda _{j}v_{j}'v_{i}=\lambda _{i}v_{j}'v_{i}}$

${\displaystyle (\lambda _{j}-\lambda _{i})v_{j}'v_{i}=0}$

${\displaystyle v_{j}'v_{i}=0}$ (different eigenvalues, in case of multiplicity, the basis can be orthogonalized)

The Rayleigh quotient can be expressed as a function of the eigenvalues by decomposing any vector ${\displaystyle x}$ on the basis of eigenvectors:

${\displaystyle x=\sum _{i=1}^{n}\alpha _{i}v_{i}}$

${\displaystyle \rho ={\frac {x'A'Ax}{x'x}}}$

${\displaystyle \rho ={\frac {(\sum _{j=1}^{n}\alpha _{j}v_{j})'A'A(\sum _{i=1}^{n}\alpha _{i}v_{i})}{(\sum _{j=1}^{n}\alpha _{j}v_{j})'(\sum _{i=1}^{n}\alpha _{i}v_{i})}}}$

Which, by orthogonality of the eigenvectors, becomes:

${\displaystyle \rho ={\frac {\sum _{i=1}^{n}\alpha _{i}^{2}\lambda _{i}}{\sum _{i=1}^{n}\alpha _{i}^{2}}}}$

If a vector ${\displaystyle x}$ maximizes ${\displaystyle \rho }$, then any vector ${\displaystyle k.x}$ (for ${\displaystyle k\neq 0}$) also maximizes it, one can reduce to the

of maximizing ${\displaystyle \sum _{i=1}^{n}\alpha _{i}^{2}\lambda _{i}}$ under the constrainst that ${\displaystyle \sum _{i=1}^{n}\alpha _{i}^{2}=1}$.

Since all the eingenvalues are non-negative, the problem is convex and the maximum occurs on the edges of the domain, namely when ${\displaystyle \alpha _{1}=1}$ and ${\displaystyle \forall i>1\alpha _{i}=0}$ (when the eigenvalues are ordered in decreasing magnitude).

This property is the basis for

.