Bayesian multivariate linear regression

Part of a series on
Regression analysis
Models
Linear regression Simple regression Polynomial regression General linear model
Generalized linear model Vector generalized linear model Discrete choice Binomial regression Binary regression Logistic regression Multinomial logistic regression Mixed logit Probit Multinomial probit Ordered logit Ordered probit Poisson
Multilevel model Fixed effects Random effects Linear mixed-effects model Nonlinear mixed-effects model
Nonlinear regression Nonparametric Semiparametric Robust Quantile Isotonic Principal components Least angle Local Segmented
Errors-in-variables
Estimation
Least squares Linear Non-linear
Ordinary Weighted Generalized Generalized estimating equation
Partial Total Non-negative Ridge regression Regularized
Least absolute deviations Iteratively reweighted Bayesian Bayesian multivariate Least-squares spectral analysis
Background
Regression validation Mean and predicted response Errors and residuals Goodness of fit Studentized residual Gauss–Markov theorem
Mathematics portal
v t e

In statistics, Bayesian multivariate linear regression is a

.

Details

Consider a regression problem where the

, grouped into a vector ${\displaystyle \mathbf {x} _{i}}$ of length k (where a dummy variable with a value of 1 has been added to allow for an intercept coefficient). This can be viewed as a set of m related regression problems for each observation i: where the set of errors ${\displaystyle \{\epsilon _{i,1},\ldots ,\epsilon _{i,m}\}}$ are all correlated. Equivalently, it can be viewed as a single regression problem where the outcome is a ${\displaystyle \mathbf {y} _{i}^{\mathsf {T}}}$ and the regression coefficient vectors are stacked next to each other, as follows:

The coefficient matrix B is a ${\displaystyle k\times m}$ matrix where the coefficient vectors ${\displaystyle {\boldsymbol {\beta }}_{1},\ldots ,{\boldsymbol {\beta }}_{m}}$ for each regression problem are stacked horizontally:

The noise vector ${\displaystyle {\boldsymbol {\epsilon }}_{i}}$ for each observation i is jointly normal, so that the outcomes for a given observation are correlated:

We can write the entire regression problem in matrix form as:

where Y and E are ${\displaystyle n\times m}$ matrices. The design matrix X is an ${\displaystyle n\times k}$ matrix with the observations stacked vertically, as in the standard linear regression setup:

The classical, frequentists

solution is to simply estimate the matrix of regression coefficients ${\displaystyle {\hat {\mathbf {B} }}}$ using the :

To obtain the Bayesian solution, we need to specify the conditional likelihood and then find the appropriate conjugate prior. As with the univariate case of linear Bayesian regression, we will find that we can specify a natural conditional conjugate prior (which is scale dependent).

Let us write our conditional likelihood as^[1]

writing the error ${\displaystyle \mathbf {E} }$ in terms of ${\displaystyle \mathbf {Y} ,\mathbf {X} ,}$ and ${\displaystyle \mathbf {B} }$ yields

We seek a natural conjugate prior—a joint density ${\displaystyle \rho (\mathbf {B} ,\Sigma _{\epsilon })}$ which is of the same functional form as the likelihood. Since the likelihood is quadratic in ${\displaystyle \mathbf {B} }$, we re-write the likelihood so it is normal in ${\displaystyle (\mathbf {B} -{\hat {\mathbf {B} }})}$ (the deviation from classical sample estimate).

Using the same technique as with Bayesian linear regression, we decompose the exponential term using a matrix-form of the sum-of-squares technique. Here, however, we will also need to use the Matrix Differential Calculus (Kronecker product and vectorization transformations).

First, let us apply sum-of-squares to obtain new expression for the likelihood:

We would like to develop a conditional form for the priors:

where ${\displaystyle \rho ({\boldsymbol {\Sigma }}_{\epsilon })}$ is an inverse-Wishart distribution and ${\displaystyle \rho (\mathbf {B} |{\boldsymbol {\Sigma }}_{\epsilon })}$ is some form of normal distribution in the matrix ${\displaystyle \mathbf {B} }$. This is accomplished using the vectorization transformation, which converts the likelihood from a function of the matrices ${\displaystyle \mathbf {B} ,{\hat {\mathbf {B} }}}$ to a function of the vectors ${\displaystyle {\boldsymbol {\beta }}=\operatorname {vec} (\mathbf {B} ),{\hat {\boldsymbol {\beta }}}=\operatorname {vec} ({\hat {\mathbf {B} }})}$.

Write

Let

where ${\displaystyle \mathbf {A} \otimes \mathbf {B} }$ denotes the Kronecker product of matrices A and B, a generalization of the outer product which multiplies an ${\displaystyle m\times n}$ matrix by a ${\displaystyle p\times q}$ matrix to generate an ${\displaystyle mp\times nq}$ matrix, consisting of every combination of products of elements from the two matrices.

Then

which will lead to a likelihood which is normal in ${\displaystyle ({\boldsymbol {\beta }}-{\hat {\boldsymbol {\beta }}})}$.

With the likelihood in a more tractable form, we can now find a natural (conditional) conjugate prior.

Conjugate prior distribution

The natural conjugate prior using the vectorized variable ${\displaystyle {\boldsymbol {\beta }}}$ is of the form:^[1]

where and

Posterior distribution

Using the above prior and likelihood, the posterior distribution can be expressed as:^[1]

where ${\displaystyle \operatorname {vec} (\mathbf {B} _{0})={\boldsymbol {\beta }}_{0}}$. The terms involving ${\displaystyle \mathbf {B} }$ can be grouped (with ${\displaystyle {\boldsymbol {\Lambda }}_{0}=\mathbf {U} ^{\mathsf {T}}\mathbf {U} }$) using: with

This now allows us to write the posterior in a more useful form:

This takes the form of an inverse-Wishart distribution times a Matrix normal distribution:

and

The parameters of this posterior are given by:

See also

• Bayesian linear regression
• Matrix normal distribution

References

This article includes a list of general references, but it lacks sufficient corresponding inline citations. Please help to improve this article by introducing more precise citations. (November 2010) (Learn how and when to remove this template message)

• ISBN 0-471-57428-7
  .
• Geisser, S. (1965). "Bayesian Estimation in Multivariate Analysis".
  JSTOR 2238083
  .
• Tiao, G. C.; Zellner, A. (1964). "On the Bayesian Estimation of Multivariate Regression".
  JSTOR 2984424
  .