Multinomial logistic regression

This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "Multinomial logistic regression" – news · newspapers · books · scholar · JSTOR (November 2011) (Learn how and when to remove this template message)

In

Multinomial logistic regression is known by a variety of other names, including polytomous LR,

Introduction

There are multiple equivalent ways to describe the mathematical model underlying multinomial logistic regression. This can make it difficult to compare different treatments of the subject in different texts. The article on logistic regression presents a number of equivalent formulations of simple logistic regression, and many of these have analogues in the multinomial logit model.

The idea behind all of them, as in many other statistical classification techniques, is to construct a linear predictor function that constructs a score from a set of weights that are linearly combined with the explanatory variables (features) of a given observation using a dot product:

${\displaystyle \operatorname {score} (\mathbf {X} _{i},k)={\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i},}$

where X_i is the vector of explanatory variables describing observation i, β_k is a vector of weights (or

• The observed outcomes are different variants of a disease such as hepatitis (possibly including "no disease" and/or other related diseases) in a set of patients, and the explanatory variables might be characteristics of the patients thought to be pertinent (sex, race, age, blood pressure, outcomes of various liver-function tests, etc.). The goal is then to predict which disease is causing the observed liver-related symptoms in a new patient.
• The observed outcomes are the party chosen by a set of people in an election, and the explanatory variables are the demographic characteristics of each person (e.g. sex, race, age, income, etc.). The goal is then to predict the likely vote of a new voter with given characteristics.

Linear predictor

As in other forms of linear regression, multinomial logistic regression uses a linear predictor function ${\displaystyle f(k,i)}$ to predict the probability that observation i has outcome k, of the following form:

${\displaystyle f(k,i)=\beta _{0,k}+\beta _{1,k}x_{1,i}+\beta _{2,k}x_{2,i}+\cdots +\beta _{M,k}x_{M,i},}$

where ${\displaystyle \beta _{m,k}}$ is a

This formulation is also known as the Additive Log Ratio transform commonly used in compositional data analysis. In other applications it’s referred to as “relative risk”.^[7]

If we exponentiate both sides and solve for the probabilities, we get:

${\displaystyle \Pr(Y_{i}=k)\,=\,{\Pr(Y_{i}=K)}\;e^{{\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i}}\;\;\;\;,\;\;k<K}$

Using the fact that all K of the probabilities must sum to one, we find:

${\displaystyle \Pr(Y_{i}=K)\,=\,1-\sum _{j=1}^{K-1}\Pr(Y_{i}=j)\,=\,1-\sum _{j=1}^{K-1}{\Pr(Y_{i}=K)}\;e^{{\boldsymbol {\beta }}_{j}\cdot \mathbf {X} _{i}}\;\;\Rightarrow \;\;\Pr(Y_{i}=K)\,=\,{\frac {1}{1+\sum _{j=1}^{K-1}e^{{\boldsymbol {\beta }}_{j}\cdot \mathbf {X} _{i}}}}}$.

We can use this to find the other probabilities:

${\displaystyle \Pr(Y_{i}=k)={\frac {e^{{\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i}}}{1+\sum _{j=1}^{K-1}e^{{\boldsymbol {\beta }}_{j}\cdot \mathbf {X} _{i}}}}\;\;\;\;,\;\;k<K}$.

The fact that we run multiple regressions reveals why the model relies on the assumption of independence of irrelevant alternatives described above.

Estimating the coefficients

The unknown parameters in each vector β_k are typically jointly estimated by

As in the binary case, we need an extra term ${\displaystyle -\ln Z}$ to ensure that the whole set of probabilities forms a probability distribution, i.e. so that they all sum to one:

${\displaystyle \sum _{k=1}^{K}\Pr(Y_{i}=k)=1}$

The reason why we need to add a term to ensure normalization, rather than multiply as is usual, is because we have taken the logarithm of the probabilities. Exponentiating both sides turns the additive term into a multiplicative factor, so that the probability is just the Gibbs measure:

${\displaystyle \Pr(Y_{i}=k)={\frac {1}{Z}}e^{{\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i}}\;\;\;\;,\;\;k\leq K}$.

The quantity Z is called the partition function for the distribution. We can compute the value of the partition function by applying the above constraint that requires all probabilities to sum to 1:

${\displaystyle 1=\sum _{k=1}^{K}\Pr(Y_{i}=k)\;=\;\sum _{k=1}^{K}{\frac {1}{Z}}e^{{\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i}}\;=\;{\frac {1}{Z}}\sum _{k=1}^{K}e^{{\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i}}}$

Therefore:

${\displaystyle Z=\sum _{k=1}^{K}e^{{\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i}}}$

Note that this factor is "constant" in the sense that it is not a function of Y_i, which is the variable over which the probability distribution is defined. However, it is definitely not constant with respect to the explanatory variables, or crucially, with respect to the unknown regression coefficients β_k, which we will need to determine through some sort of optimization procedure.

The resulting equations for the probabilities are

${\displaystyle \Pr(Y_{i}=k)={\frac {e^{{\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i}}}{\sum _{j=1}^{K}e^{{\boldsymbol {\beta }}_{j}\cdot \mathbf {X} _{i}}}}\;\;\;\;,\;\;k\leq K}$.

Or generally:

${\displaystyle \Pr(Y_{i}=c)={\frac {e^{{\boldsymbol {\beta }}_{c}\cdot \mathbf {X} _{i}}}{\sum _{j=1}^{K}e^{{\boldsymbol {\beta }}_{j}\cdot \mathbf {X} _{i}}}}}$

The following function:

${\displaystyle \operatorname {softmax} (k,x_{1},\ldots ,x_{n})={\frac {e^{x_{k}}}{\sum _{i=1}^{n}e^{x_{i}}}}}$

is referred to as the softmax function. The reason is that the effect of exponentiating the values ${\displaystyle x_{1},\ldots ,x_{n}}$ is to exaggerate the differences between them. As a result, ${\displaystyle \operatorname {softmax} (k,x_{1},\ldots ,x_{n})}$ will return a value close to 0 whenever ${\displaystyle x_{k}}$ is significantly less than the maximum of all the values, and will return a value close to 1 when applied to the maximum value, unless it is extremely close to the next-largest value. Thus, the softmax function can be used to construct a

The softmax function thus serves as the equivalent of the logistic function in binary logistic regression.

Note that not all of the ${\displaystyle \beta _{k}}$ vectors of coefficients are uniquely identifiable. This is due to the fact that all probabilities must sum to 1, making one of them completely determined once all the rest are known. As a result, there are only ${\displaystyle k-1}$ separately specifiable probabilities, and hence ${\displaystyle k-1}$ separately identifiable vectors of coefficients. One way to see this is to note that if we add a constant vector to all of the coefficient vectors, the equations are identical:

${\displaystyle {\begin{aligned}{\frac {e^{({\boldsymbol {\beta }}_{c}+C)\cdot \mathbf {X} _{i}}}{\sum _{k=1}^{K}e^{({\boldsymbol {\beta }}_{k}+C)\cdot \mathbf {X} _{i}}}}&={\frac {e^{{\boldsymbol {\beta }}_{c}\cdot \mathbf {X} _{i}}e^{C\cdot \mathbf {X} _{i}}}{\sum _{k=1}^{K}e^{{\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i}}e^{C\cdot \mathbf {X} _{i}}}}\\&={\frac {e^{C\cdot \mathbf {X} _{i}}e^{{\boldsymbol {\beta }}_{c}\cdot \mathbf {X} _{i}}}{e^{C\cdot \mathbf {X} _{i}}\sum _{k=1}^{K}e^{{\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i}}}}\\&={\frac {e^{{\boldsymbol {\beta }}_{c}\cdot \mathbf {X} _{i}}}{\sum _{k=1}^{K}e^{{\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i}}}}\end{aligned}}}$

As a result, it is conventional to set ${\displaystyle C=-{\boldsymbol {\beta }}_{K}}$ (or alternatively, one of the other coefficient vectors). Essentially, we set the constant so that one of the vectors becomes 0, and all of the other vectors get transformed into the difference between those vectors and the vector we chose. This is equivalent to "pivoting" around one of the K choices, and examining how much better or worse all of the other K-1 choices are, relative to the choice we are pivoting around. Mathematically, we transform the coefficients as follows:

${\displaystyle {\begin{aligned}{\boldsymbol {\beta }}'_{k}&={\boldsymbol {\beta }}_{k}-{\boldsymbol {\beta }}_{K}\;\;\;,\;k<K\\{\boldsymbol {\beta }}'_{K}&=0\end{aligned}}}$

This leads to the following equations:

${\displaystyle \Pr(Y_{i}=k)={\frac {e^{{\boldsymbol {\beta }}'_{k}\cdot \mathbf {X} _{i}}}{1+\sum _{j=1}^{K-1}e^{{\boldsymbol {\beta }}'_{j}\cdot \mathbf {X} _{i}}}}\;\;\;\;,\;\;k\leq K}$

Other than the prime symbols on the regression coefficients, this is exactly the same as the form of the model described above, in terms of K-1 independent two-way regressions.

As a latent-variable model

It is also possible to formulate multinomial logistic regression as a latent variable model, following the two-way latent variable model described for binary logistic regression. This formulation is common in the theory of discrete choice models, and makes it easier to compare multinomial logistic regression to the related multinomial probit model, as well as to extend it to more complex models.

Imagine that, for each data point i and possible outcome k=1,2,...,K, there is a continuous

This latent variable can be thought of as the utility associated with data point i choosing outcome k, where there is some randomness in the actual amount of utility obtained, which accounts for other unmodeled factors that go into the choice. The value of the actual variable ${\displaystyle Y_{i}}$ is then determined in a non-random fashion from these latent variables (i.e. the randomness has been moved from the observed outcomes into the latent variables), where outcome k is chosen if and only if the associated utility (the value of ${\displaystyle Y_{i,k}^{\ast }}$) is greater than the utilities of all the other choices, i.e. if the utility associated with outcome k is the maximum of all the utilities. Since the latent variables are

${\displaystyle {\begin{aligned}\Pr(Y_{i}=1)&=\Pr(Y_{i,1}^{\ast }>Y_{i,2}^{\ast }{\text{ and }}Y_{i,1}^{\ast }>Y_{i,3}^{\ast }{\text{ and }}\cdots {\text{ and }}Y_{i,1}^{\ast }>Y_{i,K}^{\ast })\\\Pr(Y_{i}=2)&=\Pr(Y_{i,2}^{\ast }>Y_{i,1}^{\ast }{\text{ and }}Y_{i,2}^{\ast }>Y_{i,3}^{\ast }{\text{ and }}\cdots {\text{ and }}Y_{i,2}^{\ast }>Y_{i,K}^{\ast })\\\cdots &\\\Pr(Y_{i}=K)&=\Pr(Y_{i,K}^{\ast }>Y_{i,1}^{\ast }{\text{ and }}Y_{i,K}^{\ast }>Y_{i,2}^{\ast }{\text{ and }}\cdots {\text{ and }}Y_{i,K}^{\ast }>Y_{i,K-1}^{\ast })\\\end{aligned}}}$

Or equivalently:

${\displaystyle \Pr(Y_{i}=k)\;=\;\Pr(\max(Y_{i,1}^{\ast },Y_{i,2}^{\ast },\ldots ,Y_{i,K}^{\ast })=Y_{i,k}^{\ast })\;\;\;\;,\;\;k\leq K}$

Let's look more closely at the first equation, which we can write as follows:

${\displaystyle {\begin{aligned}\Pr(Y_{i}=1)&=\Pr(Y_{i,1}^{\ast }>Y_{i,k}^{\ast }\ \forall \ k=2,\ldots ,K)\\&=\Pr(Y_{i,1}^{\ast }-Y_{i,k}^{\ast }>0\ \forall \ k=2,\ldots ,K)\\&=\Pr({\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}+\varepsilon _{1}-({\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i}+\varepsilon _{k})>0\ \forall \ k=2,\ldots ,K)\\&=\Pr(({\boldsymbol {\beta }}_{1}-{\boldsymbol {\beta }}_{k})\cdot \mathbf {X} _{i}>\varepsilon _{k}-\varepsilon _{1}\ \forall \ k=2,\ldots ,K)\end{aligned}}}$

There are a few things to realize here:

1. In general, if ${\displaystyle X\sim \operatorname {EV} _{1}(a,b)}$ and ${\displaystyle Y\sim \operatorname {EV} _{1}(a,b)}$ then ${\displaystyle X-Y\sim \operatorname {Logistic} (0,b).}$ That is, the difference of two
   independent identically distributed extreme-value-distributed variables follows the logistic distribution, where the first parameter is unimportant. This is understandable since the first parameter is a location parameter
   , i.e. it shifts the mean by a fixed amount, and if two values are both shifted by the same amount, their difference remains the same. This means that all of the relational statements underlying the probability of a given choice involve the logistic distribution, which makes the initial choice of the extreme-value distribution, which seemed rather arbitrary, somewhat more understandable.
2. The second parameter in an extreme-value or logistic distribution is a scale parameter, such that if ${\displaystyle X\sim \operatorname {Logistic} (0,1)}$ then ${\displaystyle bX\sim \operatorname {Logistic} (0,b).}$ This means that the effect of using an error variable with an arbitrary scale parameter in place of scale 1 can be compensated simply by multiplying all regression vectors by the same scale. Together with the previous point, this shows that the use of a standard extreme-value distribution (location 0, scale 1) for the error variables entails no loss of generality over using an arbitrary extreme-value distribution. In fact, the model is
   nonidentifiable
   (no single set of optimal coefficients) if the more general distribution is used.
3. Because only differences of vectors of regression coefficients are used, adding an arbitrary constant to all coefficient vectors has no effect on the model. This means that, just as in the log-linear model, only K-1 of the coefficient vectors are identifiable, and the last one can be set to an arbitrary value (e.g. 0).

Actually finding the values of the above probabilities is somewhat difficult, and is a problem of computing a particular order statistic (the first, i.e. maximum) of a set of values. However, it can be shown that the resulting expressions are the same as in above formulations, i.e. the two are equivalent.

Estimation of intercept

When using multinomial logistic regression, one category of the dependent variable is chosen as the reference category. Separate odds ratios are determined for all independent variables for each category of the dependent variable with the exception of the reference category, which is omitted from the analysis. The exponential beta coefficient represents the change in the odds of the dependent variable being in a particular category vis-a-vis the reference category, associated with a one unit change of the corresponding independent variable.

Likelihood function

The observed values ${\displaystyle y_{i}\in {0,1,\dots K}}$ for ${\displaystyle i=1,\dots ,n}$ of the explained variables are considered as realizations of stochastically independent, categorically distributed random variables ${\displaystyle Y_{1},\dots ,Y_{n}}$.

The likelihood function for this model is defined by:

${\displaystyle L=\prod _{i=1}^{n}P(Y_{i}=y_{i})=\prod _{i=1}^{n}\left(\prod _{j=1}^{K}P(Y_{i}=j)^{\delta _{j,y_{i}}}\right),}$ where the index ${\displaystyle i}$ denotes the observations 1 to n and the index ${\displaystyle j}$ denotes the classes 1 to K. ${\displaystyle \delta _{j,y_{i}}={\begin{cases}1{\text{ for }}j=y_{i}\\0{\text{ otherwise}}\end{cases}}}$ is the Kronecker delta.

The negative log-likelihood function is therefore the well-known cross-entropy: :${\displaystyle -\log L=-\sum _{i=1}^{n}\sum _{j=1}^{K}\delta _{j,y_{i}}\log(P(Y_{i}=j)).}$

Application in natural language processing

In

• Logistic regression
• Multinomial probit

References