Categorical Logistic Regression

Polychotomous outcomes

• When the outcome has more than two values with no implied ordering
• Examples: political party affiliation, preferred transport mode, species
• Cannot use binary logistic regression (too few categories)
• Cannot use ordinal logistic regression (no ordering)

The model

Choose category 1 as reference. For \(l = 2, \ldots, L\):

\[ \log\!\left(\frac{\Pr(y_i = l)}{\Pr(y_i = 1)}\right) = \beta_{l0} + \sum_k \beta_{lk} x_{ki} \]

• \(L - 1\) sets of coefficients, one per non-reference category
• Each set is a standard log-odds against the reference

The softmax representation

Implied probabilities:

\[ \Pr(y_i = l) = \frac{e^{z_{li}}}{1 + \sum_{l^\prime=2}^L e^{z_{l^\prime i}}} \]

where \(z_{li} = \beta_{l0} + \sum_k \beta_{lk} x_{ki}\) and \(z_{1i} = 0\)

• All probabilities positive, sum to 1
• Probabilities change nonlinearly with predictors

Fitting with multinom

library(nnet)
data("admit", package = "pscl")

M_9 <- multinom(score ~ gre.quant, data = admit, trace = FALSE)
summary(M_9)

• One row of coefficients per non-reference category
• Coefficients are log odds relative to the reference category

Predicted probabilities

admit_new <- tibble(gre.quant = seq(300, 800, by = 100))
add_predictions(admit_new, M_9, type = "prob")

Manual softmax calculation

For gre.quant = 600:

z <- coef(M_9) %*% c(1, 600)
z_all <- c(0, z)
round(exp(z_all) / sum(exp(z_all)), 3)

This reveals exactly what the softmax function computes

Relationship to binary logistic regression

• When \(L = 2\), categorical logistic regression reduces exactly to binary logistic regression
• multinom and glm(..., family = binomial) give the same estimates
• Categorical logistic regression is the general case

Summary

• Categorical logistic regression models log odds of each category against a reference
• \(L-1\) sets of coefficients for \(L\) categories
• Probabilities computed via the softmax function
• multinom from nnet fits the model in R