Logistic Regression - Multiclass

Multiclass Logistic Regression

Softmax Function

Generalizes logistic regression to multiple classes:

$$P(y=k|x)=\frac{e^{z_k}}{{\sum }_{j=1}^K{e}^{z_j}}$$

Where:

• $z_k=w_k^{T}x+b_k$ for class k
• K = number of classes

Properties:

• Outputs sum to 1
• All outputs are positive
• Can be interpreted as probabilities

Multiclass Cost Function

Categorical Cross-Entropy Loss:

$$J(w)=-\frac{1}{m}\sum _{i=1}^m\sum _{k=1}^K{y}_k^{(i)}\log (h_k(x^{(i)}))$$

Where $y_k^{(i)}$ is 1 if sample i belongs to class k, else 0 (one-hot encoding).

One-vs-Rest (OvR)

Alternative approach for multiclass:

• Train K binary classifiers
• Each classifier: class k vs all other classes
• Predict class with highest probability

One-vs-One (OvO)

Another multiclass approach:

• Train $\frac{K(K-1)}{2}$ binary classifiers
• Each classifier: class i vs class j
• Use voting to determine final class