Logistic Regression - Binary

Binary Logistic Regression

The Logistic Function (Sigmoid)

Maps any real-valued number to a value between 0 and 1:

$$\sigma (z)=\frac{1}{1+e^{-z}}$$

Where $z=w^{T}x+b$ (linear combination of features)

Properties:

• Output range: (0, 1)
• S-shaped curve
• Differentiable everywhere
• Symmetric around 0.5

Model Equation

$$P(y=1|x)=\frac{1}{1+e^{-(w^{T}x+b)}}$$

Prediction:

• If $P(y=1|x)\ge 0.5$ → Predict class 1
• If $P(y=1|x)<0.5$ → Predict class 0

Log-Odds (Logit)

The logit is the natural logarithm of the odds:

$$\text{logit}(p)=\ln \left(\frac{p}{1-p}\right)=w^{T}x+b$$

This transforms the probability to a linear function of features.

Cost Function (Log Loss)

Also called Binary Cross-Entropy Loss:

$$J(w)=-\frac{1}{m}\sum _{i=1}^m[y^{(i)}\log (h(x^{(i)}))+(1-y^{(i)})\log (1-h(x^{(i)}))]$$

Where:

• $m$ = number of samples
• $y^{(i)}$ = actual label (0 or 1)
• $h(x^{(i)})$ = predicted probability

Why not use MSE?

• MSE with sigmoid creates a non-convex optimization problem
• Log loss is convex → guaranteed to find global minimum

Training

Use gradient descent to minimize the cost function:

$$w:=w-\alpha \frac{\partial J}{\partial w}$$

Where $\alpha$ is the learning rate.