Activation Function

Activation functions are used to add non-linearity to neural networks, enabling them to learn complex patterns beyond linear relationships.

ReLU (Rectified Linear Unit)

Formula: $\text{max}(z,0)$

Properties:

• Negative inputs → 0
• Positive inputs → unchanged
• Most commonly used in hidden layers
• Overcomes Vanishing Gradient problem

Advantages:

• Computationally efficient
• Helps with gradient flow in deep networks
• Sparse activation (many neurons output 0)

Disadvantages:

• “Dying ReLU” problem - neurons can get stuck outputting 0

Code:

import torch.nn as nn
model = nn.Sequential(
    nn.Linear(64, 128),
    nn.ReLU(),
    nn.Linear(128, 10)
)

LeakyReLU

Formula: $\text{max}(\alpha z,z)$ where $\alpha$ is a small constant (typically 0.01)

Properties:

• Positive inputs → unchanged (same as ReLU)
• Negative inputs → scaled by negative_slope (default 0.01)
• Gradients for negative inputs are non-zero
• Solves the “dying ReLU” problem

Code:

import torch.nn as nn
model = nn.Sequential(
    nn.Linear(64, 128),
    nn.LeakyReLU(negative_slope=0.01),
    nn.Linear(128, 10)
)

Sigmoid

Formula: $\sigma (z)=\frac{1}{1+e^{-z}}$

Properties:

• Output range: (0, 1)
• S-shaped curve
• Smooth gradient

Use Cases:

• Binary classification (output layer)
• Probability estimation
• Logistic regression

Advantages:

• Clear probability interpretation
• Smooth differentiable function

Disadvantages:

• Vanishing gradient problem
• Not zero-centered
• Computationally expensive (exp function)

Softmax

Formula: $\text{softmax}(z_i)=\frac{e^{z_i}}{{\sum }_{j=1}^K{e}^{z_j}}$

Properties:

• Output range: (0, 1)
• Outputs sum to 1 (probability distribution)
• Multi-class generalization of sigmoid

Use Cases:

• Multi-class classification (output layer)
• Classes: 0, 1, 2, 3, 4, 5, …
• Outputs probability distribution across all classes

Code:

import torch.nn as nn
model = nn.Sequential(
    nn.Linear(128, 10),
    nn.Softmax(dim=1)  # Apply softmax across class dimension
)

Tanh (Hyperbolic Tangent)

Formula: $\tanh (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$

Properties:

• Output range: (-1, 1)
• Zero-centered
• S-shaped curve (similar to sigmoid)

Advantages:

• Zero-centered (better than sigmoid)
• Stronger gradients than sigmoid

Disadvantages:

• Still suffers from vanishing gradient

ELU (Exponential Linear Unit)

Formula:

$$\text{ELU}(z)=\{\begin{array}{ll}z& \text{if }z>0\\ \alpha (e^z-1)& \text{if }z\le 0\end{array}$$

Properties:

• Smooth everywhere
• Negative values push mean activation closer to zero
• Can produce negative outputs

Choosing Activation Functions

Hidden Layers

Default choice: ReLU

• Fast computation
• Works well in most cases
• Good for deep networks

When to use LeakyReLU:

• Experiencing dying ReLU problem
• Want to ensure gradient flow for negative values

When to use Tanh:

• Need zero-centered activations
• Recurrent Neural Networks (RNNs)

Output Layer

Binary Classification:

• Use Sigmoid
• Outputs probability between 0 and 1

Multi-class Classification:

• Use Softmax
• Outputs probability distribution across classes

Regression:

• Use Linear (no activation)
• Or ReLU if output should be non-negative

Summary Table

Activation	Range	Use Case	Pros	Cons
ReLU	[0, ∞)	Hidden layers	Fast, no [[Vanishing Gradient	vanishing gradient]]
LeakyReLU	(-∞, ∞)	Hidden layers	Solves dying ReLU	Extra hyperparameter
Sigmoid	(0, 1)	Binary output	Probability interpretation	[[Vanishing Gradient
Softmax	(0, 1) sum=1	Multi-class output	Probability distribution	[[Vanishing Gradient
Tanh	(-1, 1)	RNN, hidden layers	Zero-centered	[[Vanishing Gradient
ELU	(-α, ∞)	Hidden layers	Smooth, negative capable	Slower than ReLU