Explore how different activation functions affect gradient flow, training speed, and the dreaded “dead neuron” problem. Select a function to see its properties and training behavior.
Formula
f(x) = max(0, x)
Output Range
[0, ∞)
Gradient Behavior
1 for x > 0, 0 for x < 0: sparse, efficient backprop
Best For
Key Insight
Fast and sparse, but neurons can permanently die if weights push inputs negative.
Stylized curves showing the typical convergence pattern of each function; not data from a real training run
ReLU and Leaky ReLU converge fastest. Sigmoid typically struggles in deep networks due to vanishing gradients. These illustrative curves exaggerate the pattern for clarity.
ReLU-only: neurons receiving always-negative inputs never fire. Illustrative demo: counts are simulated, not from a trained network.
3 out of 8 neurons are dead, never activating regardless of input. This wastes 38% of network capacity. Consider Leaky ReLU to prevent this.
Click a row to explore that function
| Function | Range | Max Gradient | Vanishing Grad? | Dead Neurons? | Best For |
|---|---|---|---|---|---|
ReLU f(x) = max(0, x) | [0, ∞) | 1 (for x > 0) | ✓ | ✗ | Hidden layers in deep networksCNNs and image recognition |
Sigmoid f(x) = 1 / (1 + e⁻ˣ) | (0, 1) | 0.25 (at x=0) | ✗ | ✓ | Binary classification outputProbability estimationLSTM/GRU gates |
Tanh f(x) = tanh(x) | (-1, 1) | 1 (at x=0) | ✗ | ✓ | Hidden layers (older networks)LSTM cell/candidate values |
Leaky ReLU f(x) = x > 0 ? x : 0.01x | (-∞, ∞) | 1 (for x > 0) | ✓ | ✓ | Modern deep networksWhen ReLU causes dead neurons |