Slide the dropout probability and watch neurons randomly deactivate. Understand how dropout prevents overfitting by training an ensemble of thinned networks simultaneously.
Each hidden neuron is dropped independently with probability 50% at every training step, so about 50% are dropped on average and the exact count varies. In this demo, input neurons use a reduced rate (15%) and output neurons are never dropped.
p = 0.5 comes from the original dropout paper (Srivastava et al. 2014) for fully connected layers. Modern convolutional and transformer architectures typically use much lower rates (around 0.1 to 0.3) or rely on batch/layer normalization and other regularizers instead.
Training Mode: Neurons randomly dropped each forward pass, forcing robust representations
Inference mode: All neurons are active but activations are scaled by (1 − p) to compensate for the extra neurons present compared to training. Try toggling between modes to see the difference.
Training and validation losses on the same dataset, with and without dropout. Illustrative curves showing the typical qualitative pattern, not a logged training run.
Each training step uses a different random subset of neurons, effectively training a unique sub-network. At inference, the full network approximates the average of all sub-networks.
Modern frameworks use inverted dropout: multiply activations by 1/(1−p) during training so no rescaling is needed at inference time. This means the same network weights work unchanged at test time, cleaner and faster.