Watch information flow through recurrent cells step by step. See how LSTMs solve the vanishing gradient problem with gates and cell state.
Sign in to save your progress
Core Formula
hₜ = tanh(Wₓxₜ + Wₕhₜ₋₁ + b)Sequence: “The cat sat on the mat”. Step through to watch the RNN process each token.
During backpropagation, gradients are multiplied at every timestep, starting from the loss at the end of the sequence. In RNNs, multiplying by values <1 repeatedly causes them to shrink to near-zero by the time they reach the earliest tokens, so the network struggles to learn from early inputs. Along the LSTM cell-state highway, the gradient is scaled only by the forget gate at each step: when the network chooses to keep its memory (forget gate near 1), gradients pass back almost unchanged, and they shrink mainly where the network deliberately forgets. The LSTM line below is computed from the same forget-gate values shown in the cell diagram.
Historical Context
LSTMs are no longer the state of the art: Transformers (using self-attention) replaced them in 2017 with the landmark “Attention Is All You Need” paper. But understanding LSTMs gives critical intuition for why attention was needed: sequential processing is slow, and long-range dependencies are hard to capture through gating alone.