A loss function compresses model quality into one differentiable number. Drag, toggle, and sweep your way through MSE, MAE, Huber, and cross entropy to see what each one actually tells the optimizer.
The loss is the only voice the optimizer hears
Training a model means turning "how good is this?" into a single differentiable number, then pushing that number downhill. Everything gradient descent does, every weight update backpropagation delivers, starts from the derivative of the loss. Pick a loss that punishes the wrong things and the model will diligently learn the wrong things. The three experiments below make that concrete: first for regression, then for classification.
Two properties matter in practice: what the loss says when the model is slightly wrong, and what it says when the model is catastrophically wrong. Different losses answer those two questions very differently.
Interactive A: One Prediction, Three Losses
Drag the prediction along the number line (or focus it and use the arrow keys). The target is fixed at 2. Every value below is computed live from e = prediction - target.
MSE (squared error)
L = e²
25.00
gradient: -10.00
grows linearly with e
MAE (absolute error)
L = |e|
5.00
gradient: -1.00
constant magnitude 1
Huber (δ = 1.0)
L = ½e² or δ(|e| - ½δ)
4.50
gradient: -1.00
capped at ±1.0
Loss as a function of the prediction (same three functions, plotted over the whole line; the y axis is capped at 25, so MSE simply leaves the chart)
Interactive B: The Outlier Stress Test
Twelve fixed points near y = 2 + 0.8x. Toggle the outlier and watch which fitted line chases it. Both lines and both error totals are recomputed live from the visible points.
| Fitted line | Total squared error | Total absolute error |
|---|---|---|
| Least-squares line (minimizes MSE) | 0.00 (lower) | 0.03 |
| L1-optimal line (minimizes MAE) | 0.00 | 0.03 (lower) |
Each line wins its own metric: least squares always has the lowest total squared error, the L1 line always has the lowest total absolute error. The lesson is in the slope: the outlier lifts one point from y = 9.79 to y = 24, a jump of 14.2 units whose squared penalty (202 if the line ignored it) outweighs every other residual combined, so the red line tilts toward it. Under absolute error the same jump only costs 14.2, so the dashed line barely moves. Fit method: least squares is the closed-form solution; the L1 line is found by a coarse-to-fine grid search over slope and intercept, computed live in your browser.
Interactive C: Cross Entropy vs Squared Error
p is the probability your classifier assigns to the true class. Sweep it down below 0.1 and watch which loss still produces a usable training signal. Both curves and all four numbers are computed from the slider value.
Cross entropy
0.36 nats
gradient wrt logit (sigmoid, y = 1): p - 1 = -0.30
Squared error
0.09
gradient wrt logit (sigmoid, y = 1): 2(p - 1)·p·(1 - p) = -0.126
The gradient story generalizes: combine softmax with cross entropy and the gradient of the loss with respect to each logit collapses to p - y, the predicted probability minus the one-hot target. That clean, non-vanishing gradient (not tradition) is why softmax plus cross entropy is the standard pairing for classification.
Why cross entropy pairs with softmax and sigmoid
It is gradient behavior, not superstition. A sigmoid or softmax squashes logits into probabilities, and its derivative goes to zero at both ends of the squash. Chain a squared error through that derivative and the gradient dies exactly when the model is confidently wrong, the one moment you need it most. Cross entropy contains a log that cancels the squashing function's derivative, leaving the clean logit gradient p - y. This is the same loss you meet in logistic regression, where it goes by the name log loss.
sigmoid + squared error: dL/dz = 2(p - y) · p(1 - p) [vanishes near p = 0 or 1] sigmoid/softmax + CE: dL/dz = p - y [never vanishes]
Huber: the compromise
Quadratic near zero for a smooth, proportional gradient on small errors; linear beyond delta so outliers cannot dominate a batch. You get MSE's stable convergence and most of MAE's robustness, at the cost of one hyperparameter to tune.
Loss vs metric
You optimize the loss; you report metrics. Accuracy, F1, and AUC are step functions of the predictions, so their gradients are zero almost everywhere and useless for training. Train on cross entropy, then judge the model with the metrics that stakeholders care about.
Model evaluation guideChoosing in practice
Regression with trusted data: MSE. Regression with heavy tails or sensor glitches: Huber or MAE. Classification: cross entropy, essentially always. If you find yourself inventing a loss, first check whether the problem is really the data.