Build a strong model one weak tree at a time. Press Step to fit a small tree to the current residuals and add a shrunken copy of it to the ensemble. Every curve and error number on this page is computed live from the 60 training points in front of you.
Sign in to save progress
The idea: keep fixing what is still wrong
A boosted model is a running sum of small trees. It starts from a constant, the mean of the targets. Each round measures the residuals, the part of y the current ensemble still gets wrong, fits one depth-limited tree to them, and adds that tree scaled by the learning rate. For squared loss the residual is exactly the negative gradient, which is where the name comes from.
Weak learners on purpose
The trees here are the same regression trees from the Decision Trees guide, kept deliberately shallow. One shallow tree underfits badly; a sequence of them, each correcting the last, does not.
Exact split search
Every split in every tree is found by exhaustively scanning all candidate thresholds and picking the one that minimizes the summed squared error of the two sides. Nothing on this page is approximated or scripted.
Sequential, not parallel
Random forests average many independent trees to cut variance. Boosting is the opposite bet: trees are built in order, and each one only makes sense given the ensemble before it.
F_0(x) = mean(y) = 0.62 start from a constant r_i = y_i - F_m(x_i) residuals after m trees F_m+1(x) = F_m(x) + lr * tree(x) tree is fit to r by exact split search
The data is synthetic and fully disclosed: 60 training and 60 held-out test points drawn from y = f(x) + noise with a fixed seed, where f is a smooth curve with one step. Gradient boosting was formalized by Friedman (2001), "Greedy Function Approximation: A Gradient Boosting Machine", Annals of Statistics 29(5). XGBoost, LightGBM, and CatBoost are industrial implementations of the same idea.
Step 1: boost, one tree at a time
Each press of Step fits one tree to the current residuals and adds lr times it to the model. Changing the learning rate or the tree depth refits the whole sequence from round zero, so the chart always shows a genuine boosting run for the settings you chose.
Weak learner depth
Trees in ensemble
0
Train MSE
1.382
Test MSE (60 held-out)
1.460
Residual RMSE
1.176
Step 2: watch the residuals collapse
This panel shows what the next tree sees: the residuals of the current ensemble, on a fixed scale. At round zero they are just y minus the mean. Boost a few rounds and they shrink toward the zero line, which is exactly why each new tree has less and less left to fix. The dashed line is the real tree the next Step would add, already fit to these residuals.
Step 3: many weak trees vs one deep tree
The natural objection: why bother with sixty tiny trees when one big tree could fit the data directly? Toggle the overlay to fit a single depth 6 tree to the same 60 points. It appears as a dashed line in the chart above and as a reference line below. Watch its two errors: excellent on the training set, worse than your boosted ensemble on the held-out test set. That gap is variance, and shrunken weak learners are how boosting avoids it.
Try to break it
Push the learning rate to 1.00, set depth 3, and boost to 60 rounds. Train MSE keeps falling while test MSE bottoms out and turns upward: the ensemble starts memorizing noise. Small learning rates trade more rounds for a flatter, later minimum, which is why real systems pair a low rate with early stopping. That failure mode has its own guide next.
Step 4: the defining question
Each new tree in gradient boosting is fit to: