Bigger models need more data. Train 16 real networks in your browser, four widths on four data budgets of the same task, and watch the best model size shift as the data grows.
One task, sixteen networks
Every run is a tiny MLP (2 inputs, one tanh hidden layer, 1 output) trained to regress the same 2D function from noisy samples. The grid crosses four hidden widths (2, 8, 32, 128, which is 9, 33, 129, 513 parameters) with four training set sizes (8, 32, 128, 512). Each width reuses the same Xavier initialization (Glorot & Bengio, 2010) across budgets, so within a row the only thing that changes is the amount of data.
Teal: positive, purple: negative (paired with the sign in the formula, not color alone). Gold rings: the 8-point budget. White dots: the rest of the 512-point pool.
The one task every network trains on
y = sin(4 * x1) * cos(4 * x2) + 0.5 * sin(3 * x1 * x2) + noise
The 16-run grid experiment
One click trains 16 real networks in your browser: every width on every data budget, 700 full-batch Adam steps each (learning rate 0.02, weight decay 0.001), identical settings for all runs. Seeds are deterministic, so re-running reproduces exactly the same losses.
| Width (params) | 8 points | 32 points | 128 points | 512 points |
|---|---|---|---|---|
| 2 (9p) | queued | queued | queued | queued |
| 8 (33p) | queued | queued | queued | queued |
| 32 (129p) | queued | queued | queued | queued |
| 128 (513p) | queued | queued | queued | queued |
Validation MSE against the noise-free function (lower is better). Deeper teal = lower loss on a fixed log scale; the number in each cell is the loss itself. ★ marks the best width per budget. Click any finished cell to inspect its run.
Loss vs parameters, one line per data budget
This is the crossover picture. Follow any single line left to right: adding parameters helps until the gold-ringed sweet spot, then flattens or turns back up. Now compare lines: the sweet spot slides right as the budget grows, until it parks. In this grid the step from 128 to 512 points keeps width 32 on top. The right size is not a property of the model, it is a property of the model and the data together.
Run the experiment above to draw this chart from your own 16 training runs.
Where this toy stops
This grid is practitioner intuition, not a law. Two-layer tanh networks on a 2D regression task will not tell you how a billion-parameter transformer behaves, and modern deep learning adds wrinkles this setup deliberately avoids (double descent, regularization schedules, early stopping, architectures whose inductive bias changes the picture). What does carry over is the shape of the tradeoff: parameters are a budget you spend against your data, and the affordable model size grows with the dataset.
The quantitative version of this idea, power-law loss curves, compute-optimal frontiers, and the Chinchilla rule of thumb for LLMs, lives in the Scaling Laws guide. This page gives you the instinct; that one gives you the formulas.