Loss falls predictably with parameters, data, and compute, which is why frontier labs can plan a model before training it. Here you drive the actual fitted Chinchilla law: pick a FLOPs budget, split it between model size and tokens, and read the predicted loss live.
Sign in to save progress
The law, with its published constants
Kaplan et al. 2020 (arXiv:2001.08361) showed language model loss follows smooth power laws in model size, data, and compute. Hoffmann et al. 2022 (arXiv:2203.15556, the Chinchilla paper) fitted a two-term parametric form to hundreds of training runs. Every number in this guide is computed from that fit:
L(N, D) = E + A / N^alpha + B / D^beta E = 1.69 A = 406.4 B = 410.7 alpha = 0.34 beta = 0.28 constants: Hoffmann et al. 2022, arXiv:2203.15556 (Approach 3 fit) compute: C = 6 * N * D FLOPs (Kaplan et al. 2020 approximation)
E: the floor
The irreducible loss. Even an infinite model on infinite data cannot beat the intrinsic entropy of natural text. No amount of scale buys past it.
A / N^alpha: model term
The penalty for a finite parameter count. It shrinks as a power law in N, so every extra decade of parameters buys a predictable, and shrinking, slice of loss.
B / D^beta: data term
The penalty for finite training data. Because both terms decay as power laws, a fixed compute budget has a best split between the two: the compute-optimal allocation.
Spend a budget: the iso-FLOP valley
Fix a compute budget C and every choice of N drags D = C / 6N with it: a bigger model must train on fewer tokens. The curve below is the predicted loss along that constraint. Its valley is the compute-optimal point. Slide the budget and the whole valley shifts; slide the allocation and watch how much loss a lopsided split costs you.
Log scale, 1e19 to 1e26 FLOPs. The valley of the curve is the compute-optimal allocation for this budget.
0.00 is the compute-optimal N. Negative: smaller model, more tokens. Positive: bigger model, fewer tokens. C stays fixed.
Parameters N
14.6B
Tokens D = C/6N
1.14T
Tokens per param
78.2
Predicted loss
2.005
Optimal loss
2.005
Loss overhead
+0.000 (0.0%)
Frontier allocation at this budget: 14.6B parameters on 1.14T tokens (78 tokens per parameter). All values are computed live from L(N, D) with the Hoffmann et al. 2022 constants.
The frontier, and where real models landed
Sweeping the budget traces the compute-optimal frontier through the (N, D) plane. The purple dots are real published models, plotted at their published parameter and token counts. GPT-3-era models sit far to the parameter-heavy side of the frontier; Chinchilla was the correction. Llama-generation models are famously trained far past the 20-tokens-per-parameter rule of thumb because inference cost rewards small models, yet against the more data-heavy Approach 3 frontier drawn here even they land on the parameter-heavy side, a tension the fine print below unpacks.
Inspect a model
Select a model above to see its published parameter and token counts, its computed budget, and how far the Chinchilla fit places it from the compute-optimal frontier.
Read the fine print