Models and math need numbers, but "Riverside" is not one. Every way of fixing that smuggles in an assumption, and the wrong one quietly invents an order that does not exist or leaks the answer into the features.
Sign in to save progress
An encoding is a modeling assumption in disguise
A linear model cannot multiply a coefficient by "Old Town", and neither can a distance metric or a gradient. So categorical columns must become numeric ones. The catch: numbers carry structure that names never had. Numbers have an order, magnitudes, and distances, and your model will happily use all three even when they are artifacts of the encoding rather than facts about the world.
How this playground stays honest: everything is computed live in your browser from the literal 12-row table below (4 neighborhoods) and, in the last section, a 60-value sample from a seeded generator (seed 20260708). Both least-squares fits are solved exactly with the normal equations. Nothing is precomputed or faked.
The encoding lab
12 houses · 4 neighborhoods · rebuilt liveSame twelve houses, four different numeric matrices. Click through every encoding and watch the columns it hands to the model. Each one answers the same question, "what number is a neighborhood?", with a different assumption.
| Neighborhood | Price ($k) | is_Docklands | is_Hillcrest | is_Old_Town | is_Riverside |
|---|---|---|---|---|---|
| Riverside | 372 | 0 | 0 | 0 | 1 |
| Old Town | 300 | 0 | 0 | 1 | 0 |
| Docklands | 240 | 1 | 0 | 0 | 0 |
| Hillcrest | 405 | 0 | 1 | 0 | 0 |
| Old Town | 322 | 0 | 0 | 1 | 0 |
| Riverside | 388 | 0 | 0 | 0 | 1 |
| Docklands | 262 | 1 | 0 | 0 | 0 |
| Old Town | 291 | 0 | 0 | 1 | 0 |
| Hillcrest | 430 | 0 | 1 | 0 | 0 |
| Riverside | 401 | 0 | 0 | 0 | 1 |
| Old Town | 315 | 0 | 0 | 1 | 0 |
| Hillcrest | 418 | 0 | 1 | 0 | 0 |
Four categories became four columns, and every row is a single 1 among 0s. No category is claimed to be bigger, closer, or better than another. That neutrality is exactly what the width buys you.
The one-hot column explosion
One-hot is the safest default at low cardinality, but its cost is width: k categories means k columns (k minus 1 if you drop a reference level). Drag the slider from 4 neighborhoods toward zip-code territory and watch the matrix widen while the rows available to estimate each column shrink. Push it past 20 columns to see it tip.
One-hot columns
4
one per category
Ordinal / freq / target columns
1
always, at any k
Rows per column (n = 200)
50.0
200 / 4
Status
Manageable
threshold: more than 20 columns
This is one face of the curse of dimensionality: parameters grow with k while the data does not. High-cardinality columns are exactly where the compact encodings, and eventually dimensionality reduction, earn their keep.
The fit lab: what a fake order costs
Ordinal encoding is perfect when the order is real (small, medium, large). Here the order is alphabetical, so a linear model on the code is forced to draw one straight line through neighborhoods whose mean prices are not monotone in that order. One-hot gives the model a free mean per neighborhood instead.
Two least-squares fits on the same 12 rows, solved exactly with the normal equations. One regresses price on the single ordinal code. The other regresses price on the four one-hot columns. Before running it, note that the category mean prices go up, down, then up again along the alphabetical codes.
When the encoding borrows the answer: target encoding on tiny categories
Target encoding replaces each category with the mean of the label inside it. With plenty of rows per category that mean is signal. With one or two rows per category the mean IS the label, so the feature memorizes noise. Below, the prices are pure noise with zero real category effect: slide k up and watch the train fit climb while the held-out slice, split deterministically, gets worse.
Train R² at k = 6
0.071
Held-out R² at k = 6
-0.076
Avg train rows per category
6.7
Smallest category (train rows)
6
How this stays honest: the 60 prices are 350 plus uniform noise of ±40, drawn once from a seeded generator (seed 20260708), and the category has zero real effect on price. Rows with index mod 3 equal to 2 are held out (20 of 60). Each category is encoded with its train-only mean price, a line is fit by exact least squares on the train slice, and both R² values are recomputed live for every k.
This gap is a leakage signature: the feature knows things about the training labels that no honest feature could know at prediction time. Practitioners blunt it with out-of-fold means and smoothing toward the global mean. The full story, including nastier variants, lives in the data leakage guide.
Choosing in practice
| Encoding | Assumption it injects | Reach for it when |
|---|---|---|
| One-hot | None (categories independent) | Low cardinality, linear models |
| Ordinal | A real, meaningful order | Genuinely ordered levels only |
| Frequency | Prevalence predicts the target | High cardinality, quick baseline |
| Target | Category means generalize | High cardinality, with out-of-fold means and smoothing |