A pretrained model only continues text. Here you train two real bigram models in your browser, blend demonstrations into the base distribution, add preference pressure with a KL tether, and watch token probabilities shift and diversity collapse as the tether loosens. Every number is computed live.
Sign in to save progress
All the training data, on one screen
A base model is a mirror of its corpus: it learns what word tends to follow what word, and nothing about being useful. Below is the entire training signal for this guide: a raw web-style corpus, a set of assistant demonstrations, and six preference judgments. The two bigram models are trained on these lines in your browser, so every probability in the sandbox traces back to text you can read.
14 lines, 112 tokens. The base bigram model is trained on exactly these lines, nothing else.
idk the answer is wrong lol
the answer is whatever man
lol the model is broken again
the answer is wrong and the thread is locked
the response is spam and the mods are asleep
honestly the answer is a guess
the model is hype and the demo is fake
first the answer is maybe and then the answer is no
the reply is rude but the joke is funny
the model is fine idk what you mean
the forum is spam and the thread is spam
the answer is on page two lol
the response is late and the thread is dead
the model is guessing and the answer is random
The shaping pipeline, exactly as the sandbox runs it
SFT blend p_sft(w|c) = (1 - lambda) * p_base(w|c) + lambda * p_demo(w|c)
preference p_rl(w|c) proportional to p_sft(w|c) * exp( r(w) / beta )
the exact optimum of max_p E_p[r] - beta * KL(p || p_sft)
reward r(w) = wins(w) - losses(w) counted from the preference pairsContinuation, not conversation
The base model assigns probability to whatever tends to come next in its corpus. Ask it for an answer and it gives you the statistics of forum posts: wrong, whatever, spam. Nothing is broken. It is doing exactly what pretraining trained it to do.
SFT moves the mass
Instruction tuning trains on demonstrations, pulling probability toward assistant-style continuations. Real SFT is gradient descent on demonstration tokens; the lambda slider here is a transparent stand-in, a mixture of the two trained models, with the same qualitative effect: mass moves to the demonstrations.
The KL tether
Preference optimization maximizes reward minus beta times the KL divergence from the SFT model. The formula above is that objective's exact solution: beta is the tether. Tight, and the model barely moves. Loose, and reward swallows the distribution whole.
The distribution-shaping sandbox
Pick a prompt context, then shape the next-word distribution with the two sliders. The gray bars are the trained base model, the teal bars the SFT blend, the gold bars the preference-tuned assistant. All three are recomputed from the corpora and the pairs on every change.
Prompt context
How far instruction tuning pulls the distribution toward the demonstration data.
Currently: tight tether: stays near the SFT model. Slide left and watch the gold bars pile onto the highest-reward words.
Bar order within each row: base, then SFT, then preference. Bars are scaled to the largest displayed probability.
Base model
Entropy 4.66 bits, spread over 25.2 effective words. Top: spam at 12.6%.
After SFT blend
Entropy 5.08 bits, spread over 33.7 effective words. Top: clear at 8.9%. KL from base: 1.62 bits.
After preference stage
Entropy 4.66 bits, spread over 25.2 effective words. Top: helpful at 19.8%. KL from SFT: 0.14 bits.
Try to break it
Switch the context to "... and the ___" and drop beta to 0.1. The gold bars flood onto reward-bearing words like "helpful" even though they never follow "the" in either corpus. That is over-optimization: reward pressure overwhelming the language prior until the output stops being language. The KL penalty exists to prevent exactly this.
Feel the diversity collapse
Entropy numbers are abstract; samples are not. Draw from the base model, then draw from the assistant at your current slider settings, and compare how many distinct words come back. Sampling from at least two different stages completes the third goal above.
No samples drawn yet. Pick a stage and draw: each click samples 12 next words for the context "the answer is ___" from that stage's current distribution, using a seeded deterministic generator that advances on every draw.
Scope note
This guide covers the conceptual internals only: how post-training reshapes a base model's output distribution. The practice of running these methods on real models lives in the Applied AI category: RLHF walks the full reward-model pipeline, DPO trains a policy with the actual preference loss, and LoRA covers how fine-tuning is made cheap in practice.