A language model only outputs a probability distribution over the next token. The text you read depends on how you pick from it. Here a real character n-gram model trains on a visible corpus in your browser, and you decode it three different ways.
Sign in to save progress
The model: character n-gram counts over a visible corpus
Every number in this guide comes from one tiny model trained right here: for each 3-character context in the corpus below, it counts which character comes next and turns the counts into probabilities. If a context was never seen it backs off to shorter contexts. It is character-level, so its most likely text is not always made of real words, which is part of the fun. Decoding always starts from the seed "the␣".
the model can copy the prompt and the model can mix a reply. the model can learn the pattern or the model can quote the data. the model writes the text and the text flows and the text flows on. greedy decoding picks the top token and gets stuck fast. sampling picks a rare token and breaks free. beam search keeps a few paths and drops weak paths.
Corpus length
347 chars
Vocabulary
25 distinct chars
Distinct 3-char contexts
203
Greedy: always take the argmax
Greedy decoding picks the single most likely character at every step. It is deterministic: run it a thousand times and you get this exact text, 40 characters at a cumulative log-prob of -5.85 nats. Watch how it settles into the corpus's most repeated phrasing. Click any character to see the distribution it was chosen from, including everything greedy threw away.
Select any character above to see the exact distribution the model held at that step, and why greedy took what it took.
Beam search: keep the best few prefixes, not just one
Beam search expands the top 3 prefixes with their 3 most likely next characters each step, ranks all candidates by cumulative log-prob, and keeps only the best 3. Gold nodes survive the cut, gray nodes are pruned. Change the width, expand the tree level by level, and compare the full-length numbers below: a wider beam can hold on to a path that starts weaker but ends more likely than the one greedy locks onto.
| Beam | Text after 3 steps | Cumulative log-prob |
|---|---|---|
| #1 | the␣mod | -0.875 |
| #2 | the␣tex | -1.792 |
| #3 | the␣tok | -2.603 |
Level 1
Level 2
Level 3
Decoded to the full 40 characters:
Greedy cumulative log-prob
-5.85 nats
Best beam (width 3) cumulative log-prob
-5.66 nats
best beam text: the␣model␣can␣copy␣the␣model␣can␣learch␣keep
Sampling: draw from the distribution instead of maximizing it
Temperature reshapes the distribution, then top-k, top-p, or min-p decides which characters stay in the candidate pool before the draw. Every batch below really samples 100 continuations of 40 characters with a deterministic rng, then scores each one under the base model. Diversity is the fraction of unique continuations; likelihood is the mean per-char log-prob.
Below 1 sharpens the distribution toward the top character, above 1 flattens it.
Truncation method
The candidate pool right now
Next-char distribution at the seed context, after temperature, with the truncation cut applied: 4 of 4 characters stay in the pool.
Columns: tempered probability, then renormalized share of the kept pool (base probabilities before temperature: m 41.7%, t 33.3%, p 16.7%).
Run the batch at two different settings to see the tradeoff appear.