Everything a language model does is output a probability distribution over the next token. Here you train a real character-level model in your browser on a corpus you can read, watch the actual distribution it produces, and generate text by sampling from it, one draw at a time.
One job: P(next token | everything so far)
Strip away the chat interface and the billions of parameters, and a language model is a function with one output: given the text so far, a probability for every entry in its vocabulary as the next token. It never emits a sentence. Sentences appear because that function is called in a loop: predict, sample one token, append it, predict again.
model(context) -> [p(token_1), p(token_2), ..., p(token_V)] sums to 1
generate(prompt):
loop:
dist = model(text) the whole model is inside this call
next = sample(dist) one uniform draw picks a token
text = text + nextA distribution, not an answer
The model output is the full bar chart, one probability per vocabulary entry. Which bar becomes text is a separate, surprisingly small decision made by the sampler, not by the model.
Generation is repeated sampling
Each new token is one draw from the current distribution, and the draw changes the context, which changes the next distribution. That feedback loop is all that text generation is.
Characters here, tokens in real LLMs
Real LLMs predict subword tokens from vocabularies of 50k to 200k entries using a neural network. This guide uses a count-based character model so every probability on screen is checkable by hand. The loop is identical.
The entire training corpus, in plain sight
Training this model means counting: slide a window over the text below and tally which character follows each context. Nothing is hidden, so you can verify any bar in the machine by counting occurrences here yourself.
the cat sat on the mat and the dog slept by the door. the rain in spain stays mainly on the plain. she sells sea shells by the sea shore. the queen quietly quizzed the quick squid. a model that reads this text learns which letter tends to follow which letter. the model then writes new text one letter at a time by guessing the next letter again and again. the more text the model reads the better the guesses get. there is no plan and no meaning inside the machine. there is only a table of counts and a roll of the dice.
Corpus length
522
characters
Vocabulary
26
distinct characters
Training windows
521
(context, next char) pairs at order 1
Contexts learned
26
distinct 1-character contexts
Model order: how much context does a prediction see
More context means sharper, more confident distributions but fewer observations per context. Switch from bigram to trigram and watch the entropy readout below: it drops for almost every prefix in this corpus, though the starting prefix happens to be a rare exception where it ticks up. Try a few of your own. A real LLM conditions on thousands of tokens of context, learned by a neural network instead of a count table, which is why it does not run out of observations the way this table does.
The machine: type a prefix, read the distribution
Every bar below is count divided by total for the highlighted context, recomputed live from the corpus. Try a prefix ending in "q" to find a certain model, or one ending in "␣" (space) to see it hedge across many word starts. Input is limited to the corpus alphabet: letters, space, period.
Editing the prefix restarts generation. Allowed: a to z, space, period.
Current sequence (context highlighted)
the␣
Conditioning on "␣", seen 103 times in the corpus.
Entropy
3.58
bits of uncertainty
Top candidate
t
at 27.2% of 17 options
P(next character | "␣")
click any bar to force that character
The same distribution as one cumulative strip from 0 to 1
Sampling means drawing a uniform number between 0 and 1 and taking whichever slice it lands in. Bigger slice, more likely character.
Generated so far (prefix dim, sampled text bright)
the
Every bright character above was one draw from a distribution you could inspect at the moment it happened. The text looks word-like with almost no machinery because even 1 or 2 characters of context carry real statistical signal. That, scaled up, is the whole trick.
Everything downstream manipulates this bar chart
Temperature, top-k, and top-p do not change the model at all. They reshape or truncate exactly the distribution you have been staring at, before the uniform draw happens. The temperature and top-k guide picks up at the precise point this guide ends: after the model has produced its bars, before the dice are rolled.
And the choice of what a "token" is, characters here, subword pieces in production models, is its own design problem with real consequences for the distribution the model must output. That is the tokenization guide, the natural next step.