Before any model can read, text has to become data, and the recipe is older than AI: chop the text into tokens, normalize them, and count. Type in the sandbox and watch every keystroke change what the computer thinks your words are.
Text is not data until you decide what a word is
A string of characters has no rows and no columns. To analyze it, you make three choices: where one token ends and the next begins (tokenization), which surface forms count as the same word (normalization), and what to tally (counting). Each choice is a judgment call, and each one changes every number downstream. Modern language models push the same idea further by splitting words into subword pieces; that story belongs to the tokenization guide in the LLM category. This guide is the classic craft that came first: whole-word tokens, honest normalization, and bag-of-words counts.
How this playground stays honest: every count, bar, vocabulary size, and overlap score on this page is recomputed in your browser, on every keystroke, from the exact text sitting in the input boxes. Nothing is precomputed.
The sandbox below runs the full pipeline live. Edit the text and watch tokens light up as you type. Then flip each normalization switch and watch the frequency chart reshuffle: lowercasing merges Count with count, the stop list deletes the glue words, and the crude stemmer folds counted, counting, and counts into one row.
220 of 2000 characters
What the tokenizer sees
Underlined spans are word tokens: runs of letters and digits, with one apostrophe allowed inside. Dim characters (spaces, punctuation) never make it into the data. Struck-through tokens were removed by the stop list.
After the pipeline: the tokens that get counted
Teal chips show a token the pipeline rewrote, with the original struck through before the arrow. These final forms are what the frequency chart counts.
Normalization pipeline (applied in this order)
a, an, the, and, or, but, if, of, to, in, on, at, by, for, with, from, as, is, are, was, were, be, been, am, it, its, this, that, these, those, i, you, he, she, we, they, them, his, her, their, my, your, our, not, no, so, do, did, does, have, has, had, will, would, can, could, than, then, there, here, what, when, who, how, all, some, such, very, just, also, into, over, about, again
Matching is exact, so with Lowercase off, a capitalized "The" slips past the entry "the". Stage order is part of the pipeline design.
Tokens kept
38
Distinct words
32
Stop words cut
0
Word frequency (top 12 of 32 distinct)
This bag-of-words vector is the oldest text representation in the book, and it still powers plenty of search and classification. Flip the switches above and watch rows merge and vanish.
Every switch is a modeling decision
Lowercasing
Usually right, sometimes destructive: it merges Apple the company with apple the fruit, and US the country with us the pronoun. Folding case throws away signal on purpose.
Stop words
There is no universal list. Dropping "not" is harmless for topic counts and catastrophic for sentiment. The list is part of your model, so read it before you trust it.
Stemming
The stemmer here is a handful of suffix rules with guards, and it says so. Even real stemmers like Porter's make ugly stems; the trade is fewer distinct forms for less readable words.
Once text is counts, documents become comparable. The oldest comparison is vocabulary overlap: which distinct words do two documents share? Edit either document below, or paste your own pair, and watch the Jaccard score move. The section 1 switches apply here too, because two documents can only be compared through the same pipeline.
Vocabulary after your pipeline: 21 distinct words
Vocabulary after your pipeline: 21 distinct words
Shared vocabulary (Jaccard overlap)
No normalization active: raw tokens compared exactly.
Jaccard similarity of the two vocabularies
J = |A ∩ B| / |A ∪ B| = 17 / 25 = 0.68
1.00 means identical vocabularies, 0.00 means no word in common. Turn on lowercasing or stemming in section 1 and watch the overlap move: normalization decides which words count as the same word.
The 17 shared words