Tokeniser playground
About this app
Text splits into tokens: byte-level BPE is trained live, the merge slider shows frequent words fusing into one token, plus token details, a language comparison and the share of the context window.
What a tokenizer does
A language model reads neither letters nor whole words. A tokenizer splits the text into tokens and replaces each one with a number, its ID; the model only receives that sequence of numbers. Many models build their vocabulary with byte-pair encoding (BPE): start with the 256 possible bytes, count the most frequent adjacent pair in a training corpus, merge it into a new token and repeat. Common words end up as a single token, while rare words, typos, emojis and under-represented languages stay split into several pieces, in the extreme down to single bytes.
What the app shows
The tokenizer is genuine byte-level BPE, the method used by GPT-2, trained live in your browser on a small corpus that is mostly English with some German and a little Python. Your text appears split into coloured tokens, and you can switch to words or characters. A slider steps the training forwards and backwards merge by merge, listing the most frequent pairs, the new vocabulary tokens and the corpus words that contain them. Statistics give characters, bytes, words, tokens, characters per token and the share of a context window. Clicking a token shows its ID, bytes and the merges it came from.
Things to try
Compare the presets: an English and a German sentence, a very long compound word, typos, emojis, code and numbers, and watch how many tokens each word needs. Set the merge slider to 0 and play the training: at first every token is a single byte, then frequent syllables and words fuse. The “strawberry” example shows why counting letters is hard for a model: the letters inside a token are invisible to it.
Frequently asked questions
- What is a token in ChatGPT and other language models?
- A token is a piece of text the tokenizer treats as one unit, such as a common word, a syllable, a punctuation mark or a single byte. Each token has a number in the vocabulary, and the model processes only those numbers. Pricing and context window size are measured in tokens, not characters or words.
- How does byte-pair encoding (BPE) work?
- BPE starts from a base vocabulary, in byte-level BPE the 256 possible bytes. It finds the most frequent adjacent pair of tokens in the training corpus and merges it into a new token, repeating until the target vocabulary size is reached. New text is tokenized by applying the learned merges in the same order.
- Why do language models struggle to count the r’s in “strawberry”?
- The model does not see the word as a sequence of letters but as one or a few tokens, each with its own ID. Which letters a token contains is not directly part of that input. Letter-level tasks are therefore surprisingly hard for a model even though they are trivial for people.
- Is this the actual GPT tokenizer?
- No. The app uses the same method as GPT-2, byte-level BPE, but trains it live in the browser on a tiny corpus with few merges. GPT-2 has 50,257 tokens and newer models 100,000 to 200,000. The exact splits therefore differ, while the principle is the same.
Subject: Linguistics & phonetics
More from Linguistics & phonetics
- Meaning trajectory
- Context shifts meaning
- n-gram text generator
- Language tree
- Vocal tract — move the tongue, hear the vowel (3D)
- From picture to letter
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps