Skip to content
All apps

n-gram text generator

About this app

Text from counts of the last n − 1 tokens: count table P(w|h), the path from gibberish to plausible text to verbatim copying, the explosion of possible contexts, smoothing and backoff; German and English corpora.

How an n-gram model works

An n-gram model predicts the next token from the previous n − 1 tokens only. Its probabilities are plain counts from a text corpus: P(w | context) = count(context + w) / count(context). Generating text means looking up the row for the current context, sampling according to the counts, appending the token and repeating. As n grows there are V^(n−1) possible contexts for a vocabulary of size V, yet a corpus of N tokens contains at most N of them, so almost every cell of the table is zero. Smoothing such as add-one, or backing off to shorter contexts, handles contexts that were never seen.

What the app shows

The app generates text from one of five small corpora (cats, fairy tale, space, cookbook, news) or all five combined, in English or German, at character or word level. Every step shows the count table for the current context together with the formula. A chart compares possible and actually observed contexts for each n. Passages copied verbatim from the corpus are highlighted, and tapping one shows its source in the original text. Figures such as candidates per step (perplexity) and the share of copied text put the output in context.

Things to try

At character level, raise n step by step from 1 to 6: random letters become syllables, then real words, then long copied passages. At word level much of the output is already a copy at n = 3, because the corpus is small. Type your own opening that never appears in the corpus and switch between plain counting, backoff and add-one. Plain counting gets stuck, add-one floods rare contexts with random tokens, and backoff falls back to shorter contexts.

Frequently asked questions

What is an n-gram language model?
An n-gram model is a statistical language model that estimates the probability of the next word or character from the n − 1 preceding tokens only. The probabilities come from counting in a corpus. A bigram model looks back one token, a trigram model two.
Why does an n-gram model copy its training text when n is large?
Long contexts usually occur only once in a corpus, so they have exactly one continuation. At those points the model has no choice and reproduces the original text word for word. The smaller the corpus, the sooner this memorisation sets in.
What is the difference between add-one smoothing and backoff?
Add-one (Laplace) smoothing adds 1 to every count so that no token has probability zero, at the cost of giving a lot of probability to tokens never seen in that context. Backoff drops the oldest token of an unseen context and looks up a shorter one instead. Both address the many zeros in an n-gram count table.
What can I do in this app?
You pick a corpus, a language, character or word level and an order n from 1 to 6, then generate text step by step or continuously. The count table shows every sampling step and a chart shows how the number of possible contexts explodes. With your own opening and the three smoothing options you can see what happens with unseen contexts.

Subject: Linguistics & phonetics

More from Linguistics & phonetics

A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps