Tokeniser playground

About this app

Text splits into tokens: byte-level BPE is trained live, the merge slider shows frequent words fusing into one token, plus token details, a language comparison and the share of the context window.

Subject: Linguistics & phonetics

More from Linguistics & phonetics

• Meaning trajectory
• Context shifts meaning
• n-gram text generator
• Language tree
• Vocal tract — move the tongue, hear the vowel (3D)
• From picture to letter

A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps