Context shifts meaning
About this app
Static vs. contextual word vector in 3D: an ambiguous word starts between two islands of meaning and is pulled layer by layer by the context words, in proportion to their attention weights.
Static and contextual word vectors
A static embedding such as word2vec gives every word exactly one vector, whatever sentence it appears in. For an ambiguous word like bank (financial institution or riverbank) that vector sits between the senses, closer to the more frequent one. A transformer starts from such a vector and rewrites it layer by layer. Through attention the word looks at the other words in the sentence, and each one pulls the vector a little towards itself, weighted by its relevance. The result is a contextual vector that carries the sense meant in that sentence. Function words such as “the” or “at” contribute little.
What the app shows
Two islands of meaning sit in a 3D space with the ambiguous word between them. You can choose bank, bat, apple and mouse (in German Bank, Schloss, Birne and Maus), each with example sentences. In contextual mode the vector travels from layer 0 to layer 6, while ropes and bars show how strongly each context word pulls through attention. The reading panel gives the share of each sense and the cosine. In static mode the vector stays in the same place in every sentence. The space is an illustrative sketch with hand-placed coordinates and simplified attention, not measurements from a real model.
Things to try
Pick a word and a sentence, switch from static to contextual and play the layers. Use “Show all sentences at once” to see the same word land on different islands depending on the sentence. Assemble your own context from the context words and watch which ones pull hard and which barely move the vector. Without any context the vector stays where it is.
Frequently asked questions
- What is the difference between static and contextual embeddings?
- Static embeddings such as word2vec or GloVe assign each word one fixed vector regardless of the sentence. Contextual embeddings from transformer models such as BERT or GPT compute a separate vector for each occurrence of a word, depending on the surrounding text. This lets a model tell the senses of an ambiguous word apart.
- How does a transformer resolve word ambiguity?
- In each layer, self-attention computes how much a word attends to every other word in the sentence and mixes their information into its vector accordingly. Over several layers the vector of an ambiguous word shifts towards the intended sense. Words such as “loan” or “money” pull “bank” towards the financial meaning.
- What is attention in language models?
- Attention is a mechanism in which each token computes weights for all other tokens, usually via a softmax over similarity scores. These weights are used to form a mixture of the other tokens’ information. A word can thus focus on the words that matter for its meaning.
- Does the app show data from a real language model?
- No. The coordinates are hand-placed and limited to three dimensions, and the layers use simplified attention. Real models have hundreds of dimensions and 12 to 96 layers. The pattern of context words pulling a vector towards the right sense layer by layer is the same.
Subject: Linguistics & phonetics
More from Linguistics & phonetics
- Meaning trajectory
- n-gram text generator
- Language tree
- Tokeniser playground
- Vocal tract — move the tongue, hear the vowel (3D)
- From picture to letter
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps