Meaning trajectory
About this app
Real word vectors — English GloVe, German fastText (cc.de.300): relations as parallel arrows, king − man + woman ≈ queen with hits, near misses and learned bias, cosine against distance and sentences as a path with next-word candidates from GPT-2 or GerPT2.
Word vectors and analogies
A word embedding maps each word to a point in a space with hundreds of dimensions. It is learned from large amounts of text, so words that appear in similar contexts end up close together. Similarity is usually measured as the cosine of the angle between two vectors. Some relations show up as a roughly parallel offset, such as gender or country to capital. That is why king − man + woman lands near queen; the answer is the nearest neighbour by cosine, with the input words excluded. It often works by a small margin, sometimes fails, and the vectors also pick up stereotypes from their training text.
What the app shows
Every point is a real word: GloVe vectors in the English interface, fastText vectors (cc.de.300) in the German one. The vectors are reduced to 100 dimensions; the picture is a 3D or 2D projection, while all numbers are computed in 100-D. There are three views. Analogy covers gender, capital, past tense, comparative, plural and stereotype relations with hits, near misses and notes. Similarity shows cosine, angle, lengths and distance of two words with neighbour lists. Sentence joins the word vectors into a path, with next-word candidates from GPT-2 or GerPT2.
What to look for
In the analogy view, show more pairs of the same relation and check how parallel the arrows really are. Untick the option that excludes the input words: often an input word itself comes out on top. In the similarity view, opposites such as good and bad turn out to have a high cosine, because they occur in the same contexts. Normalise all vectors to length 1 and distance and cosine give the same ranking. Next-word probabilities exist only for the example sentences; for your own sentences the app shows the path alone.
Frequently asked questions
- Why does king − man + woman give roughly queen?
- In word embeddings some relations appear as an approximately constant offset, for example from male to female. Subtracting the vector for man from king and adding woman lands near queen. The answer is the nearest neighbour by cosine with the input words excluded; without that exclusion king itself often ranks first.
- What is the difference between cosine similarity and Euclidean distance?
- Cosine similarity measures only the angle between two vectors, Euclidean distance also their lengths. Word vectors differ in length, so the two measures can rank neighbours differently. If all vectors are normalised to length 1, distance = √(2 − 2 cos) and both rankings agree.
- Are word embeddings biased?
- Yes. Word vectors reproduce the statistical patterns of their training text, including gender stereotypes about occupations. Analogies such as man is to doctor as woman is to ? can therefore return stereotyped answers. This was described, among others, by Bolukbasi et al. (2016).
- Which data does this app use?
- The English interface uses 100-dimensional GloVe vectors, the German interface fastText vectors (cc.de.300) reduced to 100 dimensions, each limited to a fixed vocabulary. The picture is a projection; every number is computed in 100-D. Next-word candidates in the sentence view were precomputed with GPT-2 or GerPT2.
Subject: Linguistics & phonetics
More from Linguistics & phonetics
- Context shifts meaning
- n-gram text generator
- Language tree
- Tokeniser playground
- Vocal tract — move the tongue, hear the vowel (3D)
- From picture to letter
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps