Learn AI Create the Future — with Pranay
Main site

The Embedding Family

King minus man plus woman, and why it works less often than you think

Word vectors sometimes support real addition and subtraction, so that king minus man plus woman lands near queen, but the famous trick works far less reliably than the popular examples suggest.

Read these first

On this page 6
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. The honest part
  5. Remember this
  6. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Word vectors sometimes let you do maths on meaning, so that "king" minus "man" plus "woman" lands you near "queen".

Think about a family tree drawn on paper, cousins spaced apart by how closely related they are. Now imagine every relationship on that tree — parent to child, sibling to sibling — gets drawn as the exact same arrow. Wherever it appears on the page.

If that were true, you could take the arrow from "man" to "king" and place its tail on "woman". Its head would land close to "queen". Word vectors sometimes behave this way, at least approximately. That surprising fact was one of the moments that made people take embeddings seriously.

Why it exists

When Word2Vec vectors were first published in 2013, researchers noticed something nobody had trained the model to do. Certain relationships — country to capital, male to female version of a role — showed up as the same direction and distance. No matter which pair of words you picked.

This became the famous demonstration. Take the vector for "king", subtract "man", add "woman", and search for the closest word to the result. The answer, in the original papers, was "queen".

That result spread everywhere — conference talks, textbooks, news articles — as proof that embeddings had captured something like real understanding. The truth turned out to be more complicated, and more interesting.

How it works

   vector("king")  -  vector("man")  +  vector("woman")   ~=   vector("queen")

   Picture it as walking on the meaning-map:

     from "man", walk to "king"          <- this is the "royalty" direction
     take that exact same walk from "woman"
     you land close to "queen"

This only works because the model placed "king" and "queen" a consistent distance apart from "man" and "woman". Not because it understands royalty, gender or families. It is a side effect of how these words get used in similar surrounding contexts, across huge amounts of text.

Where you have already seen it

  • "Did you mean...?" and autocomplete tools sometimes lean on this kind of vector maths.
  • AI demos and conference talks, where this exact example is often the first thing shown.
  • Word-similarity plugins in some search and writing tools, built on the same vector space.

The honest part

This trick works reliably for a small, well-chosen set of famous examples. It works far less reliably across the thousands of relationships you could test. Plural forms, comparative adjectives, rarer country-capital pairs — and especially anything beyond gender and geography.

Researchers tested this systematically, across many relationship types, not only a handful of favourites. Accuracy came out considerably lower than the famous king-queen example suggests. The demo picks its examples carefully. Real coverage is patchier.

There is a second, more serious problem: this same maths surfaces real-world bias baked into the training text. Ask certain analogy questions about professions, and the answers reflect stereotypes present in the data, not any underlying truth. This is not a minor bug — it is examined directly in the developer block below.

Remember this

  • Word vector arithmetic sometimes works, because some word relationships end up as a consistent direction in the vector space.
  • The famous "king minus man plus woman equals queen" example is real. It is one of the better cases, not a typical one.
  • The same trick can surface gender and other social bias baked into whatever text the vectors were trained on.

What to learn next

Developer — Code and libraries.

Testing the trick on a real, pretrained GloVe model — including a case where it reveals bias rather than insight, shown honestly.

Setup

bash
pip install gensim

The famous example, for real

word_arithmetic.py
import gensim.downloader as api

model = api.load("glove-wiki-gigaword-50")

print("king - man + woman:")
for word, score in model.most_similar(positive=["king", "woman"], negative=["man"], topn=5):
    print(f"  {score:.3f}  {word}")

print()
print("paris - france + italy:")
for word, score in model.most_similar(positive=["paris", "italy"], negative=["france"], topn=5):
    print(f"  {score:.3f}  {word}")
Output
king - man + woman:
  0.852  queen
  0.766  throne
  0.759  prince
  0.747  daughter
  0.746  elizabeth

paris - france + italy:
  0.847  rome
  0.777  milan
  0.767  turin
  0.759  venice
  0.757  madrid

Both examples work as advertised. "Queen" is the top result for the first, "rome" for the second. This is real output from a real, published model.

Line by line

positive=["king", "woman"], negative=["man"] is gensim's way of expressing king - man + woman. Positive vectors are added, negative vectors subtracted, and the library searches the whole vocabulary for the closest match to the resulting point, excluding the three input words themselves.

Excluding the input words matters more than it looks. Without that exclusion, "king" itself is often the closest point to king - man + woman, since subtracting and re-adding similar-sized vectors does not move you very far. Most published demonstrations of this trick quietly exclude the query words — a detail worth knowing before treating the result as more impressive than it is.

Where this goes wrong

word_arithmetic_bias.py
import gensim.downloader as api

model = api.load("glove-wiki-gigaword-50")

print("doctor - man + woman:")
for word, score in model.most_similar(positive=["doctor", "woman"], negative=["man"], topn=5):
    print(f"  {score:.3f}  {word}")
Output
doctor - man + woman:
  0.840  nurse
  0.766  child
  0.757  pregnant
  0.752  mother
  0.752  patient

Run the same trick on "doctor" instead of "king", and the top result is "nurse", not another word for "doctor". This is not the model being clever about medical roles — it is the model faithfully reflecting a gender association present in the training text, where "doctor" co-occurred with male-coded context more often, and "nurse" with female-coded context. The vector arithmetic is doing exactly the same thing as the king-queen example. The result here makes the underlying bias visible instead of impressive.

Common mistakes

Treating one working example as proof the model "understands" the relationship. The king-queen result and the doctor-nurse result come from the identical mechanism. One flatters the model. The other exposes it. Neither is evidence of understanding on its own.

Testing only hand-picked, famous examples. Try ten random country-capital pairs instead of Paris-France, and expect a meaningfully lower success rate than the cherry-picked classic examples suggest.

Forgetting the input words must be excluded from the search. Skipping this step — a common mistake when reimplementing the trick from scratch — often returns one of the three input words outright, making the result look far worse than it is.

Try it yourself

Test model.most_similar(positive=["programmer", "woman"], negative=["man"], topn=5) and look closely at the results. Compare what you find against the doctor-nurse example above — this is a well-documented instance of exactly the same pattern in embeddings trained on real-world text.

What to learn next

Researcher — Mathematics and papers.

The formal claim, and how it is usually computed

The classic analogy task asks: given a : b :: c : ?, find:

text
d* = argmax over w in V, w not in {a, b, c} of  cos( v_b - v_a + v_c , v_w )
  • This is the 3CosAdd method (Mikolov et al., 2013): treat the analogy as vector offset, find the nearest neighbour to the resulting point.

Levy & Goldberg (2014) proposed 3CosMul, which reformulates the objective multiplicatively rather than additively:

text
d* = argmax over w of  ( cos(w, b) * cos(w, c) ) / ( cos(w, a) + eps )
  • eps is a small constant avoiding division by zero.
  • This formulation was shown to improve analogy accuracy over 3CosAdd, particularly by reducing the tendency of one dominant term to swamp the others — a single very large cos(w, a) term, for instance, cannot single-handedly veto an otherwise strong candidate the way it can under simple addition.

Measured accuracy, honestly

The original Word2Vec papers reported accuracy in the 60-70% range on their own curated analogy test set (the "Google analogy dataset", covering syntactic relations like plurals alongside semantic relations like capitals). Later, more systematic evaluation told a less flattering story.

Gladkova, Drozd & Matsuoka (2016), Analogy-based Detection of Morphological and Semantic Relations with Word Embeddings: What Works and What Doesn't, introduced the BATS dataset covering 40 relation types across four categories (inflectional morphology, derivational morphology, lexicographic semantics, encyclopedic semantics), and found accuracy varies drastically by relation type — strong on some inflectional patterns, considerably weaker on derivational morphology and many semantic relations, with substantial gaps between what the famous handful of demo examples suggest and what holds across the full relation space.

Linzen (2016), Issues in Evaluating Semantic Spaces Using Word Analogies, raised a methodological point directly relevant to the developer block above: excluding the three input words from the candidate search is not a neutral implementation detail. It measurably inflates reported accuracy relative to leaving them in, since "regressing to one of the inputs" is a common failure mode the exclusion rule hides.

Nissim, van Noord & van der Goot (2020), Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor, went further, arguing that the popular "debiasing" framing of examples like doctor-nurse is itself methodologically shaky — the same analogy machinery, applied slightly differently (for instance, by not excluding the profession word itself from the search), can be made to return "doctor" for both "he" and "she", undermining the common rhetorical use of one cherry-picked bias example as a definitive demonstration either way. Their point is not that bias is absent, but that this specific evaluation method is a poor, easily manipulated tool for measuring it.

Why the geometry is imperfect, structurally

Ethayarajh, Duvenaud & Hirst (2019), Towards Understanding Linear Word Analogies, provided a formal account: 3CosAdd succeeds exactly when the offset vectors for a relation type are approximately parallel and approximately equal in norm across different word pairs — a condition satisfied only loosely, and to varying degrees for different relation types, by embeddings trained with a co-occurrence-based objective. There is no training pressure in GloVe or Word2Vec's actual loss function that directly optimizes for this parallelogram structure; it emerges as a side effect, imperfectly, and its strength varies by exactly how consistently a relation's context patterns behave across word pairs.

Key references

  • Mikolov, T., Yih, W.-T. & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. NAACL. The original analogy demonstration.
  • Levy, O. & Goldberg, Y. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. CoNLL. Introduces 3CosMul.
  • Gladkova, A., Drozd, A. & Matsuoka, S. (2016). Analogy-based Detection of Morphological and Semantic Relations with Word Embeddings: What Works and What Doesn't. NAACL Student Research Workshop.
  • Linzen, T. (2016). Issues in Evaluating Semantic Spaces Using Word Analogies. RepEval Workshop, ACL.
  • Bolukbasi, T. et al. (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. NeurIPS. The paper that popularized the bias framing of this exact analogy task.
  • Nissim, M., van Noord, R. & van der Goot, R. (2020). Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor. Computational Linguistics 46(2), 487–497.
  • Ethayarajh, K., Duvenaud, D. & Hirst, G. (2019). Towards Understanding Linear Word Analogies. ACL.

Current state and open problems

Word analogy accuracy is no longer a primary benchmark for embedding quality in current research — it correlates only weakly with performance on real downstream tasks, and its cherry-picking sensitivity is well documented. It survives mainly as a pedagogical demonstration and a historically important result that shaped early intuitions about what embeddings capture. The bias question it surfaces has not gone away; it has moved to contextual embeddings and large language models, where the evaluation methodology is considerably harder, and where the same core tension identified by Nissim et al. — that a single striking example is weak evidence either for or against systematic bias — still applies with even more force.

What to learn next