Word2Vec

The two architectures#

Mikolov et al. (2013a) proposed both in one paper. Given a corpus of tokens w_1 ... w_N and a window size c:

Skip-gram maximises the log probability of context words given the centre word:

(1/N) * sum over t = 1..N of  sum over -c <= j <= c, j != 0  of  log p(w_{t+j} | w_t)

CBOW reverses the conditioning, predicting the centre from the summed context:

(1/N) * sum over t = 1..N of  log p(w_t | w_{t-c} ... w_{t+c})

• N is the number of tokens in the corpus.
• c is the maximum window radius.
• w_t is the token at position t.

CBOW averages the context vectors before the prediction, so one update per window. Skip-gram makes 2c separate predictions per window, so it does more work per token and extracts more signal from rare words.

The softmax that had to go#

The naive parameterisation is:

p(w_O | w_I) = exp(v'_{w_O} . v_{w_I}) / sum over w = 1..V of exp(v'_w . v_{w_I})

• v_w is the input (centre) vector for word w; v'_w is the output (context) vector.
• V is the vocabulary size.
• . denotes the dot product.

The denominator costs O(V) per training example. At V = 10^6 and 10^10 training pairs this is not slow, it is impossible. Both published alternatives attack that denominator.

Hierarchical softmax#

Arrange the vocabulary as the leaves of a binary Huffman tree. The probability of a word is the product of the binary decisions along the root-to-leaf path:

p(w | w_I) = product over j = 1..L(w)-1 of  sigma( [[ n(w, j+1) = left-child(n(w, j)) ]] * v'_{n(w,j)} . v_{w_I} )

• n(w, j) is the j-th node on the path from root to the leaf for w.
• L(w) is the path length, O(log V) under a Huffman code.
• [[ x ]] is +1 when x is true and -1 otherwise.
• sigma is the logistic function.

Cost falls from O(V) to O(log V). Huffman coding gives frequent words short paths, so the average is better than log V. Hierarchical softmax remains preferable for very large vocabularies and for rare-word quality.

Negative sampling#

Mikolov et al. (2013b) replaced the multiclass problem with binary discrimination, a simplification of noise-contrastive estimation (Gutmann & Hyvärinen, 2012). The embeddings lesson states the objective and Levy & Goldberg's proof that it implicitly factorises shifted PPMI. Three implementation details are worth adding here, because all three affect results measurably and none appears in the paper's equations.

The noise distribution. Negatives are drawn from P_n(w) proportional to U(w)^0.75, where U(w) is the unigram frequency. The exponent is empirical. Levy, Goldberg & Dagan (2015) confirmed it transfers to other methods, which suggests it is correcting for frequency effects rather than for anything specific to skip-gram.

Frequent-word subsampling. Each token is discarded during corpus reading with probability:

P(discard w) = 1 - sqrt( t / f(w) )

• f(w) is the relative frequency of w in the corpus.
• t is a threshold, typically 10^-5.

This is not a speed optimisation with a side effect. Deleting frequent tokens effectively widens the window over the surviving tokens, because removed words no longer occupy window slots. Reported gains from subsampling are substantial, especially for rare words.

Dynamic window size. The released C code does not use a fixed window c. For each token it draws c' ~ Uniform{1, ..., c} and uses that. The effect is a distance-dependent weighting: a word at offset d is included with probability (c - d + 1) / c. Papers that report "window = 5" and implement a fixed window are not reproducing the original method.

Cost#

Per training pair, with D the embedding dimension and k the number of negatives:

With k = 5 and D = 300, negative sampling costs roughly 1800 multiply-accumulates per pair. Memory is dominated by the two V x D tables: at V = 10^6 and D = 300 in float32, that is 2.4 GB. Training is embarrassingly parallel under Hogwild-style lock-free asynchronous SGD (Recht et al., 2011), which the original C implementation uses.

Hyperparameters carry more weight than architecture#

Levy, Goldberg & Dagan (2015), Improving Distributional Similarity with Lessons Learned from Word Embeddings (TACL), is the most useful paper in this area and the least cited relative to its value.

They transplanted word2vec's implementation choices — subsampling, dynamic windows, context distribution smoothing, adding context vectors to word vectors at the end — into PPMI and SVD baselines. Most of the reported advantage of word2vec over count-based methods disappeared.

The practical conclusion is uncomfortable and durable. Published comparisons between embedding methods frequently compare hyperparameter tuning budgets rather than methods.

Two specific settings from that work are worth applying:

• Context distribution smoothing with the same 0.75 exponent improves PPMI as much as it improves SGNS.
• w + c, summing the input and output vectors instead of discarding the output vectors, helps on similarity tasks at no cost. Most implementations discard Wo by default.

Where the line went after this#

GloVe (Pennington et al., 2014) and fastText (Bojanowski et al., 2017) are covered in embeddings. fastText's subword decomposition matters most for the Indic languages, where a single lemma surfaces in many inflected forms and a type-level vocabulary fragments badly.

Beyond words. The skip-gram objective is indifferent to what a "token" is. Any sequence of discrete items works.

• item2vec (Barkan & Koenigstein, 2016) — purchase sessions as sentences.
• node2vec (Grover & Leskovec, 2016) — biased random walks on a graph as sentences, interpolating between breadth-first and depth-first exploration with two parameters p and q.
• DeepWalk (Perozzi et al., 2014) — uniform random walks, the precursor.

These remain live techniques. For a recommendation system over ids with no pretrained model available, an SGNS objective over interaction sequences is still a strong and very cheap baseline.

Key references#

• Mikolov, T., Chen, K., Corrado, G. & Dean, J. (2013a). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781
• Mikolov, T., Sutskever, I., Chen, K., Corrado, G. & Dean, J. (2013b). Distributed Representations of Words and Phrases and their Compositionality. arXiv:1310.4546
• Goldberg, Y. & Levy, O. (2014). word2vec Explained: Deriving Mikolov et al.'s Negative-Sampling Word-Embedding Method. arXiv:1402.3722
• Levy, O., Goldberg, Y. & Dagan, I. (2015). Improving Distributional Similarity with Lessons Learned from Word Embeddings. TACL 3.
• Gutmann, M. & Hyvärinen, A. (2012). Noise-Contrastive Estimation of Unnormalized Statistical Models. JMLR 13.
• Recht, B., Ré, C., Wright, S. & Niu, F. (2011). Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent. NeurIPS 24.
• Grover, A. & Leskovec, J. (2016). node2vec: Scalable Feature Learning for Networks. arXiv:1607.00653

Honest assessment in 2026#

Word2Vec is no longer a competitive text representation. Contextual encoders beat it on every language-understanding benchmark, and contrastively trained sentence encoders beat it on every retrieval benchmark, usually by a wide margin.

Its remaining value is threefold and genuine. It is the clearest existing worked example of a self-supervised pretext task. Its implicit-factorisation result connects the neural and count-based literatures in a way no later model does as cleanly. And the SGNS objective is still the default first attempt when the tokens are arbitrary ids rather than language.

The open question it left behind is unresolved: nobody has a principled account of why the 0.75 exponent, dynamic windows and subsampling help as much as they do. They are load-bearing empirical choices sitting underneath a decade of downstream work.

What to learn next#

• Attention — the mechanism that replaced static vectors.
• BERT — masked pretraining and contextual representations.
• Embeddings — the representation view, count-based methods and modern encoders.