The paper in four sentences
A computer can represent each word as a list of a few hundred numbers, a point in a high-dimensional space, so that words used in similar ways end up close together.
Earlier neural networks learned such lists, but slowly, because they did a lot of expensive work on every word they read.
This paper throws away the expensive middle layer and trains two very simple models: one guesses a word from its neighbours (CBOW), the other guesses the neighbours from a word (skip-gram).
These cheap models are fast: CBOW read 1.6 billion words in under a day (skip-gram took about two), and their vectors capture relationships: the step from "man" to "woman" is roughly the same as the step from "king" to "queen".
Idea · §1
Words as points on a map
To a computer, a word is just a string of letters. "Cat" and "kitten" look no more related than "cat" and "carburettor". The simplest fix, giving each word its own ID number, keeps that blindness: every word is equally different from every other.
A better idea is to give each word a vector: a list of numbers, say 300 of them, which you can think of as coordinates of a point in a space with 300 directions. If similar words get nearby points, then a system that has learned something about "cat" automatically knows a little about "kitten". These are called word vectors or word embeddings.
Where do the numbers come from? From an old observation in linguistics: you can tell a lot about a word from the company it keeps. Words that appear in similar sentences ("I fed the ___", "the ___ purred") probably mean similar things. So the computer reads huge amounts of text and adjusts each word's numbers until words that share contexts share neighbourhoods.
TakeawayA word vector places each word as a point in space, learned from which words it appears near, so that similar words sit close together.
Check yourself
- Why are plain word ID numbers a poor representation?
- They carry no notion of similarity: word 412 and word 413 are no more alike than word 412 and word 9,000, so nothing learned about one word transfers to another.
Idea · §3 · Fig. 1
Two very simple guessing games
The networks people used before this paper to learn word vectors were full language models: they read the previous few words and tried to predict the next one, passing everything through a large "hidden layer" of artificial neurons. That hidden layer was where most of the computing time went.
Mikolov and colleagues ask: what if we only care about the word vectors, not about building a good predictor? Then the hidden layer can go. They propose two stripped-down games.
- CBOW ("continuous bag of words"): take the words around a gap, average their vectors, and guess the missing middle word. "Bag of words" means the order of the neighbours is ignored.
- Skip-gram: the reverse. Take one word and guess each of the words around it.
Each time a guess is wrong, the vectors involved are nudged slightly so the right answer becomes more likely next time. Over billions of guesses, words that appear in similar company get similar vectors. Below, step a skip-gram window along a sentence and watch which training examples it produces.
TakeawayCBOW guesses a word from its neighbours; skip-gram guesses the neighbours from a word. Both drop the costly hidden layer of earlier models.
Check yourself
- In skip-gram with a window of 2 on each side, how many guesses does one word in the middle of a sentence produce?
- Four: two words to the left and two to the right, each a separate guess.
- Why would anyone want to give up the hidden layer?
- It was the main computing cost. Without it, training is fast enough to read billions of words, and more data improves the vectors more than the hidden layer did.
Idea · §4.1 · Table 1
Meaning as directions you can add
The surprising part is not just that similar words cluster. The direction from one word to another seems to carry meaning. The arrow from "France" to "Paris" points roughly the same way as the arrow from "Italy" to "Rome": it is a "capital of" direction. So you can do arithmetic: start at Paris, subtract France, add Italy, and look up which word is nearest to where you land.
The authors turn this into a test. They built about 8,900 "semantic" questions (capitals, currencies, family relations) and about 10,700 "syntactic" ones (big to biggest, walk to walked, mouse to mice). A question counts as correct only if the nearest word is exactly the expected one. The words in the question itself are not allowed as answers, a detail that matters more than it looks.
Try it on a toy map below. The positions are hand-placed to illustrate the idea in two dimensions; real word vectors have hundreds.
TakeawayIn good word vectors, consistent relationships become consistent directions, so analogies can be solved by adding and subtracting vectors.
Check yourself
- Write the arithmetic for "walking is to walked as swimming is to ?".
- vector(walked) − vector(walking) + vector(swimming), then find the nearest word, hoping for "swam".
- Why are the question words excluded from the answers?
- The arithmetic often lands closest to one of the words you started with (especially C), so without the rule many questions would be "answered" with an input word.
Idea · §4.2–4.4 · Tables 2–6
Cheap models plus lots of text win
The big result is a trade. The simple models are a little less clever per word read, but they are so much cheaper that they can read far more, and reading more helps a lot. In the paper's tests, accuracy on the analogy questions keeps rising when the vectors get bigger and the training text gets longer, as long as both grow together (Table 2).
Trained on 783 million words with 300 numbers per word, skip-gram answered about 53% of the analogy questions exactly right. Older publicly available word vectors scored between 2% and 25% on the same test (Table 4). On a cluster of computers, using a Google News collection of about 6 billion words, the best skip-gram reached about 66% while training in roughly 2.5 days (Table 6). CBOW processed 1.6 billion words in under a day; skip-gram took about two days.
The two models also differ in character: skip-gram was much better on meaning-type questions (capitals, family), CBOW a bit better on grammar-type questions (plurals, tenses), and CBOW trains faster.
TakeawaySimple models that can read billions of words beat clever models that can only read millions.
Check yourself
- Why is "53% correct" more impressive than it sounds?
- Only an exact match counts, among a vocabulary of many thousands of words; near-synonyms or slight variants count as wrong.
Idea · §6–7 · What came next
From fixed vectors to words in context
Months later the same team published a follow-up that made skip-gram faster and better still, with a trick called negative sampling (instead of scoring every word in the vocabulary, compare the right neighbour against a handful of random ones) and by skipping some very common words like "the". They released the program as word2vec, and it became one of the most used tools in language processing.
Word2vec's vectors have a basic limitation: each word gets exactly one point, whatever it means in context. "Bank" in "river bank" and "bank account" share one vector. Later models such as BERT solved this by computing a fresh vector for each word every time it appears, based on the whole sentence. The idea of learning meaning from neighbouring words survives in them; the single fixed vector per word does not.
TakeawayWord2vec made good word vectors cheap and popular; modern models keep the idea of learning from context but give each word a different vector in each sentence.
Check yourself
- What can a single vector per word not capture?
- Different meanings of the same word in different contexts, like "bank" (river) versus "bank" (money).
Abstract, restated
Two novel log-linear architectures (CBOW and continuous skip-gram) for computing continuous vector representations of words from very large data sets. Quality is measured on a word-similarity task and compared with the previously best neural-network techniques: accuracy improves substantially at much lower computational cost; high-quality vectors are learned from a 1.6-billion-word data set in less than a day, and give state-of-the-art performance on a new test set of syntactic and semantic word similarities. Workshop track, ICLR 2013 (arXiv 1301.3781, Jan 2013; v3 Sep 2013).
Section · §2 · Eqs. 1–3
Where the cost of neural language models goes
The paper frames everything as training complexity, proportional to
with E epochs (3–50 typical), T training words (up to 109) and Q the per-word cost of the architecture. For a feed-forward NNLM (Bengio et al., 2003) with N context words, projection size D, hidden size H and vocabulary V:
The H×V output term dominates naively, but with a hierarchical softmax it becomes H×log2(V). The paper builds the hierarchy as a Huffman tree over word frequencies, so frequent words get short codes; it states that this needs about log2(unigram perplexity(V)) outputs and gives roughly a 2× speed-up for a one-million-word vocabulary over a balanced tree. Once the output is cheap, the remaining bottleneck is the non-linear hidden layer: N×D×H for the NNLM, H×H for the RNNLM.
Training used DistBelief (§2.3): asynchronous mini-batch gradient descent with Adagrad across many model replicas sharing a central parameter server.
Section · §3 · Eqs. 4–5 · Fig. 1
The two log-linear architectures
CBOW (§3.1). Remove the non-linear hidden layer and share the projection layer across all context positions, so context vectors are averaged into one projection (order is lost, hence "bag of words"). With four history and four future words as input, a log-linear classifier predicts the middle word:
Continuous skip-gram (§3.2). Each current word is the input to a log-linear classifier that predicts words within a range before and after it. For each training word a number R is sampled uniformly in ⟨1; C⟩ and the R previous and R following words are used as labels, giving 2R classifications; distant words are thereby sampled less often. With C = 10 in the experiments:
The 2013 paper specifies these models only at this level; the objective it optimises is the usual softmax log-likelihood of the target word given the input, computed through the hierarchical softmax. The explicit skip-gram objective, (1/T) Σt Σ−c≤j≤c, j≠0 log p(wt+j | wt) with p(wO | wI) = exp(v′wO⊤vwI) / Σw exp(v′w⊤vwI), is written out in the companion paper (Mikolov et al., NeurIPS 2013). Compare the per-word costs for typical settings below.
Check yourself
- With hierarchical softmax, what dominates Q for the NNLM and why do the new models avoid it?
- The N×D×H projection-to-hidden term. CBOW and skip-gram have no hidden layer; the projection feeds the (hierarchical) output directly.
- Why is skip-gram roughly C times costlier per word than CBOW?
- It makes up to 2R separate output predictions per input word (on average about C), whereas CBOW makes one prediction from the averaged context.
Section · §4.1 · Table 1
The Semantic-Syntactic Word Relationship test
The test set contains 8,869 semantic and 10,675 syntactic questions in 5 and 9 categories (Table 1), built by listing word pairs by hand and combining pairs into questions; only single-token words are used. Examples: Athens–Greece / Oslo–Norway (common capitals), Angola–kwanza / Iran–rial (currency), brother–sister / grandson–granddaughter (man–woman), great–greater / tough–tougher (comparative), mouse–mice / dollar–dollars (plural nouns).
A question "a : b :: c : ?" is answered by computing X = vec(b) − vec(a) + vec(c) and returning the word with highest cosine similarity to X, excluding the question words. Only exact matches count; the paper notes that synonyms are therefore counted as mistakes and that 100% is likely unreachable.
The figure below trains a real skip-gram on a tiny synthetic corpus in your browser, so you can see distributional similarity emerge from nothing but co-occurrence.
Section · §4.2–4.5 · Tables 2–7
Results
Scaling data and dimensionality (Table 2). CBOW on a 30k-word vocabulary subset: with 50 dimensions, accuracy saturates near 23% even at 783M words; with 600 dimensions it climbs from 24.0% (24M words) to 50.4% (783M). Both must grow together.
Architecture comparison (Table 3), 640-dimensional vectors trained on the same 320M words (82K vocabulary):
| Model | Semantic % | Syntactic % | MSR relatedness % |
|---|---|---|---|
| RNNLM | 9 | 36 | 35 |
| NNLM | 23 | 53 | 47 |
| CBOW | 24 | 64 | 61 |
| Skip-gram | 55 | 59 | 56 |
Against public vectors (Table 4). Collobert–Weston (50-d, 660M words) 11.0% total; Turian 1.8–2.1%; Mnih 5.8–8.8%; Mikolov RNNLM (640-d) 24.6%; Huang 12.3%. CBOW (300-d, 783M words) 36.1%; skip-gram 53.3%.
Epochs versus data (Table 5). One epoch on twice the data matches or beats three epochs: 1-epoch skip-gram on 1.6B words reaches 53.8% in 2 days, versus 53.3% for 3 epochs on 783M words in 3 days. 1-epoch CBOW on 783M words takes 0.3 days.
| DistBelief, Google News 6B (Table 6) | Dim | Sem. | Syn. | Total | Days × CPU cores |
|---|---|---|---|---|---|
| NNLM | 100 | 34.2 | 64.5 | 50.8 | 14 × 180 |
| CBOW | 1000 | 57.3 | 68.9 | 63.7 | 2 × 140 |
| Skip-gram | 1000 | 66.1 | 65.1 | 65.6 | 2.5 × 125 |
Microsoft Sentence Completion (§4.5, Table 7). 1,040 sentences with one missing word and five choices. Skip-gram alone scores 48.0% (below the 49% of LSA similarity), but combined with RNNLMs it reaches 58.9%, a new state of the art over 55.4% for RNNLMs alone.
Learned relationships (§5, Table 8) from a 300-d skip-gram on 783M words include France–Paris → Italy: Rome, Japan: Tokyo; big–bigger → cold: colder, quick: quicker; Sarkozy–France → Merkel: Germany. The table also shows misses (big–bigger → small: larger), and the authors estimate exact-match accuracy on this set at about 60%.
Check yourself
- What does Table 2 say about choosing vector size?
- Beyond some point, adding dimensions or data alone gives diminishing returns; accuracy improves most when dimensionality and training data are increased together.
- Why does Table 3 hold dimensionality and data fixed?
- To compare architectures fairly: differences then reflect the model, not the compute budget. Skip-gram's semantic advantage (55 vs 24) is the striking result.
Section · Limitations and lineage
What held up and what did not
- Companion paper. Mikolov, Sutskever, Chen, Corrado & Dean (NeurIPS 2013), "Distributed Representations of Words and Phrases and their Compositionality", added negative sampling (a simplified noise-contrastive objective, log σ(v′wO⊤vwI) + Σk log σ(−v′wk⊤vwI)), subsampling of frequent words and phrase vectors. SGNS (skip-gram with negative sampling) is what "word2vec" usually means today.
- What it computes. Levy & Goldberg (2014) showed SGNS implicitly factorises a shifted pointwise mutual information matrix, linking it to classical count-based distributional semantics. GloVe (Pennington et al., 2014) fits co-occurrence statistics directly; fastText (Bojanowski et al., 2017) adds subword n-grams, addressing the paper's own note that the models have no access to morphology.
- Analogy evaluation. The exclusion of question words is load-bearing: later analyses (e.g. Linzen 2016; Nissim, van Noord & van der Goot 2020) showed that without it the nearest vector is very often c or b, and that offsets reflect similarity more than clean relational structure. Treat "king − man + woman ≈ queen" (which the paper attributes to earlier RNNLM work, ref. [20]) as an illustration, not a law.
- Bias. Bolukbasi et al. (2016) showed the same arithmetic yields occupational gender stereotypes learned from the corpus.
- Static vectors. One vector per type cannot represent polysemy. Contextual models (ELMo 2018, GPT, BERT) replaced fixed embeddings with representations computed from the whole sentence; their input embedding layers are the direct descendants of this paper's projection matrix.
Reference
Glossary
- Word vector
- A dense real-valued representation of a word; also embedding or distributed representation.
- NNLM
- Feed-forward neural network language model (Bengio et al., 2003).
- RNNLM
- Recurrent neural network language model.
- Projection layer
- A shared lookup table mapping each word to its D-dimensional vector.
- Log-linear
- A model whose log-probabilities are linear in its parameters given the inputs; here, no hidden non-linearity.
- CBOW
- Continuous bag-of-words: predict the centre word from the average of context vectors.
- Skip-gram
- Predict each context word from the centre word.
- Hierarchical softmax
- Factorises the output distribution along a binary (here Huffman) tree, costing about log2V per prediction.
- Negative sampling
- From the companion paper: train the true pair against k sampled "noise" words instead of a full softmax.
- DistBelief
- Google's distributed training framework with asynchronous SGD and a parameter server.