Build a sentiment analyser

Formulas here are written in plain text, since the site renders no maths typesetting library.

TF-IDF#

scikit-learn's TfidfVectorizer computes, for term t in document d across a corpus of N documents:

tf-idf(t, d)  =  tf(t, d)  *  idf(t)

idf(t)  =  ln( (1 + N) / (1 + df(t)) )  +  1        [smooth_idf=True, the default]

• tf(t, d) is the raw count of t in d.
• df(t) is the number of documents containing t.
• The +1 terms implement add-one smoothing, as though one extra document contained every term, so idf never divides by zero.
• The trailing +1 prevents terms appearing in every document from being zeroed out entirely.

Each document vector is then L2-normalised, so || x_d ||_2 = 1. Note that this differs from the textbook definition (Spärck Jones, 1972), where idf = log(N / df(t)). Numbers from scikit-learn will not match a hand calculation from a textbook, which causes a great deal of confusion.

The L2 normalisation interacts directly with regularisation strength: because every feature value is bounded by 1 and typically far smaller, the effective penalty at a given C is much heavier than on raw counts. This is why C=1.0 under-fits on small TF-IDF matrices.

Logistic regression#

p(y = 1 | x)  =  sigma( w . x + b ),     sigma(z) = 1 / (1 + exp(-z))

Fitted by minimising the L2-penalised negative log-likelihood:

J(w)  =  C * sum_i [ -y_i log p_i - (1 - y_i) log(1 - p_i) ]  +  (1/2) || w ||_2^2

• C is the inverse regularisation strength. Large C means a weak penalty. Note that scikit-learn scales the loss by C rather than the penalty by lambda; they are equivalent up to C = 1 / lambda.
• The objective is convex in w, so the optimum is global. lbfgs reaches it reliably at this scale.

The decision function is linear in the features, so each coefficient w_t is exactly the change in log-odds contributed per unit of tf-idf(t). That is why sorting clf.coef_[0] gives a directly interpretable word ranking — an interpretability property that does not survive any non-linear model.

Complexity. Vectorising is O(total tokens). Fitting with lbfgs is O(n * nnz) per iteration where nnz is average non-zeros per row. Both are trivial at corpus sizes below a few million documents, which is why this baseline remains the correct first move.

Why bag-of-words cannot represent negation#

The representation is a multiset. For any permutation pi of the token sequence, phi(x) = phi(pi(x)). "not good" and "good not" map to the identical vector, so no function of phi(x) can distinguish them.

Adding n-grams enlarges the feature space but does not change the character of the failure. An n-gram model is a bag over spans of length up to n; it can represent negation only for the specific spans observed in training, and the number of possible negated phrases grows combinatorially. Negation scope in English can also extend well beyond a window — "I did not find the acting, the writing, or the direction convincing" negates three constituents at distance.

The classical workaround (Das and Chen, 2001; used by Pang, Lee and Vaithyanathan, 2002) tags every token between a negation cue and the next punctuation with a NOT_ prefix, so good becomes NOT_good as a distinct feature. It measurably helps and is a heuristic, not a solution: it mis-scopes on subordinate clauses and doubles the vocabulary.

Sequence models — LSTMs, and then transformers — represent negation because the hidden state at each position depends on the preceding tokens. Socher et al. (2013) made this concrete with the Stanford Sentiment Treebank, which annotates sentiment at every node of a parse tree specifically so that compositional effects can be trained and measured, rather than assumed.

Benchmarks and their ceilings#

• Pang and Lee (2004) polarity dataset: 2,000 movie reviews. Small, and thoroughly over-fitted by the field.
• IMDB (Maas et al., 2011): 50,000 reviews, deliberately polarised — ratings of 5 and 6 are excluded. Real reviews include the middle, so IMDB accuracy overstates deployed performance.
• SST-2 (Socher et al., 2013): 67,000 sentence-level items with phrase-level annotation. Part of GLUE.
• Amazon multi-domain (Blitzer, Dredze and Pereira, 2007): the standard testbed for cross-domain transfer, and the source of the durable finding that sentiment lexicons are domain-specific.

Two ceilings bound all of these. Annotator agreement: inter-annotator agreement on fine-grained sentiment commonly lands in the 0.7 to 0.8 Cohen's kappa range, and a model cannot exceed the reliability of its labels. Benchmark saturation: SST-2 leaderboard scores exceed 97%, above plausible label quality, which means remaining gains partly measure fitting to annotation artefacts.

Sarcasm and irony#

SemEval-2018 Task 3 (Van Hee et al.) is the reference evaluation. Top systems reached F1 around 0.7 on binary irony detection in tweets — far below the numbers routinely quoted for sentiment.

The difficulty is structural rather than a modelling deficiency. Irony is signalled by a mismatch between the utterance and a situation the text does not describe. "Great, another two hour delay" requires world knowledge that delays are undesirable, plus the recognition that "great" is incongruent with it. Datasets built from #sarcasm self-labelling also carry a well-documented bias: authors tag sarcasm when it might be missed, so the corpus over-represents unusually explicit cases.

Treat any product claim of reliable sarcasm detection with scepticism, and measure it on your own data before you believe it.

What a production system looks like#

Document-level polarity is rarely the deliverable. Aspect-based sentiment analysis (ABSA) extracts (aspect, opinion, polarity) triples, so "great camera, terrible battery" yields two opposite judgements. SemEval-2014 Task 4 (Pontiki et al.) defined the standard formulation, and modern systems cast it as sequence labelling or as generation of structured triples.

A deployed pipeline typically layers:

1. Language identification and routing. Indian review streams are heavily code-mixed — Hindi or Tamil written in Latin script, interleaved with English. Monolingual English models degrade badly here; see Khanuja et al. (2020) on the GLUECoS benchmark.
2. Spam and incentivised-review filtering, before any sentiment is computed. Otherwise you measure the marketing budget.
3. Aspect extraction, then per-aspect polarity.
4. Aggregation with uncertainty, reporting intervals rather than a single score.
5. Drift monitoring. Product vocabulary and complaint themes shift continuously, so a static model silently degrades. Track the rate of out-of-vocabulary tokens as an early warning.

The unglamorous items — 1, 2 and 5 — determine whether the system is trusted. Model architecture rarely does.

Papers#

• Spärck Jones, A Statistical Interpretation of Term Specificity and its Application in Retrieval, Journal of Documentation, 1972.
• Pang, Lee and Vaithyanathan, Thumbs up? Sentiment Classification using Machine Learning Techniques, EMNLP 2002 — the paper that started the field.
• Pang and Lee, A Sentimental Education, ACL 2004.
• Blitzer, Dredze and Pereira, Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification, ACL 2007.
• Maas, Daly, Pham, Huang, Ng and Potts, Learning Word Vectors for Sentiment Analysis, ACL 2011 — introduces IMDB.
• Socher, Perelygin, Wu et al., Recursive Deep Models for Semantic Compositionality over a Sentiment Treebank, EMNLP 2013 — introduces SST.
• Wang and Manning, Baselines and Bigrams, ACL 2012 — why this baseline is hard to beat.
• Pontiki et al., SemEval-2014 Task 4: Aspect Based Sentiment Analysis, SemEval 2014.
• Van Hee, Lefever and Hoste, SemEval-2018 Task 3: Irony Detection in English Tweets, SemEval 2018.
• Khanuja, Dandapat, Srinivasan, Sitaram and Choudhury, GLUECoS: An Evaluation Benchmark for Code-Switched NLP, ACL 2020 — arxiv.org/abs/2004.12376

What to learn next#

• Text classification — the general method, beyond sentiment.
• Embeddings — representations that know "reimbursed" and "refund" are related.
• BERT — the sequence model that handles the negation cases this project cannot.