Learn AI Create the Future — with Pranay
Main site

Sequence Labelling and Structure

Conditional random fields

A conditional random field tags a whole sequence at once instead of one word at a time, so it can rule out impossible tag combinations like a sentence ending mid-entity.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A conditional random field tags an entire sentence as one connected decision, instead of guessing each word's tag in isolation.

Think about proofreading a form where every field depends on the one before it. Someone ticks "Married" in one box. The next box asks for a spouse's name, and it should not be left blank — the two answers have to agree with each other. You do not check each box alone. You check that the whole form makes sense together.

A conditional random field, or CRF, tags a sentence the same way. It does not decide each word's tag by itself and hope the tags next to it happen to agree. It picks the single best tag sequence for the whole sentence. Tags that do not fit together get ruled out from the start.

Why it exists

Imagine tagging words one at a time, with no knowledge of neighbouring tags. Nothing stops the tagger from producing O I-PERSON O — an "inside a person's name" tag with no matching "beginning" before it. That sequence makes no grammatical sense. Yet a word-by-word classifier can produce it, because it never looks at what it already guessed for the word before.

A CRF fixes this by scoring entire tag sequences, not individual tags. It learns which tag pairs tend to follow each other. B-PERSON is often followed by I-PERSON, almost never by B-LOCATION mid-name. The model uses that learned pattern to throw out sequences that break the rules, even when each individual tag looked plausible on its own.

How it works

A CRF combines two kinds of evidence: how well a tag fits the word itself, and how well a tag fits the tag next to it.

  Word:        Reserve    Bank     of      India
  Tag guess:   B-ORG      I-ORG    I-ORG   I-ORG

  Evidence 1 (word-level):  does "Reserve" look like the start of an ORG name?
  Evidence 2 (tag-level):   how often does B-ORG get followed by I-ORG?

  The CRF adds both kinds of evidence together across the whole sentence,
  then picks the single best-scoring tag sequence -- not the best tag
  for each word considered alone.

Crucially, the CRF checks every plausible sequence in one pass. It keeps the one with the highest total score, using an efficient algorithm rather than trying every combination by brute force.

Where you have already seen it

  • Older commercial NER systems, including early versions of the Stanford NER tagger, were built on CRFs. Many are still deployed in production, valued for speed and predictability over squeezing out the last point of accuracy.
  • Handwriting and speech recognition used CRFs for the same reason. Recognising one letter or sound in isolation is unreliable. Recognising it alongside its neighbours is much more accurate.
  • Modern neural NER models still borrow the idea. Many BERT-based taggers add a small CRF layer on top of the neural network, for exactly the reason described above. It stops the network from predicting an impossible tag sequence.

Remember this

  • A CRF scores whole tag sequences, not individual tags in isolation.
  • It combines "does this tag fit this word" with "does this tag fit next to the previous tag".
  • Even modern neural taggers often keep a small CRF layer on top, purely to enforce valid tag transitions.

What to learn next

Developer — Code and libraries.

sklearn-crfsuite wraps the classic CRFsuite library in a scikit-learn-shaped API. It trains in milliseconds on CPU for data this size.

Setup

bash
pip install sklearn-crfsuite

Minimal runnable code

crf_ner.py
import sklearn_crfsuite

# Tiny hand-labelled NER dataset: each sentence is a list of (word, POS, BIO-tag).
train_sents = [
    [("Priya", "NNP", "B-PER"), ("flew", "VBD", "O"), ("to", "IN", "O"),
     ("Chennai", "NNP", "B-LOC"), (".", ".", "O")],
    [("Ravi", "NNP", "B-PER"), ("works", "VBZ", "O"), ("at", "IN", "O"),
     ("Infosys", "NNP", "B-ORG"), (".", ".", "O")],
    [("Meera", "NNP", "B-PER"), ("lives", "VBZ", "O"), ("in", "IN", "O"),
     ("Mumbai", "NNP", "B-LOC"), (".", ".", "O")],
    [("Arjun", "NNP", "B-PER"), ("joined", "VBD", "O"), ("TCS", "NNP", "B-ORG"),
     ("yesterday", "NN", "O"), (".", ".", "O")],
]

def word_features(sent, i):
    word, pos = sent[i][0], sent[i][1]
    feats = {
        "word.lower()": word.lower(),
        "word.istitle()": word.istitle(),
        "word.isupper()": word.isupper(),
        "postag": pos,
        "BOS": i == 0,
    }
    if i > 0:
        feats["-1:word.lower()"] = sent[i - 1][0].lower()
    return feats

def sent_features(sent):
    return [word_features(sent, i) for i in range(len(sent))]

def sent_labels(sent):
    return [label for _, _, label in sent]

X_train = [sent_features(s) for s in train_sents]
y_train = [sent_labels(s) for s in train_sents]

crf = sklearn_crfsuite.CRF(algorithm="lbfgs", max_iterations=100)
crf.fit(X_train, y_train)

test_sent = [("Kavya", "NNP"), ("works", "VBZ"), ("at", "IN"), ("Wipro", "NNP"), (".", ".")]
test_feats = [word_features([(w, p, "") for w, p in test_sent], i) for i in range(len(test_sent))]
pred = crf.predict([test_feats])[0]

for (w, p), tag in zip(test_sent, pred):
    print(f"{w:8} {tag}")
Output
Kavya    B-PER
works    O
at       O
Wipro    B-ORG
.        O

The model correctly tags "Kavya" and "Wipro" as a person and an organisation, despite never seeing either name during training — it generalised from features like capitalisation and part-of-speech, not from memorising names.

Line by line

word_features builds a dictionary per word, not a fixed-size vector. This is a hallmark of classic feature-based CRFs: each feature is a named, human-readable signal — "is this word capitalised", "what is the previous word" — rather than a learned embedding. You can add or remove a feature and immediately reason about what changed.

-1:word.lower() is a transition-adjacent feature, looking one word to the left. This is different from the CRF's own tag-transition modelling — it is still a feature of the current position, only one that peeks at a neighbouring word's identity, not its tag.

crf.fit runs L-BFGS optimisation to find feature weights that make the gold tag sequences score higher than any other sequence, for every training sentence at once — this is what makes it a sequence model rather than four independent per-word classifiers.

crf.predict runs Viterbi decoding to find the single highest-scoring tag sequence for the test sentence, considering all four positions jointly rather than tagging word by word.

Common mistakes

Treating the feature dictionary as optional boilerplate. A CRF's accuracy is almost entirely a function of feature quality. A CRF with word.lower() alone will badly underperform one that also has capitalisation, suffixes and neighbouring words — unlike a neural network, it will not discover useful features on its own.

Forgetting word.istitle(), then wondering why proper nouns are missed. Capitalisation is one of the single strongest signals for named entities in English. Dropping it is a common and costly mistake.

Expecting a CRF trained on four sentences to generalise broadly. This toy example works because the test name and company happen to share features — capitalised, single token — with the training examples. A real CRF needs hundreds to thousands of labelled sentences per entity type.

Try it yourself

Remove the "postag" feature from word_features and retrain. Compare the predicted tags for the test sentence — part-of-speech is a strong signal for CRFs, and losing it can change predictions on ambiguous words.

What to learn next

Researcher — Mathematics and papers.

The model

A linear-chain CRF defines a conditional distribution over tag sequences given the input:

text
P(y | x) = (1 / Z(x)) * exp( sum over i=1..n of ( sum over k of lambda_k * f_k(y_{i-1}, y_i, x, i) ) )
  • x is the input sequence, y the tag sequence.
  • f_k(y_{i-1}, y_i, x, i) is a feature function — it can depend on the current and previous tag, the whole input, and the position i. This is the key generalisation over an HMM, which restricts emissions to depend on x_i alone.
  • lambda_k is the learned weight for feature f_k.
  • Z(x) = sum over all possible y' of exp(...) is the partition function, normalising the scores of every possible tag sequence into a valid probability distribution.

Because Z(x) sums over exponentially many sequences (|T|^n of them), it cannot be computed by enumeration. The forward-backward algorithm computes it in O(n * |T|^2) time by exploiting the chain structure — the same dynamic-programming trick an HMM uses for its own normalisation.

Why "conditional" matters

An HMM is generative: it models P(x, y) and derives P(y | x) via Bayes' rule, which forces an independence assumption on how x_i is generated given y_i. A CRF models P(y | x) directly and discriminatively, placing no restriction on how features of x may depend on each other. This is what allows overlapping, non-independent features — word identity, capitalisation, suffix, neighbouring words — all firing at once without violating any modelling assumption. Lafferty, McCallum & Pereira (2001) frame this explicitly as fixing the "label bias problem" that affects MEMMs, a related discriminative model with a weaker normalisation scheme.

Training and inference

Training maximises L2-regularised conditional log-likelihood over the training set via L-BFGS or a related quasi-Newton optimiser; each gradient step itself requires one forward-backward pass per training sequence, giving O(n * |T|^2) per sequence per iteration.

Inference — finding y* = argmax_y P(y | x) — uses the Viterbi algorithm, also O(n * |T|^2), tracking the highest-scoring path to each (position, tag) pair rather than summing over all paths.

CRFs as an output layer inside neural models

Huang, Xu & Yu (2015) attach a linear-chain CRF layer on top of a BiLSTM: the LSTM replaces hand-built feature functions with learned contextual representations, and the CRF layer still supplies the tag-transition matrix and Viterbi decoding, guaranteeing valid tag sequences. Lample et al. (2016) show this combination, BiLSTM-CRF, matching or beating feature-heavy CRFs on CoNLL-2003 English NER without any hand-engineered features at all.

For transformer encoders, the marginal benefit of a CRF layer shrinks further, since self-attention already conditions each token's representation on the entire sequence, including tags implicitly encoded in context. Empirically, a plain softmax classifier on top of BERT comes close to a BERT-CRF on clean benchmark data; the CRF layer's main remaining value is a hard guarantee against illegal tag transitions in production, not raw accuracy.

Key references

  • Lafferty, J., McCallum, A. & Pereira, F. (2001). Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. ICML. The original CRF paper.
  • Sutton, C. & McCallum, A. (2012). An Introduction to Conditional Random Fields. Foundations and Trends in Machine Learning. The standard tutorial-length reference.
  • Huang, Z., Xu, W. & Yu, K. (2015). Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv:1508.01991
  • Lample, G. et al. (2016). Neural Architectures for Named Entity Recognition. arXiv:1603.01360

Current state and open problems

Pure feature-based CRFs have been largely displaced by neural taggers for state-of-the-art accuracy, but remain in active production use where labelled data is scarce, latency budgets are tight, or interpretability of the feature weights matters for debugging and compliance. The CRF layer — the transition-scoring and Viterbi-decoding machinery, detached from hand-built features — persists inside many neural sequence labellers as a lightweight way to guarantee structurally valid output, which is a constraint that pure token classifiers, including most transformer-based ones, do not enforce on their own.

What to learn next