Learn AI Create the Future — with Pranay
Main site

OCR and Document Vision

CTC loss

CTC lets you train a reader on "this strip says CAT" without ever labelling which pixels are the C, by scoring every alignment at once instead of picking one.

Read these first

On this page 9
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. What blank is really for
  6. Where you have already seen it
  7. The honest part
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

CTC trains a reader when you know what the strip says, but not where each letter sits.

CTC stands for connectionist temporal classification. It is a 2006 name for lining up a sequence with a picture, without being told how.

The analogy you have already lived

Think of hearing a song and knowing the lyrics, but not knowing the exact second each word is sung. You can still sing along. You match the words to the sound as it goes, sliding a little to fit.

Now imagine writing down the exact millisecond every syllable starts, for ten thousand songs. That is the labelling job CTC removes.

Why it exists

To train a model to read, you need labelled examples. The obvious label is a box around every letter.

Drawing those boxes is brutal. A page of handwriting has hundreds of letters, and half of them touch. Two annotators will disagree on where the m ends.

What is easy to collect is the whole answer: this strip says PLATFORM. No positions, only the text.

CTC turns that cheap label into a usable training signal. It runs inside most speech systems too, where the same problem appears with sound.

How it works

Your model looks at the strip in thin vertical slices. For each slice it guesses a letter, or the special symbol blank, which means "nothing finishes here".

   slices:   1    2    3    4
   guess:    -    a    -    b     ->  reading rule  ->  "ab"
   guess:    a    a    -    b     ->  reading rule  ->  "ab"
   guess:    -    -    a    b     ->  reading rule  ->  "ab"

The reading rule is two steps, in this order. Squash runs of the same symbol into one. Then delete the blanks.

Notice something. Three different slice-by-slice guesses all read as ab. There are fifteen in total for four slices.

CTC does not choose one of them. It adds up how likely all of them are, and pushes that total upward. The model is never told which alignment is right, so it never has to be right about the alignment.

What blank is really for

Blank looks like a technicality. It is the piece that makes double letters possible.

   a a a a   ->  squash  ->  a       one letter
   a a - a   ->  squash  ->  a - a   ->  delete blanks  ->  aa

Without a blank there is no way to say "two a's in a row". This is why a CTC alphabet always has one more symbol than your actual character set.

Where you have already seen it

  • Number plate readers, reading a plate with no letter boxes drawn.
  • Speech-to-text on your phone, matching words to audio nobody time-stamped.
  • Handwriting recognition on a form, where letters run together.
  • The recognition stage of most open-source OCR engines.

The honest part

This is confusing for almost everyone the first time. Read it twice — that is normal.

The stubborn part is that a loss can be defined over a set of possibilities rather than one right answer. Most losses you have met compare one prediction with one label. This one compares one prediction with every labelling that would be acceptable.

Remember this

  • CTC trains a reader using only the final text, with no letter positions.
  • It scores every way the letters could line up, and adds them together.
  • The blank symbol is what lets a repeated letter exist.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install torch==2.5.1 numpy==1.26.4

The clearest way to understand CTC is to compute it twice: once by listing every path by hand, once with the algorithm that real code uses. Then check both against PyTorch.

CTC, computed three ways

ctc_by_hand.py
import itertools
import numpy as np
import torch
import torch.nn as nn

SYMBOLS = ["-", "a", "b"]          # index 0 is the blank
T = 4                              # four timesteps
probs = np.random.default_rng(64).dirichlet(np.ones(3), size=T)   # a fake network output

print("what the network said (one row per timestep, each row sums to 1):")
for t, row in enumerate(probs):
    print(f"  t={t}   " + "   ".join(f"{s}: {p:.3f}" for s, p in zip(SYMBOLS, row)))


def collapse(path):
    """The CTC rule: drop repeats first, then drop blanks."""
    out = []
    for s in path:
        if not out or s != out[-1]:
            out.append(s)
    return "".join(c for c in out if c != "-")


def brute_force(target):
    """Every path of length T whose collapse equals target, and their total probability."""
    hits = []
    for path in itertools.product(SYMBOLS, repeat=T):
        if collapse(path) == target:
            p = float(np.prod([probs[t][SYMBOLS.index(s)] for t, s in enumerate(path)]))
            hits.append(("".join(path), p))
    hits.sort(key=lambda e: -e[1])
    return sum(p for _, p in hits), hits


total, hits = brute_force("ab")
print(f"\nways to write 'ab' in {T} timesteps: {len(hits)} different paths")
for path, p in hits[:6]:
    print(f"   {path}    probability {p:.5f}")
print(f"   ... and {len(hits) - 6} rarer ones")
print(f"   total probability of 'ab' = {total:.6f}")


def ctc_forward(target):
    """The alpha recursion: the same sum, in O(T * len) instead of 3^T."""
    ext = ["-"]
    for c in target:
        ext += [c, "-"]                      # a blank around and between every label
    S = len(ext)
    a = np.zeros((T, S))
    a[0, 0] = probs[0][0]
    a[0, 1] = probs[0][SYMBOLS.index(ext[1])]
    for t in range(1, T):
        for s in range(S):
            v = a[t - 1, s] + (a[t - 1, s - 1] if s > 0 else 0.0)
            if ext[s] != "-" and s > 1 and ext[s] != ext[s - 2]:
                v += a[t - 1, s - 2]         # skipping is allowed, but only over a blank
            a[t, s] = v * probs[t][SYMBOLS.index(ext[s])]
    return a[T - 1, S - 1] + a[T - 1, S - 2], ext, a


forward_total, ext, alpha = ctc_forward("ab")
print(f"\nthe padded label sequence the algorithm walks: {ext}")
print("alpha table (rows = timesteps, columns = positions in that padded sequence):")
for t in range(T):
    print("   " + "  ".join(f"{v:.5f}" for v in alpha[t]))
print(f"\nforward algorithm total = {forward_total:.6f}")
print(f"brute force total       = {total:.6f}")
print(f"paths counted           = {len(hits)} by brute force, {3 ** T} were checked")

logp = torch.log(torch.tensor(probs, dtype=torch.float64))[:, None, :]   # T, batch, classes
loss = nn.CTCLoss(blank=0, reduction="none")(
    logp, torch.tensor([[1, 2]]), torch.tensor([T]), torch.tensor([2]))
print(f"\ntorch nn.CTCLoss        = {loss.item():.6f}")
print(f"-log(our total)         = {-np.log(total):.6f}")

print("\na repeated letter is harder to write than two different ones:")
for target in ["ab", "aa"]:
    tot, ex = brute_force(target)
    print(f"   '{target}' -> {len(ex):2d} paths, total probability {tot:.6f}")
print("   'aaaa' collapses to 'a', so 'aa' needs a blank in the middle")

greedy_path = "".join(SYMBOLS[i] for i in probs.argmax(1))
print(f"\ngreedy decode picks the best symbol per step: '{greedy_path}' -> "
      f"'{collapse(greedy_path)}'")
ranked = sorted(((brute_force(t)[0], t) for t in
                 ["", "a", "b", "ab", "ba", "aa", "bb", "aba", "bab"]), reverse=True)
print("but summing over paths ranks the strings differently:")
for p, t in ranked[:4]:
    print(f"   {('(empty)' if t == '' else t):<8} {p:.6f}")
Output
what the network said (one row per timestep, each row sums to 1):
  t=0   -: 0.544   a: 0.302   b: 0.153
  t=1   -: 0.436   a: 0.434   b: 0.130
  t=2   -: 0.796   a: 0.163   b: 0.041
  t=3   -: 0.014   a: 0.117   b: 0.869

ways to write 'ab' in 4 timesteps: 15 different paths
   -a-b    probability 0.16352
   a--b    probability 0.09107
   aa-b    probability 0.09070
   --ab    probability 0.03359
   -aab    probability 0.03345
   aaab    probability 0.01856
   ... and 9 rarer ones
   total probability of 'ab' = 0.450834

the padded label sequence the algorithm walks: ['-', 'a', '-', 'b', '-']
alpha table (rows = timesteps, columns = positions in that padded sequence):
   0.54449  0.30202  0.00000  0.00000  0.00000
   0.23734  0.36749  0.13165  0.03925  0.00000
   0.18895  0.09851  0.39737  0.02208  0.03125
   0.00263  0.03367  0.00690  0.45009  0.00074

forward algorithm total = 0.450834
brute force total       = 0.450834
paths counted           = 15 by brute force, 81 were checked

torch nn.CTCLoss        = 0.796657
-log(our total)         = 0.796657

a repeated letter is harder to write than two different ones:
   'ab' -> 15 paths, total probability 0.450834
   'aa' ->  5 paths, total probability 0.049349
   'aaaa' collapses to 'a', so 'aa' needs a blank in the middle

greedy decode picks the best symbol per step: '---b' -> 'b'
but summing over paths ranks the strings differently:
   ab       0.450834
   b        0.177817
   bb       0.111471
   bab      0.080253

Reading the output carefully

The three totals agree to six decimal places. 0.450834 from listing all 81 paths, 0.450834 from the dynamic-programming recursion, and 0.796657 = -log(0.450834) from PyTorch. That agreement is the whole lesson. The loss is the negative log of a sum over alignments, with nothing hidden inside the framework.

['-', 'a', '-', 'b', '-'] is the extended label sequence. Every CTC implementation builds this: blanks inserted around and between the target labels. It turns "find all alignments" into "walk left to right through this list, allowed to stay put, step one, or skip a blank".

The alpha table is where the exponential blowup disappears. Row t column s holds the total probability of every path that has consumed the first t+1 timesteps and reached position s. Position 3 at t=3 holds 0.45009, and adding position 4's 0.00074 gives the answer. Four rows of five numbers replaced 81 enumerations, and the saving grows as $3^T$ against $T \times (2\ell+1)$.

Look at the second row, column 3 (0.03925). That entry is nonzero even though only two timesteps have passed and the target has two labels plus blanks. It got there by the skip rule: from position 1 (a) straight to position 3 (b), jumping over the blank at position 2. That single line — the s - 2 term — is what allows ab with no gap.

'ab' has 15 paths, 'aa' has 5. Repeated characters are structurally rarer under CTC, because the only legal way to write them consumes an extra timestep for the mandatory blank. Models trained with CTC are measurably worse at doubled letters, and this table is why.

The greedy decode is wrong here, and the failure is instructive. Taking the best symbol at each timestep gives - - - b, which reads as b with total probability 0.178. But ab has total probability 0.451. The single most likely path does not sit inside the most likely string. Beam search over collapsed strings fixes exactly this, and this is why production recognisers use it.

The two decoding strategies

GreedyPrefix beam search
What it maximisesBest pathBest string, approximately
Cost$O(T \lvert \mathcal{A} \rvert)$$O(T \cdot B \cdot \lvert \mathcal{A} \rvert)$
Language modelCannot use oneScore merges naturally
Typical gainBaseline1–3% absolute on hard text

For clean printed documents, greedy is usually within noise of beam search and far simpler. For handwriting, low-resolution scans, or anything where a dictionary exists, beam search with a language model is worth the complexity.

Common mistakes

Passing logits rather than log probabilities. nn.CTCLoss expects log_softmax output. This is the single most common bug, and it does not raise an error.

Wrong tensor layout. Default is (time, batch, classes). Setting batch_first on your LSTM and forgetting to permute silently trains on transposed data.

Using index 0 for a real character. blank=0 by default. Build your alphabet with a +1 offset, as in CRNN text recognition.

Input shorter than the target. If input_length < target_length + repeats, no valid path exists and the loss is infinite. zero_infinity=True hides this by setting those samples to zero, so training quietly ignores your longest words. Assert the inequality yourself.

Expecting character positions from a CTC model. The frame at which a character "fires" is not its location; CTC is free to place spikes anywhere consistent. Peak positions correlate with character positions but are not a reliable segmentation.

Expecting the loss to fall immediately. CTC models sit on a plateau emitting all-blank for hundreds of steps, then drop sharply. That plateau is not a bug, and killing a run during it is a common waste.

Try it yourself

Change T from 4 to 3 and target aa. Brute force will find exactly one path, a-a. Then set T = 2. There is now no legal path at all, the total is zero, and the loss is infinite — the failure mode that zero_infinity masks.

What to learn next

Researcher — Mathematics and papers.

The definition

From Graves, Fernández, Gomez and Schmidhuber (2006), Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks, ICML.

Let $\mathcal{A}$ be the label alphabet and $\mathcal{A}' = \mathcal{A} \cup {\varnothing}$ with $\varnothing$ the blank. A network emits $y^t_k = P(k \text{ at time } t \mid x)$ for $t = 1 \dots T$. A path $\pi \in \mathcal{A}'^{\,T}$ has probability

$$ P(\pi \mid x) = \prod_{t=1}^{T} y^{t}_{\pi_t} $$

This factorisation assumes conditional independence of outputs across time given $x$. It is the assumption that makes CTC tractable, and the reason CTC models cannot represent output dependencies — no internal language model.

The collapsing map $\mathcal{B} : \mathcal{A}'^{\,T} \to \mathcal{A}^{\leq T}$ removes repeats then blanks. The probability of a labelling $\mathbf{l}$ is the sum over its preimage:

$$ P(\mathbf{l} \mid x) = \sum_{\pi \in \mathcal{B}^{-1}(\mathbf{l})} P(\pi \mid x) $$

And the loss is $-\log P(\mathbf{l} \mid x)$.

The forward recursion

Define the extended sequence $\mathbf{l}'$ of length $S = 2|\mathbf{l}| + 1$ by inserting blanks around and between labels. Let $\alpha_t(s)$ be the total probability of all paths reaching position $s$ of $\mathbf{l}'$ at time $t$. Initialise $\alpha_1(1) = y^1_{\varnothing}$, $\alpha_1(2) = y^1_{\mathbf{l}'_2}$, all else zero. Then:

$$ \alpha_t(s) = y^{t}_{\mathbf{l}'s}\Bigl(\alpha{t-1}(s) + \alpha_{t-1}(s-1) + \mathbb{1}\bigl[\mathbf{l}'_s \neq \varnothing \;\wedge\; \mathbf{l}'s \neq \mathbf{l}'{s-2}\bigr]\,\alpha_{t-1}(s-2)\Bigr) $$

Where $\mathbb{1}[\cdot]$ is 1 when the condition holds and 0 otherwise. The indicator forbids skipping a blank that separates two identical labels — the mechanism that keeps aa distinguishable from a.

The answer is $P(\mathbf{l} \mid x) = \alpha_T(S) + \alpha_T(S-1)$.

Cost is $O(TS) = O(T|\mathbf{l}|)$ time and memory, against $|\mathcal{A}'|^T$ for enumeration.

The gradient

Define the backward variable $\beta_t(s)$ symmetrically. The derivative with respect to the unnormalised pre-softmax output $u^t_k$ has a remarkably clean form:

$$ \frac{\partial \mathcal{L}}{\partial u^t_k} = y^t_k - \frac{1}{P(\mathbf{l}\mid x)}\sum_{s \in {s\,:\,\mathbf{l}'_s = k}} \alpha_t(s)\beta_t(s) $$

The second term is the posterior probability that label $k$ is emitted at time $t$, marginalised over alignments. So the gradient is prediction minus soft target, exactly as in ordinary cross-entropy — the difference being that the target is computed by the forward-backward pass rather than supplied.

Numerically, everything above must be done in log space with the log-sum-exp trick. Products of $T$ probabilities underflow float32 for $T$ beyond a few dozen.

Known limitations, and what addresses them

Conditional independence. CTC assigns no probability mass to output structure. Two consequences: a CTC model cannot learn that q is followed by u, and its per-frame posteriors are usually peaky and overconfident. Fixes are external — a shallow-fused language model in beam search, or a different loss.

Monotonic alignment only. CTC assumes outputs advance left to right with the input. That holds for horizontal text and for speech. It fails for anything requiring reordering, which is why CTC is unsuitable for translation and for right-to-left scripts unless the input is flipped.

The peaky-spike behaviour. Trained CTC networks emit blank at the vast majority of frames with sharp spikes elsewhere. Zeyer et al. (2021) and others analyse why: the blank path dominates early, and gradient descent finds it a stable attractor. It explains the loss plateau, and it means CTC posteriors should not be read as segmentation.

Alternatives worth knowing.

  • RNN-Transducer (Graves, 2012) adds a prediction network over previous outputs, restoring output dependencies while keeping streaming. Standard in production speech; increasingly used for OCR.
  • Attention decoders drop monotonicity entirely and gain flexibility at the price of possible hallucination and repetition loops.
  • Hybrid CTC + attention (Watanabe et al., 2017) trains both heads on one encoder and decodes jointly, using CTC's monotonicity to constrain attention. Widely used in ESPnet and in strong handwriting systems.

Beyond text

CTC's reach is much wider than OCR. It is a general answer to "supervise a sequence when the alignment is unknown", used in speech recognition, in lip reading, in gesture and sign-language recognition, in music transcription, and in protein sequence tagging. The OCR case is a convenient one to learn on because the alignment is visible in the image — but the algorithm never uses that fact.

What to learn next