Speech recognition

Three eras, briefly#

Generative HMM-GMM (1980s to about 2012). Model $P(X \mid W)$ with hidden Markov models over context-dependent phone states, emissions by Gaussian mixtures over MFCC-39, decoded through a weighted finite-state transducer composing the acoustic model, lexicon and n-gram language model. Recognition maximises $P(W \mid X) \propto P(X \mid W) P(W)$.

Hybrid DNN-HMM (2012 onward). Replace GMM emissions with a neural network estimating $P(s \mid x_t)$ and divide by the state prior to get a scaled likelihood. Hinton et al. (2012), Deep Neural Networks for Acoustic Modeling in Speech Recognition, reported 20 to 30 percent relative WER reductions and effectively ended the GMM era.

End-to-end (2014 onward). Map audio directly to characters or subwords, discarding the phone lexicon. Three families dominate, described below.

CTC#

Graves et al. (2006) introduced connectionist temporal classification. Given input frames $x_{1:T}$ and target $y_{1:U}$ with $U \le T$, define an extended alphabet $\mathcal{A}' = \mathcal{A} \cup {\varnothing}$ and the collapse map $\mathcal{B}: \mathcal{A}'^{T} \to \mathcal{A}^{\le T}$ that merges repeats then deletes blanks. The likelihood marginalises over every alignment:

$$ P(y \mid x) = \sum_{\pi \in \mathcal{B}^{-1}(y)} \prod_{t=1}^{T} P(\pi_t \mid x) $$

Where $\pi$ is one frame-level alignment, $\pi_t$ its symbol at frame $t$, and $\mathcal{B}^{-1}(y)$ the set of alignments collapsing to $y$. That set is exponentially large, so the sum is computed by a forward-backward recursion over the extended target $y' = (\varnothing, y_1, \varnothing, y_2, \dots, y_U, \varnothing)$ of length $2U+1$:

$$ \alpha_t(s) = P(y'_s \mid x_t) \cdot \begin{cases} \alpha_{t-1}(s) + \alpha_{t-1}(s-1) & \text{if } y'_s = \varnothing \text{ or } y's = y'{s-2}\ \alpha_{t-1}(s) + \alpha_{t-1}(s-1) + \alpha_{t-1}(s-2) & \text{otherwise} \end{cases} $$

The three-term case is the transition that permits skipping a blank between distinct characters; the two-term case forbids skipping when it would merge a genuine double letter. Cost is $O(T \cdot U)$ per utterance. Loss is $-\log \alpha_T(2U+1) + \alpha_T(2U)$, summing the two valid end states.

The conditional independence assumption is CTC's defining weakness: outputs are independent given the encoder states, so the model cannot represent $P(y_u \mid y_{<u})$ internally. It has no implicit language model, which is why external LM fusion helps CTC systems far more than it helps attention systems.

RNN-Transducer#

Graves (2012) added a prediction network over previous non-blank outputs and a joint network combining it with the encoder:

$$ P(k \mid t, u) = \operatorname{softmax}\big(W \tanh(W_e h_t^{\text{enc}} + W_p g_u^{\text{pred}} + b)\big) $$

Where $h_t^{\text{enc}}$ is the encoder state at frame $t$ and $g_u^{\text{pred}}$ the prediction state after $u$ emitted symbols. The lattice is two-dimensional, and the forward recursion costs $O(T \cdot U \cdot |\mathcal{A}|)$ in memory, which is the practical obstacle to training it.

RNN-T removes the conditional independence assumption while remaining strictly streaming — it emits without seeing future frames. That combination is why it powers most on-device dictation, including Google's fully on-device recogniser (He et al., 2019).

Attention encoder-decoder#

Listen, Attend and Spell (Chan et al., 2015) applies sequence-to-sequence attention directly to speech, modelling $P(y_u \mid y_{<u}, x)$ with full access to the encoder. It has the strongest implicit language model of the three and is not naturally streaming, since attention is over the complete utterance. Whisper is in this family.

The Conformer encoder (Gulati et al., 2020) interleaves self-attention with convolution to capture global and local structure together, and remains the standard encoder across all three loss families.

Hybrid CTC-attention training (Watanabe et al., 2017) attaches a CTC loss to the encoder alongside the attention decoder, typically weighted around $0.3$. The CTC branch enforces monotonic alignment, which markedly stabilises early training and reduces the attention decoder's tendency to loop or truncate.

Benchmarks, with the caveats attached#

LibriSpeech (Panayotov et al., 2015) is 960 hours of read audiobooks. Strong supervised systems reach roughly 2 percent WER on test-clean and 4 to 5 percent on test-other.

Treat those numbers with suspicion when planning a product. Read speech from volunteers recording deliberately is close to the easiest possible condition. On conversational telephone speech (Switchboard, Fisher), spontaneous meetings (AMI), or accented and code-switched audio, WER for the same architectures is several times higher. Whisper's contribution was largely robustness across conditions rather than a lower number on test-clean, where it is outperformed by LibriSpeech-specialised models.

Fairness, measured#

Koenecke et al. (2020), Racial disparities in automated speech recognition, PNAS 117(14), measured five commercial ASR systems on matched interview audio and reported an average WER of 0.35 for Black speakers against 0.19 for white speakers, with the gap traced to acoustic modelling rather than language modelling. Subsequent work has found comparable disparities across accents, ages and speakers with dysarthria.

Report per-group WER with confidence intervals. Aggregate WER conceals exactly the failures that make a deployed system unusable for some of its users. See fairness metrics for the general framing.

Reading#

• Graves et al. (2006), Connectionist Temporal Classification, ICML — the CTC paper.
• Graves (2012), Sequence Transduction with Recurrent Neural Networks — arxiv.org/abs/1211.3711
• Chan et al. (2015), Listen, Attend and Spell — arxiv.org/abs/1508.01211
• Gulati et al. (2020), Conformer — arxiv.org/abs/2005.08100
• Hannun (2017), Sequence Modeling with CTC, Distill — distill.pub/2017/ctc — the clearest visual explanation available.
• Koenecke et al. (2020), Racial disparities in automated speech recognition, PNAS 117(14).

What to learn next#

• Whisper — running a real, strong recogniser on your own machine.
• Real-time audio pipelines — making recognition fast enough to feel live.
• Recurrent neural networks — the sequence models these systems were built on.