Text to speech

The classical pipeline, formally#

$$ \text{text} \xrightarrow{\text{TN}} \text{normalised} \xrightarrow{\text{G2P}} \text{phonemes} \xrightarrow{\text{duration}} \text{aligned} \xrightarrow{\text{acoustic}} \mathbf{M} \xrightarrow{\text{vocoder}} \mathbf{x} $$

Where $\mathbf{M} \in \mathbb{R}^{F \times T}$ is a mel spectrogram with $F$ bins over $T$ frames, and $\mathbf{x} \in \mathbb{R}^{N}$ the waveform with $N = T \cdot H$ for hop $H$. The vocoder's job is an upsampling of roughly $256\times$ in time while inventing phase.

Source-filter theory#

Fant (1960) established the model the developer block implements. Voiced speech is an excitation $e(t)$ convolved with the vocal tract response $h(t)$ and lip radiation $r(t)$:

$$ s(t) = e(t) * h(t) * r(t) \quad \Longrightarrow \quad S(f) = E(f)\,H(f)\,R(f) $$

$E(f)$ carries $f_0$ and its harmonics, decaying at roughly $-12$ dB/octave. $H(f)$ has resonant peaks — formants — determined by tract geometry. $R(f)$ is approximately a $+6$ dB/octave differentiator, giving the net $-6$ dB/octave tilt that pre-emphasis in MFCC extraction removes.

Each two-pole resonator in the code realises one formant with poles at $r e^{\pm i\theta}$, where $\theta = 2\pi f_c / f_s$ sets centre frequency and $r = e^{-\pi B / f_s}$ sets bandwidth $B$. Klatt (1980) built a full synthesiser on cascaded and parallel resonators of exactly this form.

Neural acoustic models#

Tacotron 2 (Shen et al., 2018) is an attention-based encoder-decoder emitting mel frames autoregressively, conditioned on character embeddings, paired with a WaveNet vocoder. It reached a mean opinion score of 4.53 against 4.58 for recorded human speech. Its weakness is the attention mechanism: with no monotonicity constraint it can skip or repeat words, and failure is catastrophic rather than graceful.

FastSpeech 2 (Ren et al., 2021) removes attention from the decoder entirely. A variance adaptor predicts per-phoneme duration, pitch and energy, and a length regulator expands the phoneme sequence to frame length. Generation is then non-autoregressive and fully parallel, giving large speedups and eliminating the skip-repeat failure mode. Duration targets come from forced alignment, originally from a teacher model and later from Montreal Forced Aligner or an internal aligner.

VITS (Kim, Kong and Son, 2021) collapses the pipeline into one end-to-end model: a conditional variational autoencoder with normalising flows, adversarial training, and a monotonic alignment search that learns alignment without external supervision. It outputs waveform directly, with no separate vocoder and no mel intermediate. It also introduces a stochastic duration predictor, so the same text produces different rhythms on different runs — a deliberate step toward one-to-many modelling.

The one-to-many problem is the field's structural difficulty: a single text has infinitely many valid renderings differing in prosody, and any model trained with a deterministic regression loss regresses to their mean, which is precisely the flat, lifeless delivery listeners complain about.

Vocoders#

HiFi-GAN (Kong, Kim and Bae, 2020) is the practical default. Its key idea is the multi-period discriminator: several discriminators each viewing the waveform reshaped by a different prime period (2, 3, 5, 7, 11), which captures the periodic structure of voiced speech that a plain 1-D discriminator misses. The generator loss combines adversarial, feature-matching and mel-spectrogram reconstruction terms.

Neural codec language models#

The current frontier reframes TTS as next-token prediction over discrete audio tokens produced by a neural codec such as EnCodec or SoundStream.

VALL-E (Wang et al., 2023) treats TTS as conditional language modelling over EnCodec residual-vector-quantiser tokens, trained on 60,000 hours. It performs zero-shot voice cloning from a 3-second enrolment and preserves the speaker's acoustic environment and emotion — because it learned to continue the prompt, not to read text neutrally.

This capability is the direct subject of voice cloning and its risks, and the safety discussion is not separable from the modelling discussion.

Evaluation#

MOS — listeners rate naturalness 1 to 5. Report confidence intervals and the number of listeners; MOS is not comparable across studies with different rater pools, instructions or reference conditions, and cross-paper MOS comparisons are routinely made and routinely invalid.

CMOS — comparative MOS on a $[-3, +3]$ scale against a reference. Far more reliable than absolute MOS for A/B decisions.

Intelligibility — run ASR over synthesised speech and measure WER. Cheap, automatic, and it catches skipped or repeated words that MOS raters forgive.

Speaker similarity — cosine similarity between speaker embeddings of reference and synthesis, using the models from speaker identification.

Real-time factor — synthesis time divided by audio duration. Below 1.0 is real time; below 0.1 is comfortable for interactive use.

No automatic metric correlates well with human judgement of prosody. This remains the honest bottleneck in TTS evaluation.

Reading#

• Fant (1960), Acoustic Theory of Speech Production — the source-filter model.
• Klatt (1980), Software for a cascade/parallel formant synthesizer, JASA 67(3).
• van den Oord et al. (2016), WaveNet — arxiv.org/abs/1609.03499
• Shen et al. (2018), Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions — arxiv.org/abs/1712.05884
• Ren et al. (2021), FastSpeech 2 — arxiv.org/abs/2006.04558
• Kim, Kong and Son (2021), VITS — arxiv.org/abs/2106.06103
• Kong, Kim and Bae (2020), HiFi-GAN — arxiv.org/abs/2010.05646
• Wang et al. (2023), Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (VALL-E) — arxiv.org/abs/2301.02111

What to learn next#

• Voice cloning and its risks — what happens when a system needs only seconds of your voice.
• Speaker identification — how a machine tells one voice from another.
• Real-time audio pipelines — fitting synthesis into a live conversation.