Speaker identification

Problem definitions#

Verification is a hypothesis test. Given enrolment $\mathcal{E}$ and test $\mathcal{T}$, decide between $H_0$ (same speaker) and $H_1$ (different), by comparing a log-likelihood ratio against a threshold:

$$ s = \log \frac{p(\mathcal{E}, \mathcal{T} \mid H_0)}{p(\mathcal{E}, \mathcal{T} \mid H_1)} \gtrless \tau $$

Identification is closed-set or open-set classification over $N$ enrolled speakers. Diarisation is unsupervised segmentation plus clustering, evaluated by diarisation error rate — the sum of missed speech, false alarm and speaker confusion time, divided by total speech.

Four generations#

GMM-UBM (Reynolds et al., 2000). Train a universal background model on pooled speech, then MAP-adapt the means per speaker. Scoring is a log-likelihood ratio against the UBM.

Supervector and i-vector (Dehak et al., 2011). Stack the adapted GMM means into a supervector $M$ and posit a low-rank total variability subspace:

$$ M = m + T w $$

Where $m$ is the UBM supervector, $T \in \mathbb{R}^{CF \times R}$ the total variability matrix, and $w \in \mathbb{R}^{R}$ (typically $R = 400$–600) the i-vector. Unlike earlier joint factor analysis, $T$ deliberately mixes speaker and channel variability, deferring their separation to the backend. This is the first true embedding in speaker recognition, and its architecture — a fixed-length vector plus a probabilistic backend — still describes the field.

x-vector (Snyder et al., 2018). A time-delay neural network over frames, a statistics pooling layer taking the mean and standard deviation across time, then affine layers whose penultimate activation is the embedding. Pooling is the crucial piece: it maps variable-length input to a fixed vector, and the standard deviation term carries genuinely useful information beyond the mean. Trained with speaker-classification cross-entropy and heavy augmentation (MUSAN noise, RIR reverberation), x-vectors decisively beat i-vectors given enough data.

ECAPA-TDNN (Desplanques et al., 2020). Adds squeeze-excitation to the Res2Net-style blocks, aggregates features across multiple layers, and replaces plain pooling with attentive statistics pooling, so frames contribute according to learned importance. It is the standard strong open baseline, reporting roughly 0.9 percent EER on VoxCeleb1 test in the original paper.

Margin losses#

Plain softmax separates training classes without making embeddings metrically useful at test time on unseen speakers. Angular margin losses fix this directly. AAM-softmax (ArcFace, Deng et al., 2019) normalises weights and features and inserts an additive angular margin:

$$ \mathcal{L} = -\frac{1}{N}\sum_{i=1}^{N} \log \frac{e^{s\cos(\theta_{y_i} + m)}} {e^{s\cos(\theta_{y_i} + m)} + \sum_{j \neq y_i} e^{s\cos\theta_j}} $$

Where $\theta_j$ is the angle between embedding $i$ and class weight $j$, $m$ the angular margin (typically 0.2), $s$ the scale (typically 30), and $y_i$ the true speaker. The margin forces intra-class compactness and inter-class separation on the hypersphere, which is exactly the geometry cosine scoring assumes at test time. This is the single largest architecture-independent gain of the last several years.

Backends#

Cosine scoring after centring and length normalisation is the standard for margin-trained embeddings — precisely the two operations demonstrated in the developer block.

PLDA (Prince and Elder, 2007; Ioffe, 2006) models an embedding as $\phi_{ij} = \mu + F h_i + \epsilon_{ij}$, where $h_i$ is a speaker factor and $\epsilon_{ij}$ within-speaker noise, and computes an exact likelihood ratio. PLDA dominated the i-vector era. With margin-trained embeddings its advantage largely disappears, and cosine scoring is both simpler and competitive.

Adaptive score normalisation (AS-norm) standardises a score against its distribution over an imposter cohort. It typically yields a 10 to 20 percent relative EER reduction for a few lines of code and is the highest-value cheap addition to a deployed system.

Evaluation#

• EER — the operating point where false accept equals false reject. A summary statistic, not a design target.
• minDCF — minimum detection cost, $C_{\text{det}} = C_{\text{miss}} P_{\text{target}} P_{\text{miss}} + C_{\text{fa}} (1 - P_{\text{target}}) P_{\text{fa}}$. NIST SRE conventionally uses $P_{\text{target}} = 0.01$ or $0.05$. This is the metric that reflects asymmetric costs.
• DET curves — false reject against false accept on a normal deviate scale, which makes the low-error region readable.

VoxCeleb 1 and 2 (Nagrani et al., 2017, 2018) are the standard public benchmarks, built from YouTube interview audio. Note the domain: celebrity interviews, largely English, in reasonable acoustic conditions. Telephone, far-field and multilingual performance are substantially worse, and VoxCeleb numbers do not transfer.

Spoofing, and why this section is not optional#

Speaker verification systems are attacked by replay, by voice conversion and by synthesis. The ASVspoof challenge series (2015 onward) benchmarks countermeasures, and the ASVspoof 2019 and 2021 logical-access conditions specifically target neural TTS and voice conversion.

Two findings matter for anyone deploying this. First, a verification system without a countermeasure accepts good synthetic speech at high rates — the embedding captures vocal-tract characteristics, which synthesis reproduces. Second, countermeasures generalise poorly to attack types absent from their training data, which is the usual state of affairs against an adversary who reads the literature.

The current framing is SASV — spoofing-aware speaker verification — treating verification and countermeasure as one jointly optimised system rather than two stacked filters. Treat published EERs on clean benchmarks as an upper bound on what you will see against an adversary. See adversarial attacks.

Reading#

• Reynolds, Quatieri and Dunn (2000), Speaker Verification Using Adapted Gaussian Mixture Models, Digital Signal Processing 10.
• Dehak et al. (2011), Front-End Factor Analysis for Speaker Verification, IEEE TASLP 19(4).
• Snyder et al. (2018), X-vectors: Robust DNN Embeddings for Speaker Recognition, ICASSP.
• Desplanques et al. (2020), ECAPA-TDNN — arxiv.org/abs/2005.07143
• Deng et al. (2019), ArcFace — arxiv.org/abs/1801.07698
• Nagrani et al. (2017), VoxCeleb — arxiv.org/abs/1706.08612
• Wu et al. (2015 onward), ASVspoof challenge series — asvspoof.org

What to learn next#

• Voice cloning and its risks — why a voice fingerprint is not a lock.
• Audio classification — the same embedding idea applied to sounds instead of people.
• Embeddings — the general idea of turning things into comparable points.