Audio features and MFCCs

The MFCC pipeline, stated precisely#

Given a frame $x[n]$ of length $N$ with window $w[n]$:

1. Pre-emphasis (optional, common in classical systems):

$$ x'[n] = x[n] - \alpha\, x[n-1], \qquad \alpha \approx 0.97 $$

A first-order high-pass that flattens the roughly $-6$ dB/octave spectral tilt of voiced speech, boosting the high-frequency formants that carry consonant identity.

2. Power spectrum:

$$ P[k] = \frac{1}{N}\left| \sum_{n=0}^{N-1} x'[n]\,w[n]\, e^{-i2\pi kn/N} \right|^2 $$

3. Mel filterbank. Map hertz to mel with the HTK formula:

$$ m(f) = 2595 \log_{10}!\left(1 + \frac{f}{700}\right), \qquad f(m) = 700\left(10^{m/2595} - 1\right) $$

Place $M$ triangular filters with centres uniformly spaced in mel between $f_{\min}$ and $f_{\max}$, each spanning its two neighbouring centres. Filter $j$ has response $H_j[k]$, and the band energy is:

$$ E_j = \sum_{k} H_j[k]\, P[k], \qquad j = 1, \dots, M $$

Typical $M$ is 26 to 40 for MFCC, and 80 for neural log-mel front-ends.

4. Log compression: $L_j = \log(E_j + \epsilon)$. This approximates loudness perception and, importantly, converts the multiplicative source-filter relationship into an additive one.

5. Discrete cosine transform, type II:

$$ c_i = \sum_{j=1}^{M} L_j \cos!\left[\frac{\pi i}{M}\left(j - \tfrac{1}{2}\right)\right], \qquad i = 0, \dots, M-1 $$

Keep $i = 0, \dots, 12$. Here $c_i$ is the $i$-th cepstral coefficient, $L_j$ the log mel band energy, and $M$ the number of filters.

Why the DCT is there#

Speech production is well modelled as a source convolved with a filter: $s(t) = e(t) * h(t)$, where $e$ is the glottal excitation and $h$ the vocal tract impulse response. In the frequency domain this is a product, $|S(f)| = |E(f)| \cdot |H(f)|$, and the logarithm turns it into a sum:

$$ \log|S(f)| = \log|E(f)| + \log|H(f)| $$

The excitation term is a fast-varying ripple, with period equal to $f_0$ across frequency. The vocal tract term is a slowly varying envelope. Taking a further transform along the frequency axis separates them by rate of variation: low quefrency coefficients carry the envelope, high ones carry the pitch harmonics. Truncating to 13 coefficients is a low-pass operation in quefrency, keeping the vocal tract and discarding the pitch.

This is the cepstrum, named by Bogert, Healy and Tukey (1963) by reversing the first syllable of "spectrum" — hence the companion coinages quefrency, liftering and alanysis. The DCT stands in for the inverse Fourier transform because the log power spectrum is real and even, which makes the two equivalent up to scaling.

The DCT has a second, more practical role: it approximately decorrelates the filterbank energies. Adjacent mel bands overlap and are strongly correlated, and diagonal-covariance Gaussian mixture models — the acoustic model of choice from the 1980s until roughly 2012 — depend on that decorrelation to be viable. When the acoustic model became a neural network, the constraint vanished, and with it the main reason for step 5.

Deltas and liftering#

Static coefficients ignore dynamics. The standard remedy appends regression estimates of the first and second time derivatives:

$$ d_t = \frac{\sum_{\theta=1}^{\Theta} \theta\,(c_{t+\theta} - c_{t-\theta})}{2\sum_{\theta=1}^{\Theta}\theta^2}, \qquad \Theta = 2 $$

Giving 13 static, 13 delta and 13 delta-delta coefficients, the 39-dimensional vector that dominated ASR for two decades.

Cepstral liftering rescales $c_i \leftarrow (1 + \frac{L}{2}\sin\frac{\pi i}{L}) c_i$ with $L = 22$, equalising the variance across coefficient index. It matters for GMM systems and is irrelevant for neural ones.

Cepstral mean and variance normalisation subtracts the per-utterance mean and divides by the per-utterance standard deviation. Because a linear channel is additive in the log-spectral domain, subtracting the mean removes a fixed channel response exactly. This remains one of the highest-value preprocessing steps in existence for mismatched-channel conditions.

Where MFCCs stand today#

The empirical picture, stated without hedging:

• For GMM-HMM systems, MFCC-39 was the correct choice, and alternatives measurably lost.
• For neural acoustic models with more than a few hundred hours of data, log-mel filterbank energies (typically 80 bins, no DCT) consistently match or beat MFCCs. The DCT discards information that a network can exploit. Whisper, Conformer and wav2vec 2.0 fine-tuning recipes all use log-mel or raw waveform.
• For low-resource, low-compute, or interpretable settings — keyword spotting on a microcontroller, a classifier over a few hundred labelled clips, forensic analysis needing an auditable pipeline — MFCCs remain a strong and defensible baseline.

Reporting an MFCC baseline alongside a log-mel one costs almost nothing and is good practice. Claiming MFCCs are obsolete is as wrong as claiming they are current best practice.

Alternatives worth knowing#

PLP (Hermansky, 1990) applies equal-loudness pre-emphasis and intensity-loudness cube-root compression before an all-pole model, and is more robust than MFCC under some noise conditions.

Gammatone filterbank features replace triangular filters with gammatone filters fitted to auditory nerve responses.

Learned front-ends — SincNet, LEAF, and the convolutional encoder in wav2vec 2.0 — are covered in waveforms and spectrograms.

Self-supervised representations are the genuine successor. wav2vec 2.0, HuBERT and WavLM produce contextual frame embeddings that outperform every hand-designed feature on nearly every downstream task, at the cost of a large pretrained encoder in your pipeline.

Reading#

• Davis and Mermelstein (1980), Comparison of Parametric Representations for Monosyllabic Word Recognition, IEEE TASSP 28(4) — the MFCC paper.
• Bogert, Healy and Tukey (1963), The Quefrency Alanysis of Time Series for Echoes — where the cepstrum comes from.
• Hermansky (1990), Perceptual Linear Predictive Analysis of Speech, JASA 87(4).
• Stevens, Volkmann and Newman (1937), A Scale for the Measurement of the Psychological Magnitude Pitch, JASA 8(3) — the original mel experiments.
• Baevski et al. (2020), wav2vec 2.0 — arxiv.org/abs/2006.11477

What to learn next#

• Speech recognition — turning these frame fingerprints into words.
• Audio classification — training a real classifier on features like these.
• Feature engineering — the same compression instinct, applied to tables instead of sound.