Waveforms and spectrograms

The discrete Fourier transform#

For a finite sequence $x[n]$ of length $N$:

$$ X[k] = \sum_{n=0}^{N-1} x[n]\, e^{-i 2\pi k n / N}, \qquad k = 0, \dots, N-1 $$

Where $x[n]$ is the sampled signal, $N$ is the transform length, $k$ indexes the frequency bins, and $X[k] \in \mathbb{C}$. Bin $k$ corresponds to the physical frequency $f_k = k f_s / N$, so bin spacing is $\Delta f = f_s / N$.

The FFT (Cooley and Tukey, 1965) computes this in $O(N \log N)$ rather than $O(N^2)$ by recursively factoring the transform. For real input, Hermitian symmetry $X[N-k] = \overline{X[k]}$ means only $\lfloor N/2 \rfloor + 1$ bins are independent, which is what rfft returns.

The short-time Fourier transform#

$$ X[m, k] = \sum_{n=0}^{N-1} x[n + mH]\, w[n]\, e^{-i 2\pi k n / N} $$

Where $m$ is the frame index, $H$ is the hop size in samples, $w[n]$ is the analysis window of length $N$, and $N$ is the FFT length. The spectrogram is $|X[m,k]|^2$; the magnitude spectrogram $|X[m,k]|$ is what most pipelines use.

Frame rate is $f_s / H$. For $f_s = 16000$ and $H = 160$ you get 100 frames per second, the near-universal convention in speech systems, chosen so one frame equals 10 ms.

Window functions#

The rectangular window has a narrow main lobe ($2 f_s / N$ between nulls) and dreadful sidelobes, the first at $-13$ dB, decaying at 6 dB per octave. Tapered windows trade main-lobe width for sidelobe suppression:

Hann is the default in speech work because its fast rolloff matters more than its first-sidelobe height when the signal is harmonic.

For invertibility, the window and hop must satisfy the constant overlap-add condition, $\sum_m w[n - mH] = \text{const}$ for all $n$. Hann with $H = N/2$ satisfies it exactly. The weaker NOLA condition (nonzero overlap-add) is what scipy.signal.istft actually requires.

The uncertainty limit#

Define the time spread $\sigma_t$ and frequency spread $\sigma_f$ of a window as the standard deviations of $|w(t)|^2$ and $|W(f)|^2$. Then:

$$ \sigma_t \, \sigma_f \geq \frac{1}{4\pi} $$

This is the Gabor limit (Gabor, 1946), mathematically identical to the Heisenberg inequality — both are properties of Fourier pairs, not of physics or of measurement quality. Equality holds only for a Gaussian window.

The consequence is concrete. A 25 ms window cannot resolve frequency more finely than roughly 40 Hz, whatever zero-padding you apply. Zero-padding interpolates the spectrum onto a denser grid; it adds no information and does not improve resolution. This is the single most common misunderstanding in applied audio work.

Multi-resolution methods sidestep it only by changing the question: wavelets and constant-Q transforms (Brown, 1991) vary the window length with frequency, buying fine timing at high frequencies and fine pitch at low ones.

The mel scale#

Linear frequency bins mismatch human hearing, which discriminates far better at low frequencies. The mel scale (Stevens, Volkmann and Newman, 1937) applies a roughly logarithmic warp. The common HTK form is:

$$ m = 2595 \log_{10}!\left(1 + \frac{f}{700}\right) $$

A mel spectrogram projects the linear magnitude spectrogram through a bank of overlapping triangular filters spaced evenly in mel. For 16 kHz speech, 80 mel bins from a 400-sample window is the modern default, used by Whisper, Tacotron 2 and most neural vocoders. It compresses 201 linear bins to 80 with negligible loss of what matters for speech. This is covered further in audio features and MFCCs.

Phase, and why discarding it hurts#

$|X[m,k]|$ throws away $\arg X[m,k]$. Recognition tolerates this well; reconstruction does not.

Griffin and Lim (1984) recover phase by alternating projection between the set of spectrograms with the target magnitude and the set of consistent STFTs of real signals. It converges to a local optimum and produces a characteristic metallic artefact. Modern neural vocoders (HiFi-GAN, Kong et al., 2020) learn the magnitude-to-waveform map directly and comprehensively outperform it.

Consistency is the underlying issue: an arbitrary array of magnitudes is generally not the STFT of any real signal, because overlapping frames impose constraints that an unconstrained array violates.

Learned front-ends#

Several lines of work replace the fixed STFT with learned filters. SincNet (Ravanelli and Bengio, 2018) parameterises the first convolutional layer as band-pass sinc filters with learnable cutoffs. LEAF (Zeghidour et al., 2021) learns filterbank, pooling and compression jointly. wav2vec 2.0 (Baevski et al., 2020) uses a plain strided convolutional encoder over raw waveform.

The honest summary: learned front-ends match but rarely decisively beat log-mel on large supervised datasets, while costing more compute and more tuning. Whisper, trained on 680k hours, still uses a fixed 80-bin log-mel front-end. Treat "learn the front-end" as a research direction, not a default.

Reading#

• Cooley and Tukey (1965), An Algorithm for the Machine Calculation of Complex Fourier Series, Math. Comp. 19(90).
• Gabor (1946), Theory of Communication, J. IEE 93(26) — the time-frequency uncertainty relation.
• Harris (1978), On the Use of Windows for Harmonic Analysis with the DFT, Proc. IEEE 66(1) — still the reference table for window properties.
• Griffin and Lim (1984), Signal Estimation from Modified Short-Time Fourier Transform, IEEE TASSP 32(2).
• Ravanelli and Bengio (2018), Speaker Recognition from Raw Waveform with SincNet — arxiv.org/abs/1808.00158

What to learn next#

• Audio features and MFCCs — compressing a spectrogram into the numbers a model needs.
• Convolutional neural networks — the architecture that reads a spectrogram as an image.
• Noise reduction — editing the spectrogram, then turning it back into sound.