How computers hear sound

The sampling theorem#

Continuous-time pressure $x(t)$ becomes a discrete sequence under ideal sampling at interval $T_s$:

$$ x[n] = x(nT_s), \qquad f_s = \frac{1}{T_s} $$

Where $n \in \mathbb{Z}$ is the sample index, $T_s$ is the sampling period in seconds, and $f_s$ is the sampling rate in hertz.

Sampling multiplies the signal by a Dirac comb, and multiplication in time is convolution in frequency. The spectrum of the sampled signal is therefore the original spectrum repeated at every multiple of $f_s$:

$$ X_s(f) = f_s \sum_{k=-\infty}^{\infty} X(f - k f_s) $$

Nyquist–Shannon (Shannon, 1949, building on Nyquist, 1928): if $X(f) = 0$ for all $|f| \geq f_s/2$, the copies do not overlap and $x(t)$ is exactly recoverable by sinc interpolation:

$$ x(t) = \sum_{n=-\infty}^{\infty} x[n] \,\operatorname{sinc}!\left(\frac{t - nT_s}{T_s}\right) $$

If the bandlimit condition fails, adjacent copies overlap, and the overlapping energy is indistinguishable from genuine in-band content. A component at $f_0 > f_s/2$ appears at $|f_0 - k f_s|$ for whichever integer $k$ brings it into $[0, f_s/2]$ — exactly the folding printed in the developer block.

The theorem is an idealisation twice over. It assumes strictly bandlimited signals, which necessarily have infinite time support, and it assumes infinitely long sinc reconstruction. Real converters approximate both.

Quantisation#

Rounding each sample to one of $2^b$ levels introduces an error $e[n]$. Under the standard high-resolution assumptions — $e[n]$ uniform on $[-\Delta/2, \Delta/2]$, white, and independent of the signal — the step size and error power are:

$$ \Delta = \frac{2A}{2^b}, \qquad \sigma_e^2 = \frac{\Delta^2}{12} $$

Where $A$ is the peak amplitude of the full-scale range and $b$ is the bit depth. For a full-scale sinusoid this yields the familiar bound:

$$ \mathrm{SNR}_{\mathrm{dB}} = 6.02\,b + 1.76 $$

So 16-bit audio gives about 98 dB of signal-to-noise ratio and 24-bit about 146 dB.

The independence assumption fails at low amplitudes, where the error correlates with the signal and becomes audible as harmonic distortion rather than hiss. Dither — adding roughly one least-significant-bit of noise before quantising — decorrelates the error at a cost of about 3 dB in noise floor. Lipshitz, Wannamaker and Vanderkooy (1992), Quantization and Dither: A Theoretical Survey, is the definitive treatment.

Storage cost and companding#

Linear PCM at rate $f_s$, depth $b$ bits and $C$ channels costs $f_s \cdot b \cdot C$ bits per second: 1.41 Mbit/s for CD stereo, 256 kbit/s for 16 kHz 16-bit mono.

Telephony historically used companding rather than more bits. μ-law (ITU-T G.711, North America and Japan) and A-law (Europe) apply a logarithmic amplitude map before 8-bit quantisation:

$$ F(x) = \operatorname{sgn}(x)\,\frac{\ln(1 + \mu |x|)}{\ln(1 + \mu)}, \qquad \mu = 255 $$

This delivers roughly the perceptual quality of 12-bit linear PCM in 8 bits, because the quantisation steps grow with amplitude and perceived loudness is approximately logarithmic. The same transform reappears in modern generative audio: WaveNet (van den Oord et al., 2016) modelled 8-bit μ-law samples as a 256-way categorical distribution specifically to keep the output softmax tractable.

Rate choices and why they are what they are#

The odd 44.1 kHz figure is an artefact of storing digital audio on video tape: 3 samples per line, 245 usable lines, 60 fields per second gives 44100.

Resampling#

Rate conversion by a rational factor $L/M$ is upsample by $L$ with zero stuffing, low-pass filter at $\min(\pi/L, \pi/M)$, then decimate by $M$. scipy.signal.resample_poly implements this as a polyphase filter bank, which skips computing the samples that decimation would discard and cuts cost by a factor of about $M$.

Naive slicing (x[::M]) omits the filter, and it is among the most common preprocessing bugs in audio ML pipelines. It fails silently: no exception, no warning, only permanently corrupted high-frequency content that surfaces later as an unexplained accuracy gap between training data and production audio.

Reading#

• Shannon (1949), Communication in the Presence of Noise, Proc. IRE 37(1) — the sampling theorem as usually cited.
• Nyquist (1928), Certain Topics in Telegraph Transmission Theory — the earlier bandwidth result it builds on.
• Oppenheim and Schafer, Discrete-Time Signal Processing, 3rd ed., chapter 4 — sampling and reconstruction done properly.
• Smith, The Scientist and Engineer's Guide to Digital Signal Processing — free online, unusually readable.
• van den Oord et al. (2016), WaveNet — arxiv.org/abs/1609.03499

What to learn next#

• Waveforms and spectrograms — turning that list of numbers into a picture of pitch over time.
• Audio features and MFCCs — squeezing a second of sound into a handful of useful numbers.
• NumPy — the array library every example in this section leans on.