Noise reduction

Problem statement#

Given $y[n] = x[n] + d[n]$ with speech $x$ and additive noise $d$ assumed uncorrelated, recover an estimate $\hat{x}$. In the STFT domain, writing $Y_{k,m}$, $X_{k,m}$, $D_{k,m}$ for bin $k$ and frame $m$, every method in this family computes a real or complex gain:

$$ \hat{X}{k,m} = G{k,m}\, Y_{k,m} $$

The methods differ only in how $G$ is derived.

Spectral subtraction#

Boll (1979) proposed, in the generalised power form:

$$ |\hat{X}{k,m}|^{\gamma} = \max\left(|Y{k,m}|^{\gamma} - \alpha\, |\hat{D}{k}|^{\gamma},\ \beta\, |Y{k,m}|^{\gamma}\right) $$

Where $\gamma = 1$ gives magnitude subtraction and $\gamma = 2$ power subtraction, $\alpha \geq 1$ is the over-subtraction factor, $\beta$ the spectral floor, and $|\hat{D}_k|$ the noise estimate. The developer block implements $\gamma = 1$ and sweeps $\alpha$ and $\beta$.

Berouti, Schwartz and Makhoul (1979) made $\alpha$ a decreasing function of the frame SNR, subtracting more where the signal is weak, which is the standard practical form.

Musical noise, explained properly. $|Y_{k,m}|$ fluctuates around its expectation even where only noise is present. Subtracting the mean leaves a residual whose half-wave-rectified values are non-zero at randomly scattered $(k,m)$ points. Isolated, short-duration, narrowband survivors are perceptually tonal. The floor $\beta$ works because it converts isolated spikes on a zero background into spikes on a masking background, and auditory masking then hides them. The kurtosis rise in the developer output is a direct measurement of this residual becoming spiky, and Uemura et al. (2009) proposed kurtosis ratio as a musical-noise metric on exactly that reasoning.

Wiener filtering#

The MMSE-optimal linear gain under Gaussian assumptions:

$$ G^{\text{W}}{k,m} = \frac{\xi{k,m}}{1 + \xi_{k,m}}, \qquad \xi_{k,m} = \frac{\mathbb{E}\left[|X_{k,m}|^2\right]}{\mathbb{E}\left[|D_{k,m}|^2\right]} $$

Where $\xi$ is the a priori SNR — the ratio of clean speech power to noise power, which is unknown because it depends on the quantity being estimated. The a posteriori SNR $\gamma_{k,m} = |Y_{k,m}|^2 / \mathbb{E}[|D_{k,m}|^2]$ is observable.

Ephraim and Malah (1984) resolved this with the decision-directed estimator:

$$ \hat{\xi}{k,m} = \mu \frac{|\hat{X}{k,m-1}|^2}{\mathbb{E}[|D_{k}|^2]} + (1-\mu)\max(\gamma_{k,m} - 1,\ 0) $$

with $\mu \approx 0.98$. This recursive smoothing is the single most important idea in classical speech enhancement, and it is why MMSE-STSA produces far less musical noise than spectral subtraction: heavy temporal smoothing of $\hat{\xi}$ suppresses precisely the frame-to-frame fluctuations that create isolated survivors.

The MMSE short-time spectral amplitude estimator itself minimises $\mathbb{E}[(|X| - |\hat{X}|)^2]$ under a Gaussian model, giving a gain in terms of confluent hypergeometric functions. Ephraim and Malah (1985) extended it to the log-spectral amplitude, which better matches perception and remains a competitive classical baseline forty years later.

Noise estimation#

The genuinely hard subproblem. Options:

• VAD-gated averaging. Update the estimate only in non-speech frames. Fails when the VAD fails, and VADs fail at low SNR.
• Minimum statistics (Martin, 2001). Track the minimum of smoothed noisy power over a sliding window and compensate for the bias of the minimum operator. Requires no VAD and handles non-stationary noise, at the cost of a lag on the order of the window.
• MMSE-based noise power estimation (Gerkmann and Hendriks, 2012). Estimate noise power with a speech-presence-probability weighting. Lower tracking delay than minimum statistics and the usual modern choice.

Neural approaches#

Masking in the T-F domain. Predict a mask applied to the noisy STFT. The ideal ratio mask is:

$$ \text{IRM}{k,m} = \left(\frac{|X{k,m}|^2}{|X_{k,m}|^2 + |D_{k,m}|^2}\right)^{1/2} $$

Real-valued masks leave phase untouched, capping quality. The complex ideal ratio mask (Williamson, Wang and Wang, 2016) predicts real and imaginary parts, enabling phase correction and measurably outperforming magnitude masks.

Time-domain separation. Conv-TasNet (Luo and Mesgarani, 2019) replaces the STFT with a learned encoder-decoder and estimates masks in that latent space, sidestepping the phase problem entirely. It substantially outperformed STFT masking on separation and reframed the field.

Complex-domain recurrent networks. DCCRN (Hu et al., 2020) uses complex convolutions and a complex LSTM, and won the DNS Challenge 2020 real-time track.

Efficiency-first. RNNoise (Valin, 2018) combines a small GRU predicting band gains with classical pitch filtering, running in well under 1 percent of a mobile core. DeepFilterNet (Schröter et al., 2022) achieves full-band enhancement in real time on a CPU with a per-frame deep filter over a small complex neighbourhood.

The DNS Challenge series (Reddy et al., 2020 onward) is the field's benchmark and provides the standard training data and real-recording test sets.

Evaluation#

• SI-SDR (Le Roux et al., 2019). Scale-invariant signal-to-distortion ratio, which fixes SDR's exploitable sensitivity to rescaling. The default objective metric for separation and enhancement.
• PESQ (ITU-T P.862) and POLQA (P.863). Intrusive perceptual quality, mapping to a MOS-like scale. PESQ was designed for codec and network degradation, and it correlates imperfectly with neural-enhancement artefacts — it can reward outputs listeners dislike.
• STOI (Taal et al., 2011). Predicts intelligibility rather than quality, and the two dissociate: enhancement often raises quality while lowering or not changing intelligibility.
• DNSMOS (Reddy et al., 2021). A non-intrusive learned MOS estimator, usable on real recordings with no clean reference.

The enhancement-for-ASR caveat is real and well documented. Front-end enhancement frequently degrades downstream recognition, because the enhancement objective targets perceptual quality while the recogniser was trained on noisy speech and relies on cues the enhancer removes. Joint training, or feeding both streams, is the standard remedy. Never assume a denoiser helps a machine listener — measure WER end to end.

Reading#

• Boll (1979), Suppression of Acoustic Noise in Speech Using Spectral Subtraction, IEEE TASSP 27(2).
• Ephraim and Malah (1984), Speech Enhancement Using a Minimum Mean-Square Error Short-Time Spectral Amplitude Estimator, IEEE TASSP 32(6).
• Martin (2001), Noise Power Spectral Density Estimation Based on Optimal Smoothing and Minimum Statistics, IEEE TSAP 9(5).
• Luo and Mesgarani (2019), Conv-TasNet — arxiv.org/abs/1809.07454
• Valin (2018), A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement — arxiv.org/abs/1709.08243
• Le Roux et al. (2019), SDR — Half-baked or Well Done? — arxiv.org/abs/1811.02508

What to learn next#

• Wake-word detection — a tiny model listening through exactly this kind of noise.
• Real-time audio pipelines — doing all of this inside a few milliseconds.
• Waveforms and spectrograms — the representation every method here edits.