Real-time audio pipelines

Latency decomposition#

Total response latency for a spoken-dialogue turn:

$$ L = \underbrace{L_{\text{buf}} + L_{\text{net}}^{\uparrow} + L_{\text{ASR}}^{\text{stream}}}_{\text{overlapped with speech}}

• \underbrace{L_{\text{end}} + L_{\text{ASR}}^{\text{final}} + L_{\text{LLM}}^{\text{TTFT}} + L_{\text{TTS}}^{\text{first}} + L_{\text{net}}^{\downarrow} + L_{\text{play}}}_{\text{perceived}} $$

Only the second group is perceived, because the first completes before the user stops speaking. This decomposition is the reason streaming architectures win: they do not reduce total work, they relocate it under the user's own speech.

The binding constraint for sustainability is per-block real-time factor:

$$ \text{RTF} = \frac{t_{\text{process}}}{t_{\text{audio}}} < 1 $$

RTF $\geq 1$ makes the queue a non-stationary process with unbounded expected length. Under variable service times, queueing theory gives the sharper practical bound: expected delay grows as $\rho/(1-\rho)$ with utilisation $\rho$, so latency degrades severely well before $\rho = 1$. Target $\rho \lesssim 0.5$, and use a bounded queue with a drop policy so overload degrades gracefully rather than catastrophically.

Streaming recognition#

Offline attention encoder-decoders attend over the full utterance and cannot stream. The streaming-capable families:

RNN-T emits without future context by construction, and is the dominant on-device choice (He et al., 2019).

Monotonic chunkwise attention (MoChA, Chiu and Raffel, 2018) restricts attention to a bounded chunk, giving a tunable latency-accuracy trade.

Chunked and cached Conformer processes fixed chunks with cached left context and limited right context. Right-context lookahead $R$ buys accuracy at exactly $R$ milliseconds of latency, and the exchange rate is the key design curve. Emformer (Shi et al., 2021) makes this efficient by caching memory across chunks.

Whisper is not streaming. Its 30-second fixed window makes chunking an external heuristic, with boundary artefacts and re-decoding cost. Treat streaming Whisper wrappers as a compromise, not a streaming model.

Latency-quality can also be shaped by the loss. FastEmit (Yu et al., 2021) adds a regulariser penalising blank emission in RNN-T, encouraging earlier token emission and cutting partial-result latency substantially at a small WER cost.

Endpointing#

The developer block shows fixed-threshold endpointing and its failure. Better formulations:

VAD-based with adaptive thresholds conditioned on observed speech rate and hesitation.

End-of-query classifiers (Maas et al., 2018; Chang et al., 2019) predict turn completion from acoustic and decoder state, using the partial hypothesis. A partial that parses as a complete request licenses a shorter wait than one ending in a determiner.

End-of-utterance as a token. Train the recogniser to emit an explicit </s>, folding endpointing into decoding and removing the separate timer.

The metric is a two-sided cost: early cutoffs truncate user speech, and late endpoints add latency. Report both — median endpoint latency and cutoff rate — because either one alone is optimised to nothing by moving the threshold.

Full-duplex and barge-in#

Half-duplex — listen, then speak, then listen — is a simplification human conversation does not obey. Turn-taking gaps in natural conversation centre near 200 ms (Stivers et al., 2009), which is shorter than most pipelines' endpoint wait alone. Overlap is normal, not exceptional.

Barge-in requires acoustic echo cancellation: an adaptive filter estimating the loudspeaker-to-microphone path and subtracting the known reference. Difficulty comes from nonlinear loudspeaker distortion, clock drift between playback and capture, and echo path changes when anything in the room moves. Residual echo suppression is a separate post-filter stage, and it is where most of the practical engineering sits.

Recent full-duplex dialogue models fold listening and speaking into one continuously running model, learning when to yield rather than being told by a timer. This is an active area rather than settled practice.

Transport#

Opus (RFC 6716) is the codec for interactive voice: 2.5 to 60 ms frames, 6 to 510 kbit/s, seamless speech-music switching, and built-in forward error correction and packet-loss concealment. Below roughly 30 ms frames, per-packet header overhead grows significant.

Jitter buffers trade latency for loss concealment. Adaptive buffers track observed jitter and are the standard, but note the interaction: an adaptive buffer that grows during congestion silently adds to $L_{\text{net}}$ and to perceived latency, so it must be monitored as part of the budget rather than treated as transport-layer detail.

WebRTC bundles echo cancellation, VAD, jitter buffering and Opus, and reimplementing any of it is rarely justified.

Measurement#

Instrument per-stage timestamps on a single request identifier and export histograms, not averages. Report p50, p95 and p99 for each stage plus end to end. Tail latency is what users perceive, and a stage with a well-behaved mean and a heavy tail — garbage collection, cold model load, a cache miss — is invisible in averages and painfully visible to a user.

Track RTF distribution per block, queue depth over time, and drop counts. Rising queue depth is the leading indicator of the failure described in the developer block, and it is observable minutes before latency becomes visible.

Reading#

• He et al. (2019), Streaming End-to-End Speech Recognition for Mobile Devices — arxiv.org/abs/1811.06621
• Chiu and Raffel (2018), Monotonic Chunkwise Attention — arxiv.org/abs/1712.05382
• Shi et al. (2021), Emformer — arxiv.org/abs/2010.10759
• Yu et al. (2021), FastEmit — arxiv.org/abs/2010.11148
• Stivers et al. (2009), Universals and cultural variation in turn-taking in conversation, PNAS 106(26).
• Valin et al. (2012), Definition of the Opus Audio Codec, RFC 6716.

What to learn next#

• Model serving — running these models behind an API that holds its latency.
• Latency and throughput — the general trade this whole lesson is one case of.
• Whisper — why an offline recogniser needs work before it can stream.