Why run a model on the device?

Framing the decision properly#

End-to-end latency for the cloud path is a sum of terms that are usually collapsed into one number and should not be:

$$ L_{\text{cloud}} = t_{\text{capture}} + t_{\text{encode}} + \frac{S}{B_{\uparrow}} + \text{RTT} + t_{\text{queue}} + t_{\text{infer}}^{\text{server}} + \frac{S'}{B_{\downarrow}} $$

• $S$ — request payload size in bytes, $S'$ the response size.
• $B_{\uparrow}, B_{\downarrow}$ — available uplink and downlink bandwidth in bytes per second.
• $\text{RTT}$ — network round-trip time, plus TLS and connection setup on a cold connection.
• $t_{\text{queue}}$ — server-side queueing delay, which grows without bound as utilisation approaches one.
• $t_{\text{encode}}$ — JPEG or Opus encoding, frequently forgotten and frequently 5–15 ms.

The device path is short by comparison:

$$ L_{\text{device}} = t_{\text{capture}} + t_{\text{preprocess}} + t_{\text{infer}}^{\text{device}} $$

Two consequences follow. First, $L_{\text{cloud}}$ has a hard floor set by the speed of light and the mobile radio, independent of how fast your servers are: LTE control-plane wake-up alone costs tens of milliseconds after an idle period. Second, $L_{\text{cloud}}$ has a heavy tail, because $t_{\text{queue}}$ and RTT both have long-tailed distributions. Reporting only the median hides the failure mode users actually notice.

Tail latency is the real argument#

For a request touching $n$ independent services, the probability that none exceeds its own p99 is $0.99^n$. At $n = 10$ roughly one request in ten exceeds a p99 somewhere. Dean and Barroso (2013), The Tail at Scale, is the canonical treatment.

On-device inference has no fan-out, no queueing and no radio. Its latency distribution is tight, bounded by thermal state rather than by network variance. For interactive perception — autofocus, gesture tracking, live captions — the p99 is the specification and the median is decoration.

Energy, honestly#

On-device is not automatically the lower-energy option. The comparison is between local compute energy and radio energy:

$$ E_{\text{cloud}} \approx P_{\text{radio}} \cdot \left(t_{\text{tx}} + t_{\text{tail}}\right), \qquad E_{\text{device}} \approx P_{\text{compute}} \cdot t_{\text{infer}} $$

The term people omit is $t_{\text{tail}}$: the cellular radio stays in a high-power RRC-connected state for several seconds after a transmission before demoting. A single small request can therefore cost as much energy as several seconds of radio-on time. Balasubramanian et al. (2009), Energy consumption in mobile phones, measured this tail directly and found it dominating the energy cost of small, infrequent transfers.

The practical rule that falls out: many small requests are the worst pattern. If you must use the network, batch and coalesce so the radio wakes once.

Privacy is a formal property, not a location#

"The data stayed on the device" is a claim about information flow, and moving the matrix multiply is neither necessary nor sufficient for it.

• Model outputs leak. Membership inference (Shokri et al., 2017) recovers whether a record was in the training set from confidence scores alone.
• Gradients leak. Federated learning keeps raw data local yet transmits updates from which inputs can be reconstructed (Zhu et al., Deep Leakage from Gradients, 2019). Differential privacy is the mitigation with a proof attached; on-device execution alone is not.
• Telemetry leaks. Crash dumps and analytics routinely carry the very input the on-device path was meant to protect.

On-device inference is a strong default and a weak guarantee. If you need a guarantee, state the threat model and use a mechanism with a proof attached.

Cascades and the economics of confidence#

The two-stage cascade in the developer block has a clean cost model. With device cost $c_d$, cloud cost $c_c$, and escalation rate $\alpha$ (the fraction of inputs the device defers):

$$ \mathbb{E}[\text{cost}] = c_d + \alpha \, c_c, \qquad \mathbb{E}[\text{error}] = (1 - \alpha)\varepsilon_d^{\text{acc}} + \alpha \varepsilon_c $$

where $\varepsilon_d^{\text{acc}}$ is the device model's error on the inputs it accepts and $\varepsilon_c$ the cloud model's error on those it receives. The whole design problem is making $\varepsilon_d^{\text{acc}} \ll \varepsilon_d$ by deferring exactly the hard inputs.

That requires calibrated confidence, and neural network softmax outputs are famously not calibrated (Guo et al., 2017, On Calibration of Modern Neural Networks). Temperature scaling on a held-out set is the cheapest fix and usually enough. Without it, the escalation threshold is a number with no meaning.

Reading#

• Dean and Barroso, The Tail at Scale, CACM 2013.
• Balasubramanian, Balasubramanian and Venkataramani, Energy consumption in mobile phones: a measurement study, IMC 2009.
• Guo et al., On Calibration of Modern Neural Networks, ICML 2017 — arxiv.org/abs/1706.04599
• Shokri et al., Membership Inference Attacks Against Machine Learning Models, IEEE S&P 2017 — arxiv.org/abs/1610.05820
• Kang et al., Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge, ASPLOS 2017 — splitting one network across device and server at the cheapest cut point.

What to learn next#

• Model compression — how a model gets small enough to ship.
• Latency and throughput — measuring the speed claims in this lesson properly.
• Model serving — what you are choosing not to do when you go on-device.