AI on a Raspberry Pi

Setup#

On the Pi, running 64-bit Raspberry Pi OS:

sudo apt update && sudo apt install -y python3-pip
pip install onnxruntime numpy psutil

onnxruntime has official aarch64 wheels, so this is a download rather than a compile. Use the 64-bit OS — the 32-bit build is slower and has thinner package support.

The benchmark below runs unchanged on your laptop too, which is the point: measure it in both places and compare.

A camera-shaped benchmark#

Weights are random. Latency depends on a model's shape, not on what it has learned, so a randomly initialised network measures exactly like a trained one of the same architecture. That keeps this script to a few seconds with no download.

import os, time, statistics as st, numpy as np, psutil, torch, torch.nn as nn, onnxruntime as ort
from onnxruntime.quantization import quantize_static, CalibrationDataReader, QuantType, QuantFormat

torch.manual_seed(0)
cnn = nn.Sequential(
    nn.Conv2d(3, 32, 3, stride=2, padding=1), nn.ReLU(),
    nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Conv2d(64, 128, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Conv2d(128, 128, 3, padding=1), nn.ReLU(),
    nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(128, 10),
).eval()
torch.onnx.export(cnn, torch.zeros(1, 3, 128, 128), "camera.onnx",
                  input_names=["image"], output_names=["scores"], opset_version=13)

rng = np.random.default_rng(0)

class Calib(CalibrationDataReader):
    """Real inputs belong here. Noise is fine for a speed test and wrong for an accuracy test."""
    def __init__(self, n=20):
        self.data = iter([{"image": rng.standard_normal((1, 3, 128, 128), dtype=np.float32)} for _ in range(n)])
    def get_next(self):
        return next(self.data, None)

quantize_static("camera.onnx", "camera_int8.onnx", Calib(), quant_format=QuantFormat.QDQ,
                activation_type=QuantType.QUInt8, weight_type=QuantType.QInt8)

print("camera.onnx      : %7.1f KB, %d parameters"
      % (os.path.getsize("camera.onnx") / 1024, sum(p.numel() for p in cnn.parameters())))
print("camera_int8.onnx : %7.1f KB" % (os.path.getsize("camera_int8.onnx") / 1024))
print()

frame = rng.standard_normal((1, 3, 128, 128), dtype=np.float32)
proc = psutil.Process()
print("model              threads   ms/frame   frames/second   extra RAM")
for path in ["camera.onnx", "camera_int8.onnx"]:
    for threads in [1, 2, 4]:
        before = proc.memory_info().rss
        so = ort.SessionOptions()
        so.intra_op_num_threads = threads      # a Pi has four cores; try all of them
        so.inter_op_num_threads = 1
        sess = ort.InferenceSession(path, so, providers=["CPUExecutionProvider"])
        for _ in range(20):
            sess.run(None, {"image": frame})
        rss = (proc.memory_info().rss - before) / 1024 / 1024
        times = []
        for _ in range(100):
            t = time.perf_counter(); sess.run(None, {"image": frame}); times.append((time.perf_counter() - t) * 1000)
        ms = st.median(times)
        print("%-18s %7d   %8.2f   %13.0f   %6.1f MB" % (path, threads, ms, 1000 / ms, rss))
        del sess

camera.onnx      :   947.6 KB, 242122 parameters
camera_int8.onnx :   246.0 KB

model              threads   ms/frame   frames/second   extra RAM
camera.onnx              1       2.32             432      5.2 MB
camera.onnx              2       1.21             824      2.6 MB
camera.onnx              4       0.65            1547      2.7 MB
camera_int8.onnx         1       0.85            1173      1.6 MB
camera_int8.onnx         2       0.48            2091      1.5 MB
camera_int8.onnx         4       0.34            2980      1.8 MB

That output is from a desktop, not a Pi. The file sizes and parameter count will reproduce exactly for you; the timings will not, and they are the whole point of running it yourself. Expect a Raspberry Pi to be very much slower — this is a desktop CPU with far higher clocks and wider vector units. Run the script on both and write down the ratio, because that ratio is what you will need for every future estimate.

The extra RAM column is noisy. The first row absorbs one-off allocations that later rows reuse, so read it as an order of magnitude, not a measurement.

Two results worth carrying with you#

Threads scale, almost linearly. 2.32 ms on one thread, 0.65 ms on four — a 3.6x speed-up from 4 cores. This is the single biggest free win on a Pi. ONNX Runtime defaults to a sensible thread count, but it does not always guess right in a container or under a process manager. Set intra_op_num_threads explicitly.

Here, at last, int8 was genuinely faster. 2.32 ms down to 0.85 ms on one thread — 2.7 times. Contrast that with quantisation in practice, where int8 made a small dense model five times slower.

The difference is the model and the method. This is a convolutional network quantised statically, so the whole graph runs in integers with no per-call conversion. That is exactly the case ARM CPUs are optimised for, through their integer dot-product instructions. On a Pi the gap is usually larger than on a desktop, because desktop float units are relatively stronger.

Note also that dynamic quantisation would have failed here. quantize_dynamic on a convolutional model produces ConvInteger nodes that ONNX Runtime's CPU backend does not implement:

NOT_IMPLEMENTED : Could not find an implementation for ConvInteger(10) node with name '/0/Conv_quant'

For convolutional models, use quantize_static with a calibration reader. For dense and transformer models, quantize_dynamic is fine.

Which Pi#

Published specifications, which decide what is even possible:

The 512 MB on a Pi Zero 2 W is the number that stops projects. Runtime, model, camera buffers and the operating system all come out of it. Check extra RAM in your own run and add the model file size and your image buffers before you order anything.

Adding real acceleration#

Two options that change the arithmetic completely, both int8-only:

• Raspberry Pi AI HAT+ — a Hailo-8L or Hailo-8 accelerator over PCIe on a Pi 5. Rated at 13 and 26 TOPS respectively. Uses Hailo's own toolchain and its own compiled model format, not ONNX directly.
• Coral USB Accelerator — an Edge TPU on USB. Requires a fully-integer TFLite model compiled with edgetpu_compiler. Anything the compiler cannot place falls back to the CPU, silently.

Both mean a second conversion step and a second set of operator restrictions. Get the model working on the CPU first, with a measured baseline. Then decide whether you actually need the accelerator, because frequently you do not.

Common mistakes#

Under-powering it. A Pi 5 wants a 5.1 V, 5 A supply; a Pi 4 wants 5.1 V, 3 A. Phone chargers cause silent throttling. Check with:

vcgencmd get_throttled

0x0 means fine. Anything else is a power or heat problem, and it is the correct first suspect whenever a Pi is inexplicably slow.

Running the 32-bit OS. Fewer wheels, no NEON improvements from the 64-bit ABI, and a hard 4 GB memory ceiling. Use 64-bit.

Writing logs to the SD card in a loop. SD cards have limited write cycles. Log to /dev/shm or a USB SSD, and boot from an SSD if the Pi will run for years.

Benchmarking without a heatsink. A bare Pi 5 under sustained load will throttle within minutes. See power and thermal limits for how to measure that rather than guess at it.

Assuming your laptop's numbers transfer. They do not, and the ratio is not a constant across models either.

Try it yourself#

Run the script on your laptop and on a Pi if you can borrow one, and compute frames per second for each. Then pick a real task — counting vehicles, checking a machine, reading a meter — and write down the frame rate it truly needs. Most people discover they needed 2 frames per second and were about to buy an accelerator for 30.

What to learn next#

• Latency and throughput — measuring speed so the numbers mean something.
• Power and thermal limits — why the tenth minute is slower than the first.
• Docker for ML — packaging what you built so it survives a reboot.