OpenVINO and CPU inference

Setup#

pip install openvino          # tested with 2026.3.1
pip install onnx onnxruntime torch    # to produce and cross-check the model

OpenVINO reads ONNX directly, so anything you exported for TensorRT works here unchanged.

Step one: make a model and a reference answer#

Run this in an environment with PyTorch and ONNX Runtime. It reuses the export from INT8 calibration, which produced shapes_prep.onnx.

import numpy as np, onnxruntime as ort

rng = np.random.default_rng(0)
x = rng.random((32, 1, 32, 32)).astype(np.float32)
np.save("ov_x.npy", x)

s = ort.InferenceSession("shapes_prep.onnx", providers=["CPUExecutionProvider"])
np.save("ov_ref.npy", s.run(None, {"input": x})[0])
print("saved", x.shape)

saved (32, 1, 32, 32)

Step two: load, verify, and measure#

import time, statistics
import numpy as np
import openvino as ov

core = ov.Core()
print("OpenVINO", ov.__version__)
print("devices :", core.available_devices)
print("CPU     :", core.get_property("CPU", "FULL_DEVICE_NAME"))

# OpenVINO reads ONNX directly. convert_model turns it into OpenVINO IR,
# which loads faster and can be saved next to your service.
model = core.read_model("shapes_prep.onnx")
print("\ninputs :", [(i.any_name, i.partial_shape) for i in model.inputs])
print("outputs:", [(o.any_name, o.partial_shape) for o in model.outputs])

ov.save_model(model, "shapes_ov.xml")          # writes shapes_ov.xml + .bin

x = np.load("ov_x.npy")
ref = np.load("ov_ref.npy")                    # what ONNX Runtime produced

compiled = core.compile_model(model, "CPU")
out = compiled(x)[compiled.output(0)]
print(f"\noutput shape {out.shape}, "
      f"largest difference from ONNX Runtime: {np.abs(out - ref).max():.2e}")


def bench(cm, batch, n=50):
    req = cm.create_infer_request()
    xb = x[:batch]
    for _ in range(10):
        req.infer({0: xb})                     # warm up before timing anything
    ts = []
    for _ in range(n):
        t0 = time.perf_counter()
        req.infer({0: xb})
        ts.append((time.perf_counter() - t0) * 1000)
    return statistics.median(ts)


print("\nlatency by batch size, PERFORMANCE_HINT LATENCY:")
lat = core.compile_model(model, "CPU", {"PERFORMANCE_HINT": "LATENCY"})
for b in (1, 4, 16, 32):
    ms = bench(lat, b)
    print(f"  batch {b:>2}: {ms:7.3f} ms  ({b*1000/ms:8.0f} images/second)")

thr = core.compile_model(model, "CPU", {"PERFORMANCE_HINT": "THROUGHPUT"})
print("\nsame model compiled with PERFORMANCE_HINT THROUGHPUT:")
print("  optimal number of infer requests:",
      thr.get_property("OPTIMAL_NUMBER_OF_INFER_REQUESTS"))
for b in (1, 32):
    ms = bench(thr, b)
    print(f"  batch {b:>2}: {ms:7.3f} ms  ({b*1000/ms:8.0f} images/second)")

OpenVINO 2026.3.1-22476-759c5a6ab8c-releases/2026/3
devices : ['CPU', 'GPU.0', 'GPU.1']
CPU     : 13th Gen Intel(R) Core(TM) i9-13900K

inputs : [('input', <PartialShape: [?,1,32,32]>)]
outputs: [('logits', <PartialShape: [?,3]>)]

output shape (32, 3), largest difference from ONNX Runtime: 6.68e-06

latency by batch size, PERFORMANCE_HINT LATENCY:
  batch  1:   0.098 ms  (   10204 images/second)
  batch  4:   0.245 ms  (   16320 images/second)
  batch 16:   0.227 ms  (   70361 images/second)
  batch 32:   0.333 ms  (   96082 images/second)

same model compiled with PERFORMANCE_HINT THROUGHPUT:
  optimal number of infer requests: 24
  batch  1:   0.183 ms  (    5461 images/second)
  batch 32:   0.690 ms  (   46357 images/second)

Timings are hardware-dependent and move between runs. These come from one run on an Intel Core i9-13900K. Across five runs the millisecond figures varied by a factor of two or more with background load, while every comparison drawn below held in all of them. Treat the comparisons as the finding and the milliseconds as an illustration.

Reading that output#

The answer matches ONNX Runtime to 6.68e-06. That is float32 accumulation-order noise, not a correctness problem. It is also the check to run first, every time: a fast wrong answer is not an optimisation.

Batching gives an enormous throughput win for a tiny latency cost. Batch 1 takes 0.098 ms; batch 32 takes 0.333 ms. Thirty-two times the work for under four times the time — 10,204 images per second becomes 96,082. A single 32x32 image cannot fill the vector units or hide memory latency, so most of that 0.098 ms is fixed overhead that batching amortises.

OPTIMAL_NUMBER_OF_INFER_REQUESTS is 24. Under the throughput hint, OpenVINO partitioned this 24-core CPU into 24 independent streams. That number is the one you should size your request queue against; running fewer leaves cores idle, running more adds queueing delay without adding work.

The throughput hint is not automatically faster. Driven by one request at a time it is slower at both batch sizes: 0.690 ms against 0.333 ms at batch 32, and 46,357 images per second against 96,082. Each stream now gets a fraction of the machine, so a single batch runs on a fraction of the cores. The throughput hint pays off when you have many concurrent requests to keep those 24 streams busy, and costs you when you have one.

This is the most common OpenVINO mistake: setting THROUGHPUT and benchmarking with one request at a time. You measure the cost and none of the benefit.

Devices list GPU.0 and GPU.1. OpenVINO enumerates Intel integrated and discrete graphics. It does not target NVIDIA cards; for those, see the TensorRT lesson.

The command-line benchmark#

benchmark_app ships with the package and is the tool to reach for before writing any timing code of your own.

benchmark_app -m shapes_ov.xml -d CPU -hint latency -shape "[1,1,32,32]" -t 5

[Step 10/11] Measuring performance (Start inference synchronously, limits: 5000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 0.35 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count:            49081 iterations
[ INFO ] Duration:         5000.86 ms
[ INFO ] Latency:
[ INFO ]    Median:        0.06 ms
[ INFO ]    Average:       0.08 ms
[ INFO ]    Min:           0.03 ms
[ INFO ]    Max:           16.92 ms
[ INFO ] Throughput:   9814.51 FPS

Note the gap between the median of 0.06 ms and the maximum of 16.92 ms. Across repeated runs on this machine that ratio sat between roughly 75 and 280 times, and it never went away. The tail is the operating system scheduling other work, and it is what a user with a hard deadline actually experiences. Any latency claim quoting only a mean or a median is hiding it.

The median here is close to the batch-1 figure measured from Python, and generally lands a little below it, because benchmark_app excludes the Python call overhead the earlier script includes. Both are correct answers to different questions; the Python one is closer to what your service will see.

Preprocessing inside the model#

OpenVINO can fold resizing, layout conversion and normalisation into the compiled model, which removes an entire class of preprocessing mismatch and runs faster than doing it in NumPy.

from openvino.preprocess import PrePostProcessor, ColorFormat
from openvino import Type, Layout

ppp = PrePostProcessor(model)
ppp.input().tensor() \
    .set_element_type(Type.u8) \
    .set_layout(Layout("NHWC")) \
    .set_color_format(ColorFormat.BGR)
ppp.input().preprocess() \
    .convert_element_type(Type.f32) \
    .convert_color(ColorFormat.RGB) \
    .mean([123.675, 116.28, 103.53]) \
    .scale([58.395, 57.12, 57.375])
ppp.input().model().set_layout(Layout("NCHW"))
model = ppp.build()

No output block for this fragment: it is a graph transformation whose effect shows up in the compiled model, and the shapes here would need adjusting for the single-channel toy model above. On a real three-channel model it lets you feed raw uint8 BGR straight from cv2.imread with no Python preprocessing at all.

Common mistakes#

Benchmarking without warming up. The first inference compiles kernels and allocates buffers. The script above discards ten iterations for this reason.

Setting THROUGHPUT and sending one request at a time. Demonstrated above: it is slower. Match the hint to your traffic shape.

Recompiling per request. compile_model is expensive and create_infer_request is cheap. Compile once at startup, keep a pool of requests.

Assuming int8 helps everywhere. It helps a great deal on CPUs with VNNI or AMX instructions and much less on older ones. Quantise with NNCF (nncf.quantize() — create_compressed_model() is deprecated), then measure on the deployment CPU.

Using the deprecated openvino.runtime namespace. It was marked for removal in 2026.0. Import openvino directly; every example above does.

Comparing CPU and GPU numbers without including preprocessing. As the GPU preprocessing lesson measures, decode and resize frequently dominate. A CPU pipeline that avoids a host-to-device transfer can beat a GPU pipeline end to end even when the raw model is slower.

Try it yourself#

Compile the same model with {"PERFORMANCE_HINT": "THROUGHPUT"} and drive it with 24 concurrent InferRequest objects using start_async and wait, instead of one synchronous request. That is the configuration the throughput hint was designed for, and it is the only way to see it win.

What to learn next#

• ONNX — the format this whole workflow starts from.
• Latency and throughput — the trade-off the performance hints expose.
• Model serving — putting a compiled model behind an API.