Tuning CPU inference

Setup#

pip install numpy psutil threadpoolctl

Reproducing the oversubscription, on a machine pinned to 2 CPUs#

Most inference containers get 2–4 vCPUs, not a workstation's full core count. This script pins itself to 2 CPUs with psutil so the demo reflects that reality, rather than the sandbox machine's actual core count.

import time
from concurrent.futures import ThreadPoolExecutor

import numpy as np
import psutil
from threadpoolctl import threadpool_limits

psutil.Process().cpu_affinity([0, 1])   # pin this process to 2 CPUs

SIZE = 300
N_CALLS = 80
A = np.random.rand(SIZE, SIZE)
B = np.random.rand(SIZE, SIZE)


def one_matmul(_):
    return (A @ B).sum()


def run_concurrent(n_python_workers, blas_threads):
    with threadpool_limits(limits=blas_threads, user_api="blas"):
        start = time.perf_counter()
        with ThreadPoolExecutor(max_workers=n_python_workers) as pool:
            list(pool.map(one_matmul, range(N_CALLS)))
        return time.perf_counter() - start


if __name__ == "__main__":
    t_default = run_concurrent(n_python_workers=8, blas_threads=None)
    print(f"8 workers, BLAS default (unrestricted) threads, on 2 CPUs: {t_default:6.2f}s")

    t_tuned = run_concurrent(n_python_workers=8, blas_threads=1)
    print(f"8 workers, BLAS limited to 1 thread,             on 2 CPUs: {t_tuned:6.2f}s")

    t_matched = run_concurrent(n_python_workers=2, blas_threads=1)
    print(f"2 workers, BLAS limited to 1 thread,             on 2 CPUs: {t_matched:6.2f}s")

8 workers, BLAS default (unrestricted) threads, on 2 CPUs:   0.28s
8 workers, BLAS limited to 1 thread,             on 2 CPUs:   0.16s
2 workers, BLAS limited to 1 thread,             on 2 CPUs:   0.16s

Real measurements from this machine, pinned to 2 CPUs for this run. Absolute times will differ on your hardware, but the pattern is worth trusting: leaving BLAS unrestricted while also running 8 concurrent workers was 75% slower than restricting BLAS to one thread per call. Matching worker count to CPU count (2 workers, 2 CPUs) performed as well as 8 tuned workers — extra workers past your core count bought nothing here.

Line-by-line walkthrough#

psutil.Process().cpu_affinity([0, 1]) restricts this entire process to CPUs 0 and 1, simulating a small container — the realistic environment where this problem actually bites.

threadpool_limits(limits=blas_threads, user_api="blas") is the tuning knob itself: it caps how many threads the BLAS library underneath NumPy is allowed to use for the duration of the block, without needing an environment variable set before the process starts.

ThreadPoolExecutor(max_workers=n_python_workers) provides Python-level concurrency across separate matmul calls, competing for the same 2 CPUs as BLAS's own internal threads unless BLAS is told to stay small.

Common mistakes#

Never checking what BLAS backend NumPy is actually using. threadpoolctl.threadpool_info() reports it directly — OpenBLAS and MKL both default to using many threads, and assuming otherwise is a common source of surprise.

Setting thread counts with environment variables after the process starts. OMP_NUM_THREADS and OPENBLAS_NUM_THREADS are read once, at library load time. Setting them inside a running Python process usually has no effect — set them before the process starts, or use threadpool_limits as shown above.

Assuming more workers is always safer. Past your CPU count, extra workers do not add capacity — they add contention. The 2-worker and 8-worker tuned runs above performed identically, because 2 CPUs cannot usefully run more than 2 things at once.

Forgetting this changes under a container CPU limit. A Kubernetes cpu: "2" limit throttles a process to roughly 2 cores' worth of CPU time, even on a much bigger physical node. NumPy sees the node's core count when it picks a thread count by default, not the container's limit — the exact mismatch this lesson is about.

Try it yourself#

Remove the psutil.Process().cpu_affinity([0, 1]) line and rerun on your own machine. If your machine has many cores, the effect should shrink or disappear — this problem is specifically about scarce cores, not slow ones.

What to learn next#

• Concurrency in a Python inference server — the Python-level side of the same "who provides the parallelism" question.
• Profiling inference code — confirming CPU contention is the real bottleneck before tuning around it.
• Capacity planning with Little's Law — sizing worker counts against real, measured capacity rather than guesswork.