Scale to zero, and what it costs you

Setup#

pip install scikit-learn pandas joblib psutil

Measuring a real cold start versus a real warm reload#

This trains a small model, then measures the honest difference. Starting a brand-new Python process and loading everything from scratch, against reusing a process that is already warmed up.

import joblib
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.RandomState(0)
n = 400
X = pd.DataFrame({
    "income": rng.uniform(5, 80, n).round(1),
    "years": rng.uniform(0, 10, n).round(1),
    "age": rng.randint(21, 65, n).astype(float),
})
score = 0.05 * X["income"] + 0.35 * X["years"] + 0.01 * X["age"] - 3.5
y = (score + rng.normal(0, 0.8, n) > 0).astype(int)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X, y)
joblib.dump(model, "model.joblib")

import subprocess
import sys
import time

import joblib
import numpy as np

# Cold: a fresh Python process, importing every library from scratch.
code = "import joblib, pandas as pd; joblib.load('model.joblib')"
t = time.perf_counter()
subprocess.run([sys.executable, "-c", code], check=True)
cold_ms = (time.perf_counter() - t) * 1000

# Warm: the interpreter and libraries are already loaded, only the file read happens.
warm_times = []
for _ in range(5):
    t = time.perf_counter()
    joblib.load("model.joblib")
    warm_times.append((time.perf_counter() - t) * 1000)
warm_ms = sum(warm_times) / len(warm_times)

print(f"cold start (new process): {cold_ms:7.1f} ms")
print(f"warm reload (file only):  {warm_ms:7.2f} ms")
print(f"cold is roughly {cold_ms / warm_ms:.0f}x slower")

# --- the cost side: a simulation with a fixed seed, not a live measurement ---
PRICE_PER_HOUR = 0.05     # a made-up small-instance price, for illustration only
REQUESTS_PER_DAY = 300
IDLE_TIMEOUT_MIN = 10      # scale-to-zero shuts down after this much silence

rng = np.random.RandomState(3)
arrivals = np.sort(rng.uniform(0, 24 * 60, REQUESTS_PER_DAY))  # minutes since midnight

# Merge each [arrival, arrival + timeout] window with the next if they overlap.
# A merged run of windows = one "instance stayed up" stretch = one cold start.
cold_starts = 1
active_minutes = 0.0
window_start, window_end = arrivals[0], arrivals[0] + IDLE_TIMEOUT_MIN
for a in arrivals[1:]:
    if a <= window_end:
        window_end = a + IDLE_TIMEOUT_MIN
    else:
        active_minutes += window_end - window_start
        window_start, window_end = a, a + IDLE_TIMEOUT_MIN
        cold_starts += 1
active_minutes += window_end - window_start

always_on_cost = PRICE_PER_HOUR * 24
scale_to_zero_cost = PRICE_PER_HOUR * (active_minutes / 60)

print(f"\nsimulated cold starts in one day: {cold_starts}")
print(f"instance-minutes billed:          {active_minutes:.0f} of 1440")
print(f"always-on cost:     ${always_on_cost:.2f} / day")
print(f"scale-to-zero cost: ${scale_to_zero_cost:.2f} / day")

cold start (new process): 1099.6 ms
warm reload (file only):   149.28 ms
cold is roughly 7x slower

simulated cold starts in one day: 35
instance-minutes billed:          1235 of 1440
always-on cost:     $1.20 / day
scale-to-zero cost: $1.03 / day

The millisecond numbers in the first block are real measurements from this machine, taken right now. Yours will differ, with your own CPU, disk, and running programs. The gap between cold and warm — several times slower — is the part that reliably reproduces.

The second block is a simulation, not a measurement. 300 request times were drawn from a fixed random seed, and the arithmetic that follows is exact given those numbers. It illustrates the mechanic, not a real bill.

Walking through it#

The subprocess trick. Running joblib.load inside a brand-new python -c "..." process is the honest way to measure a cold start on one machine. It pays the real cost of starting an interpreter and importing every library again. That is exactly what a scale-to-zero platform does when it boots a fresh instance.

Merging overlapping windows. A naive estimate — multiply request count by idle timeout — badly overcounts. A busy period is one continuous "stayed warm" stretch, not several separate ones. The merge loop is what turns 300 individual arrival times into a small number of real active-and-idle windows.

35 cold starts, not 300. Even with a fairly short 10-minute idle timeout, most requests arrived close enough together to reuse an already-warm instance. Cold starts happen at the start of each isolated burst of traffic, not on every request.

Common mistakes#

Setting the idle timeout too short. A very short timeout — say, 30 seconds — turns nearly every request into its own cold start. The gap between typical requests outlasts it entirely. Measure your actual inter-request gaps before choosing a number.

Ignoring cold starts because "the average is fine." Averages hide the worst case. The users who matter here are the ones who arrive right after a quiet stretch. It is a small fraction of requests, but a real and repeatable one.

Assuming savings scale to any traffic level. At high enough request volume, windows merge so often that the service is almost always warm. Scale-to-zero then saves little, while still risking occasional cold starts. It earns its keep at low, bursty volume — check which one you have.

Try it yourself#

Change IDLE_TIMEOUT_MIN to 2 and rerun. Watch cold_starts rise sharply as more gaps between requests exceed the shorter timeout. The same 300 requests now trigger far more boots.

What to learn next#

• Load balancing inference traffic — once more than one instance exists, warm or cold, how requests are routed to it.
• Autoscaling lag and headroom — the general version of the cold-start problem, for scaling up rather than scaling from zero.
• Model serving — the service being scaled to zero here, and the health check that tells you when it has finished booting.