Deadlines, timeouts and retry storms

Setup#

No installation needed — this uses Python's built-in queue and threading modules only.

Naive retries versus deadline-plus-backoff, against the same overloaded server#

import queue
import random
import threading
import time

SERVICE_TIME = 0.01     # one worker finishes one attempt every 10ms (~100 req/s capacity)
CLIENT_TIMEOUT = 0.03   # a client gives up waiting for a reply after this long
OVERALL_DEADLINE = 2.0  # a client stops retrying altogether after this long
N_LOGICAL = 90           # 90 real user requests need an answer


def worker(inbox, stop_flag, attempts_done):
    while True:
        try:
            job_id = inbox.get(timeout=0.5)
        except queue.Empty:
            if stop_flag.is_set():
                return
            continue
        time.sleep(SERVICE_TIME)   # the server does not know this attempt may be abandoned
        attempts_done.append(job_id)


def client(logical_id, inbox, done_flags, attempts_submitted, backoff: bool):
    start = time.perf_counter()
    attempt = 0
    while time.perf_counter() - start < OVERALL_DEADLINE:
        inbox.put(logical_id)
        attempts_submitted.append(logical_id)
        deadline = time.perf_counter() + CLIENT_TIMEOUT
        while time.perf_counter() < deadline:
            if done_flags.get(logical_id):
                return
            time.sleep(0.001)
        if done_flags.get(logical_id):
            return
        attempt += 1
        if backoff:
            sleep_for = min(0.5, 0.01 * (2 ** attempt)) + random.uniform(0, 0.01)
            time.sleep(sleep_for)
        # naive mode: no sleep at all -- retry immediately


def run(backoff: bool):
    inbox = queue.Queue()
    stop_flag = threading.Event()
    attempts_done = []
    attempts_submitted = []
    done_flags = {}

    def watcher():
        seen = set()
        while not stop_flag.is_set():
            for job_id in attempts_done:
                if job_id not in seen:
                    seen.add(job_id)
                    done_flags[job_id] = True
            time.sleep(0.001)

    w = threading.Thread(target=worker, args=(inbox, stop_flag, attempts_done))
    v = threading.Thread(target=watcher, daemon=True)
    w.start()
    v.start()

    start = time.perf_counter()
    clients = [threading.Thread(target=client, args=(i, inbox, done_flags, attempts_submitted, backoff))
               for i in range(N_LOGICAL)]
    for c in clients:
        c.start()
    for c in clients:
        c.join()
    wall = time.perf_counter() - start

    stop_flag.set()
    w.join()
    return wall, len(done_flags), len(attempts_submitted)


if __name__ == "__main__":
    random.seed(1)
    for label, backoff in [("naive immediate retry", False), ("deadline + backoff + jitter", True)]:
        wall, succeeded, total_attempts = run(backoff)
        print(f"{label:28s} wall={wall:5.2f}s  succeeded={succeeded:3d}/{N_LOGICAL}  "
              f"attempts sent to server={total_attempts:4d}")

naive immediate retry        wall= 0.96s  succeeded= 90/90  attempts sent to server=1444
deadline + backoff + jitter  wall= 1.36s  succeeded= 90/90  attempts sent to server= 454

Both real measurements from this run — with the random seed fixed, the attempt counts are exactly reproducible, though wall-clock time will shift slightly with thread scheduling on your machine. Read this carefully — it is not the tidy result you might expect. Naive retry finished faster here (0.96s vs 1.36s). But it sent more than three times as many attempts to the server (1444 vs 454), for the same 90 successful requests.

The nuance that matters#

Naive retry "wins" from a single client's point of view in this small test, because the server here is only mildly overloaded and recovers on its own. In a real system, all those extra 1444 attempts are competing for the same limited server capacity as every other client's requests — including ones not shown in this simulation. Multiply this pattern by thousands of real users. The server-side load from retries alone can then exceed the load from genuine first-time requests — the actual mechanism behind a retry storm taking a struggling service fully down.

Line-by-line walkthrough#

worker never learns that a client gave up — it processes every attempt it receives, including ones the client has already abandoned and retried. This is realistic: most real servers have no idea a caller stopped listening.

client retries in a loop bounded by OVERALL_DEADLINE — the deadline — and each individual wait is bounded by CLIENT_TIMEOUT. The if backoff: branch is the entire difference between the two modes: a growing sleep with random.uniform(0, 0.01) jitter mixed in, versus nothing at all.

Common mistakes#

Retrying with no deadline. A loop that retries forever, on a permanently broken dependency, never gives up — it keeps adding load forever. Always bound the total effort, not only each individual attempt.

Retrying on every kind of failure. Retrying a 422 Bad Request — a client sent malformed data — 400 more times changes nothing except the noise. Retry on failures that might genuinely be temporary: timeouts, 503s, connection resets.

No jitter. Backoff without jitter still lets every client's n-th retry land at almost the same moment, if they all started retrying together. That is common, since they often all failed for the same shared reason.

Try it yourself#

Change SERVICE_TIME from 0.01 to 0.02, making the server twice as slow, and rerun both modes. Watch how much more dramatically the naive mode's attempt count grows compared to the backoff mode's — a slower server makes the retry-storm effect worse, not better.

What to learn next#

• Admission control and load shedding — the server-side complement: refusing new load instead of hoping retries behave.
• Coordinated omission — a closely related trap in how load-testing tools themselves measure retried or stalled requests.
• SLOs and error budgets for model services — deciding, in advance, how much retry traffic your service can actually absorb.