Graceful shutdown and connection draining

Setup#

No installs needed beyond the standard library.

Watching a real drain happen, in real time#

This runs actual threads with actual timing, not a simulation. Draining is fundamentally about real concurrent behaviour — what is genuinely in-flight at the moment the shutdown signal lands.

import threading
import time

SERVICE_TIME = 0.3   # how long one request takes to handle
DRAIN_AT = 1.0        # shutdown signal arrives at this simulated moment
NUM_REQUESTS = 30
GAP = 0.05             # requests arrive 50ms apart

draining = False
in_flight = 0
lock = threading.Lock()
completed = []
rejected = []


def handle_request(req_id, arrive_at):
    global in_flight
    time.sleep(max(0, arrive_at - (time.perf_counter() - start_time)))
    with lock:
        if draining:
            rejected.append(req_id)
            return
        in_flight += 1
    time.sleep(SERVICE_TIME)  # doing the actual work
    with lock:
        in_flight -= 1
        completed.append(req_id)


def begin_drain():
    global draining
    time.sleep(DRAIN_AT)
    draining = True
    print(f"[drain] shutdown signal received, {in_flight} request(s) already in flight")
    while True:  # a real server waits here until in_flight reaches 0 before exiting
        with lock:
            if in_flight == 0:
                break
        time.sleep(0.01)
    print("[drain] all in-flight requests finished, safe to exit")


start_time = time.perf_counter()
drain_thread = threading.Thread(target=begin_drain)
drain_thread.start()

workers = [threading.Thread(target=handle_request, args=(i, i * GAP)) for i in range(NUM_REQUESTS)]
for w in workers:
    w.start()
for w in workers:
    w.join()
drain_thread.join()

print(f"\ncompleted: {len(completed)}")
print(f"rejected after drain started: {len(rejected)}")
print(f"total: {len(completed) + len(rejected)} of {NUM_REQUESTS}")

[drain] shutdown signal received, 7 request(s) already in flight
[drain] all in-flight requests finished, safe to exit

completed: 21
rejected after drain started: 9
total: 30 of 30

This uses real threads and real wall-clock timing, so the exact counts can shift by one request on your machine. A request landing right at the t=1.0s boundary can go either way, depending on thread scheduling. Running this three times on the development machine produced 21/9, 21/9, and 20/10 completed/rejected. The pattern is the real point: some requests finish, later ones are rejected, none are silently dropped mid-request. It held every time.

Walking through it#

The draining check happens before in_flight is incremented. A request that arrives after the signal never gets counted as in-flight at all. It is rejected cleanly, before any work starts on it.

begin_drain blocks on in_flight == 0 before printing that it is safe to exit. This is the actual draining step: the shutdown does not proceed until every request that was already running has genuinely finished.

None of the 30 requests are unaccounted for. completed + rejected always equals 30 exactly — every request either finished properly or was rejected cleanly, and nothing was abandoned mid-flight.

Common mistakes#

No maximum wait on the drain. A single stuck request — waiting on a hung downstream call — can block a shutdown forever without one. Pair draining with a hard timeout, after which the process exits regardless.

The load balancer does not know about the drain. If traffic keeps arriving from outside after this server marks itself as draining, every one of those requests gets rejected pointlessly. The load balancer needs to stop routing here before the drain begins, usually via a health check this server starts failing on purpose.

Confusing SIGKILL with SIGTERM. On Linux, SIGTERM is the polite "please shut down" signal a drain step listens for. SIGKILL cannot be caught or handled at all — it terminates immediately. A deployment tool that gives too short a grace period before escalating to SIGKILL defeats the whole point of draining.

Testing shutdown only in a clean, idle state. The behaviour that matters is what happens under load, mid-request. Test by sending traffic and shutting down while it is still arriving, not by shutting down a server that was already quiet.

Try it yourself#

Change DRAIN_AT to 0.1 — a shutdown signal that arrives almost immediately. Watch rejected climb close to 30, because almost no requests have had a chance to start before the drain begins.

What to learn next#

• Circuit breakers and fallbacks — handling a dependency that stops answering mid-request, the reverse side of this lesson's problem.
• Docker for ML — the container this shutdown sequence runs inside, and the signals it receives.
• Multi-tenancy and noisy neighbours — another case where "in flight right now" is exactly the state that matters most.