Learn AI Create the Future — with Pranay
Main site

Batching and Concurrency

Admission control and load shedding

Admission control rejects requests fast once a server is overloaded, instead of accepting everything and making every single request wait longer and longer.

Read these first

On this page 8
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. A real example you have seen
  6. The honest part
  7. Remember this
  8. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Admission control means refusing new requests once you are full, so accepted requests still get served properly.

The analogy you have already lived

You have gone to a small, popular restaurant on a busy night. A good one stops seating new tables once the kitchen is at capacity. A host tells you the wait, or turns you away, right at the door. A bad one seats every table that walks in, and now everyone's food takes two hours, including the people who arrived first.

Turning people away at the door feels harsh. It is the only way the tables already seated get fed on time.

Why it exists

A server has a limit — call it its capacity — the number of requests per second it can genuinely finish. When requests arrive faster than that, they cannot vanish. They pile up in a queue, a waiting line inside the server, holding requests until a worker is free.

An unlimited queue seems generous. It accepts everything. But every request added to a long queue waits behind everyone already in it. The queue keeps growing, and the wait grows with it, for as long as the overload lasts. Eventually even a genuinely urgent request is stuck behind a huge backlog.

Admission control puts a limit on the queue itself. Once it is full, a new request gets an instant, cheap answer: "try again shortly." That beats a slow, expensive wait that ends the same way anyway.

How it works

Without admission control (queue has no limit):
  requests keep arriving and queueing, with no limit...
  queue: [r1][r2][r3][r4][r5][r6][r7][r8][r9][r10]...[r400]
  r400 waits behind 399 others -- however long that takes

With admission control (queue capped at, say, 15):
  queue: [r1][r2]...[r15]   <- these 15 are served properly
  r16 onward: instantly told "no room, try later" -- costs almost nothing

A real example you have seen

A ticket-booking site during a big sale, or IRCTC Tatkal booking, often shows "please wait, high traffic" and a queue position. Nobody gets let straight in. That message is a form of admission control — an honest, fast "not yet" instead of a silent, slow maybe.

The honest part

Rejecting a real user's request never feels good, and it is tempting to make the queue bigger instead. That only postpones the problem. A bigger queue still fills up under sustained overload, and now it fills up slower, hiding the warning signs until things are worse.

Remember this

  • An overloaded server without limits does not fail loudly — it gets slow for everyone, including requests that were already accepted.
  • Admission control rejects new requests fast once the queue is full, protecting the requests already being served.
  • A fast, honest rejection is cheaper for everyone than a slow, silent wait that fails anyway.

What to learn next

Developer — Code and libraries.

Setup

No installation needed — this uses Python's built-in queue and threading modules only.

A bounded queue versus an unbounded one, under real overload

admission_control.py
import queue
import threading
import time

SERVICE_TIME = 0.006   # one worker finishes one request every 6ms (~166 req/s capacity)
OFFERED_RATE = 300      # requests per second arriving -- well above capacity
DURATION = 1.5           # seconds of overload
QUEUE_CAP = 15            # bounded queue: at most this many requests waiting


def worker(inbox, results, stop_flag):
    while True:
        try:
            submitted_at = inbox.get(timeout=0.5)
        except queue.Empty:
            if stop_flag.is_set():
                return
            continue
        time.sleep(SERVICE_TIME)
        results.append(time.perf_counter() - submitted_at)


def offer_load(put_fn, duration, rate):
    gap = 1.0 / rate
    accepted, rejected = 0, 0
    end = time.perf_counter() + duration
    while time.perf_counter() < end:
        if put_fn():
            accepted += 1
        else:
            rejected += 1
        time.sleep(gap)
    return accepted, rejected


def percentile(values, p):
    s = sorted(values)
    return s[min(int(len(s) * p), len(s) - 1)]


def run(bounded: bool):
    inbox = queue.Queue(maxsize=QUEUE_CAP if bounded else 0)
    results = []
    stop_flag = threading.Event()
    t = threading.Thread(target=worker, args=(inbox, results, stop_flag))
    t.start()

    def put_fn():
        try:
            inbox.put_nowait(time.perf_counter())
            return True
        except queue.Full:
            return False

    accepted, rejected = offer_load(put_fn, DURATION, OFFERED_RATE)
    time.sleep(2.0)   # let the worker drain what it already accepted
    stop_flag.set()
    t.join()
    return accepted, rejected, results


if __name__ == "__main__":
    for label, bounded in [("unbounded queue", False), ("bounded queue (admission control)", True)]:
        accepted, rejected, latencies = run(bounded)
        p50 = percentile(latencies, 0.50) * 1000
        p99 = percentile(latencies, 0.99) * 1000
        worst = max(latencies) * 1000
        print(f"{label:34s} accepted={accepted:4d} rejected={rejected:4d} "
              f"p50={p50:7.1f}ms p99={p99:7.1f}ms worst={worst:7.1f}ms")
Output
unbounded queue                    accepted= 325 rejected=   0 p50=  331.4ms p99=  656.6ms worst=  662.5ms
bounded queue (admission control)  accepted= 240 rejected=  85 p50=  103.5ms p99=  108.4ms worst=  108.9ms

These are real timings from this run — the exact accept/reject counts and latencies will shift a little between runs, since thread scheduling is not perfectly deterministic. The pattern is the real finding: the unbounded queue accepted every request, but its p99 latency was six times worse. The bounded queue rejected 85 requests outright, and every request it did accept came back in about a tenth of a second, consistently.

Line-by-line walkthrough

queue.Queue(maxsize=QUEUE_CAP) is the entire admission-control mechanism. A capped queue, plus put_nowait, which raises queue.Full instantly instead of blocking — that exception is the "no room" answer.

offer_load fires requests at a fixed rate, regardless of whether the server is keeping up. That is what real traffic does — real users do not slow down because your server is struggling.

The unbounded run (maxsize=0) accepts everything, so its queue keeps growing for the whole 1.5-second burst, and the last request accepted waits behind the full backlog that built up before it.

Common mistakes

Rejecting with an expensive error. A rejection should be nearly free — check a counter, return immediately. Building a full error response, logging a stack trace, or touching the model at all defeats the purpose.

Choosing the queue cap arbitrarily. Pick it from your latency budget instead. If the worker takes 6ms per request and your budget is 100ms, a queue of about 15 keeps the worst case inside budget — 15 * 6ms ≈ 90ms. Work backward from the number that matters.

Applying admission control but not telling the caller why. A 503 Service Unavailable with a Retry-After header lets a well-behaved caller wait a sensible amount before trying again, rather than guessing.

Confusing this with rate limiting. Rate limiting caps how much one specific user or API key can send. Admission control caps how much the server will hold in total, from everyone combined. Production systems typically need both.

Try it yourself

Change QUEUE_CAP to 100 and rerun. The "bounded" run's p99 should climb back toward the unbounded run's number. A large cap gives you almost none of the protection, and almost none of the honesty.

What to learn next

Researcher — Mathematics and papers.

Why unbounded queues fail predictably

For a single-server queue with arrival rate $\lambda$ and service rate $\mu$, utilisation is $\rho = \lambda/\mu$. Once $\rho \geq 1$ — arrivals at or above capacity — an unbounded queue's expected length grows without bound over time; it never reaches a steady state. This is not a tuning problem, it is a mathematical certainty for as long as the overload persists. See capacity planning and Little's Law for the full relationship between $\rho$, queue length, and wait time.

Admission control converts an unstable, unbounded-delay system into a stable, bounded-delay one, at the cost of a non-zero rejection rate during overload. This is the standard load shedding trade-off: bounded latency for accepted work, in exchange for openly dropping some.

Shedding strategies beyond a simple cap

  • Priority shedding — reject low-priority traffic first (batch jobs, background refreshes) while protecting user-facing requests, using a priority field rather than pure FIFO capacity.
  • Cost-aware shedding — reject the most expensive requests first under pressure (e.g. requests asking for a very long generation), since they consume disproportionate capacity for one "slot."
  • Adaptive concurrency limits — rather than a fixed queue cap, algorithms like Netflix's concurrency-limits (built on TCP-style additive-increase/multiplicative-decrease control) adjust the accepted concurrency dynamically based on observed latency, tightening automatically when latency rises.
  • CoDel and other AQM disciplines — Active Queue Management algorithms from networking (Nichols and Jacobson, 2012) drop or reject based on how long an item has already waited, rather than queue length alone, directly bounding worst-case delay rather than queue size as a proxy for it.

Relationship to circuit breakers

Admission control protects this service from being overwhelmed by its own inbound traffic. A circuit breaker protects a service from a downstream dependency that has become slow or is failing, by refusing to call it for a cooldown period. They solve adjacent but distinct problems and are commonly deployed together.

References

  • Nichols, K. and Jacobson, V., Controlling Queue Delay, ACM Queue, 2012 — the CoDel paper, on bounding delay directly rather than queue length.
  • Fowler, M., CircuitBreaker, 2014 — the widely cited pattern description, adjacent to but distinct from admission control.
  • Google SRE Book, Chapter 21, Handling Overload — the industry-standard practical treatment of load shedding at Google's scale.

What to learn next