Learn AI Create the Future — with Pranay
Main site

Scaling and Traffic Management

Load balancing inference traffic

A load balancer decides which machine answers each request, and a bad choice can leave one machine idle while another falls behind.

Read these first

On this page 9
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. Two simple policies, and why they are not the same
  5. How it works
  6. A real example you have seen
  7. The honest part
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

A load balancer sits in front of several machines and decides which one gets each incoming request.

The analogy you have already lived

You have walked into a bank with several counters open. A guard, or a token system, sends you to whichever counter is free. Not to counter one every single time, and not to whichever counter you personally like.

If the guard sent everyone to counter one out of habit, that line would crawl. Counters two and three would sit empty. Deciding well matters as much as having enough counters in the first place.

Why it exists

Once you have more than one machine serving your model — which autoscaling will eventually give you — something has to decide. For every single request, which machine handles it?

Get that decision wrong, and adding machines stops helping. A fleet of ten machines with nine idle and one drowning behaves like a fleet of one. That is true for anyone unlucky enough to land on the drowning one.

Two simple policies, and why they are not the same

Round robin sends request 1 to machine A, request 2 to B, request 3 to C, then back to A. It cycles through in a fixed order, ignoring what each machine is currently doing.

Least loaded sends each request to whichever machine currently has the smallest backlog. That is the one closest to being free right now.

Round robin is simpler, and works well when every machine is identical and every request costs about the same. Real fleets rarely stay that tidy. Machines differ in age and speed, and some requests — a long document, a complex prompt — cost far more than others.

How it works

   requests arrive
        |
        v
   [ load balancer ]
     /    |    \
    v     v     v
 [ A ]  [ B ]  [ C ]      <- C is an older, slower machine

 round robin:   sends every third request to C regardless
 least loaded:  notices C is falling behind, sends it fewer requests

A real example you have seen

A large website behind a content delivery network never sends you to a server by fixed rotation. It routes you to whichever nearby server currently has capacity. It is the same idea underneath a familiar experience. A page loads quickly no matter how many other people are browsing at that moment.

The honest part

Least loaded sounds like it should always win, and in the numbers below it does. But it needs the load balancer to know each machine's current backlog. That is one more thing that has to be measured accurately, and kept up to date. A stale or wrong measurement can make least-loaded routing worse than simple round robin, not better.

Remember this

  • A load balancer's policy — how it picks a machine — matters as much as how many machines exist.
  • Round robin ignores current load; least loaded reacts to it.
  • Real fleets are rarely identical machines with identical requests, which is exactly when the policy choice starts to show.

What to learn next

Developer — Code and libraries.

Setup

No installs needed beyond the standard library.

Comparing the two policies on an uneven fleet

Three servers, one of them 1.6x slower than the others — a realistic stand-in for an older machine still in the fleet. 200 requests arrive over 20 seconds with randomised service times.

load_balancing.py
import random

random.seed(11)

NUM_SERVERS = 3
NUM_REQUESTS = 200

# One server is a slow, older machine. Real fleets are rarely identical.
SPEED = [1.0, 1.0, 1.6]  # server 2 takes 1.6x as long per request

arrivals = sorted(random.uniform(0, 20) for _ in range(NUM_REQUESTS))
service_base = [random.uniform(0.05, 0.15) for _ in range(NUM_REQUESTS)]


def simulate(policy):
    free_at = [0.0] * NUM_SERVERS
    waits = []
    rr_counter = 0
    for i, arrival in enumerate(arrivals):
        service = service_base[i]
        if policy == "round_robin":
            server = rr_counter % NUM_SERVERS
            rr_counter += 1
        elif policy == "least_loaded":
            # picks the server that will be free soonest — the real signal
            # a load balancer can measure: current backlog, not raw identity.
            server = min(range(NUM_SERVERS), key=lambda s: free_at[s])
        start = max(arrival, free_at[server])
        waits.append(start - arrival)
        free_at[server] = start + service * SPEED[server]
    return waits


for policy in ["round_robin", "least_loaded"]:
    waits = simulate(policy)
    waits_sorted = sorted(waits)
    avg_wait = sum(waits) / len(waits)
    p95_wait = waits_sorted[int(0.95 * len(waits_sorted))]
    print(f"{policy:12s} avg wait {avg_wait*1000:6.1f} ms   p95 wait {p95_wait*1000:6.1f} ms")
Output
round_robin  avg wait    5.5 ms   p95 wait   49.9 ms
least_loaded avg wait    1.3 ms   p95 wait    9.3 ms

This is a deterministic simulation with a fixed random seed — these exact numbers reproduce on any machine, every time you run it.

Walking through it

free_at[s] tracks the simulated moment each server becomes free. It is the load balancer's best available signal for "how backed up is this machine right now." Real load balancers estimate the same thing from active connection counts or reported queue depth.

Round robin sends a third of all traffic to the slow server, purely because it is that server's numerical turn. It has no way to notice that server keeps falling further behind.

Least loaded routes around the slow server automatically. It never needed to be told server 2 is slower. It only needed to see server 2's backlog growing, and it sent fewer requests that way as a direct consequence.

p95 wait — the time below which 95% of requests finish waiting — tells a sharper story than the average. Round robin's p95 is over 5x worse than its average. The requests unlucky enough to land on the slow server, right after several others already have, wait far longer than a typical request.

Common mistakes

Sticking with round robin after a fleet stops being identical. It is a fine default for a uniform fleet. The moment machines differ in speed, or requests differ in cost, it silently starts wasting the faster machines.

Measuring load with a stale number. A load balancer that checks each server's queue depth every 30 seconds is routing on 30-second-old information. During a fast-moving burst, that delay is long enough. A wave of new requests can land on a server that already fell behind.

Ignoring the load balancer itself as a bottleneck. The component making this decision has to run the decision logic for every single request. At high enough traffic, an expensive routing algorithm can make the load balancer the slowest part of the whole pipeline.

Try it yourself

Set SPEED = [1.0, 1.0, 1.0] — an identical fleet — and rerun. Watch how close round robin and least loaded become. With no speed difference to route around, there is nothing for least-loaded's extra information to improve on.

What to learn next

Researcher — Mathematics and papers.

Beyond round robin and least connections: power of two choices

Picking the single best of $N$ servers requires knowing every server's load — expensive to keep current at scale. Mitzenmacher's The Power of Two Choices in Randomized Load Balancing (1996, published 2001 in IEEE TPDS) shows that sampling only two servers at random and picking the less-loaded of the two captures most of the benefit of checking all $N$, at a fraction of the coordination cost. This result underlies load balancing designs across distributed systems generally, well beyond ML serving specifically.

Weighted and adaptive variants

Weighted round robin assigns each server a weight proportional to its known capacity, sending it a proportional share of requests without needing live load data — a static compromise between round robin's simplicity and least-loaded's responsiveness.

Consistent hashing routes by a hash of some request property — a user ID, a session ID — onto a ring of servers, so the same key reliably lands on the same server without a central load table. This becomes directly relevant for prefix-aware and sticky routing, where landing on the same server repeatedly is the entire point rather than a side effect.

Cost of the decision itself

At $N$ requests per second and $O(k)$ cost per routing decision — sampling $k$ servers, as in power-of-two-choices — the load balancer's own overhead scales as $O(Nk)$. For very high request rates, this motivates hardware load balancers (L4, operating on TCP/IP without inspecting request content) over software L7 balancers that can inspect and route on request content, trading routing intelligence for raw throughput.

Papers

  • Mitzenmacher, The Power of Two Choices in Randomized Load Balancing, IEEE Transactions on Parallel and Distributed Systems, 2001 (result dates to 1996) — www.eecs.harvard.edu/~michaelm/NEWWORK/postscripts/tpds.pdf
  • Karger et al., Consistent Hashing and Random Trees, STOC 1997 — the foundational paper behind consistent-hash load balancing and routing rings.

What to learn next