Learn AI Create the Future — with Pranay
Main site

Scaling and Traffic Management

Rate limiting and per-user quotas

Rate limiting caps how fast one caller can send requests, so a runaway script cannot use up capacity meant for everyone else.

Read these first

On this page 9
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. The token bucket, the classic answer
  5. How it works
  6. A real example you have seen
  7. The honest part
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Rate limiting caps how many requests one caller can send in a given time. It rejects the rest, until the caller slows down.

The analogy you have already lived

You have used a water tank with a narrow tap. However fast someone turns the handle, only so much water comes out per second. The tap itself limits the flow, protecting the thin pipe on the other side from bursting.

Without that narrow tap, one person filling a bucket fast could starve the flow to everyone else on the same pipe. The tap is not there to be unfriendly. It is there so the pipe survives, and so water reaches every tap fairly.

Why it exists

A model server has a limited amount it can do per second. One caller might be a buggy script stuck in a retry loop, a scraper, or someone abusing a free tier. Sending requests as fast as possible, it can consume most of that capacity by itself.

Everyone else calling the same server pays for that. Slower responses, or outright failures, without having done anything wrong themselves.

The token bucket, the classic answer

A common way to implement this: each caller has a bucket of tokens. Every request costs one token. Tokens refill at a fixed rate — say, three per second. If the bucket is empty, the request is rejected, or told to wait.

This naturally allows short bursts, since a caller can spend several saved-up tokens quickly. It still caps the sustained rate over time, to whatever the refill rate allows.

How it works

   bucket:  [ o o o o o ]     capacity 5, refills 3/sec

   request arrives -> take one token -> [ o o o o ]  -> allowed
   request arrives -> take one token -> [ o o o ]    -> allowed
   ... bucket empties during a burst ...
   request arrives -> bucket is empty                -> rejected
   (time passes, tokens refill)
   request arrives -> token available again           -> allowed

Quotas are the longer-horizon version of the same idea — not "three per second" but "one thousand per day." They are often tracked per user or per API key, rather than globally. One customer's heavy month then does not eat into another's.

A real example you have seen

Every free tier of a public API — weather data, maps, translation — enforces a limit like this. Go over it, and you get a clear rejection instead of a slow, half-broken response. That clear rejection is rate limiting working as intended, not a bug.

The honest part

Choosing the right numbers — how many tokens, how fast they refill — is closer to guessing well than calculating precisely. Set the limit too tight, and real users hit it during ordinary use. Set it too loose, and it stops protecting anything. Expect to revisit it after watching real traffic.

Remember this

  • Rate limiting caps how fast one caller can send requests, protecting shared capacity from any single heavy user.
  • A token bucket allows short bursts while capping the sustained rate — not the same as a flat request cap.
  • Per-caller limits matter as much as the limiting itself — a shared limit lets one heavy caller starve everyone else.

What to learn next

Developer — Code and libraries.

Setup

No installs needed beyond the standard library.

A real token bucket, and what per-user limits actually buy you

rate_limit.py
class TokenBucket:
    """Tokens refill at a fixed rate. Each request costs one token.
    `now` is passed in, not read from the clock, so this is testable
    and reproducible without a real stopwatch."""

    def __init__(self, rate_per_sec, capacity):
        self.rate = rate_per_sec
        self.capacity = capacity
        self.tokens = float(capacity)
        self.last = 0.0

    def allow(self, now):
        elapsed = now - self.last
        self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
        self.last = now
        if self.tokens >= 1.0:
            self.tokens -= 1.0
            return True
        return False


RATE, CAPACITY = 3.0, 5  # 3 requests/sec sustained, bursts up to 5

# normal: one request every 0.5s for 20s -> 40 requests, well inside the limit
normal_times = [i * 0.5 for i in range(40)]
# abusive: one request every 0.02s for 5s -> 250 requests, far over the limit
abusive_times = [i * 0.02 for i in range(250)]

print("-- one shared bucket for both users --")
shared = TokenBucket(RATE, CAPACITY)
all_events = sorted([(t, "normal") for t in normal_times] + [(t, "abusive") for t in abusive_times])
shared_allowed = {"normal": 0, "abusive": 0}
for t, user in all_events:
    if shared.allow(t):
        shared_allowed[user] += 1
print(f"normal accepted:   {shared_allowed['normal']:3d} / {len(normal_times)}")
print(f"abusive accepted:  {shared_allowed['abusive']:3d} / {len(abusive_times)}")

print("\n-- one bucket per user --")
buckets = {"normal": TokenBucket(RATE, CAPACITY), "abusive": TokenBucket(RATE, CAPACITY)}
per_user_allowed = {"normal": 0, "abusive": 0}
for t, user in all_events:
    if buckets[user].allow(t):
        per_user_allowed[user] += 1
print(f"normal accepted:   {per_user_allowed['normal']:3d} / {len(normal_times)}")
print(f"abusive accepted:  {per_user_allowed['abusive']:3d} / {len(abusive_times)}")
Output
-- one shared bucket for both users --
normal accepted:    31 / 40
abusive accepted:   18 / 250

-- one bucket per user --
normal accepted:    40 / 40
abusive accepted:   19 / 250

Purely arithmetic, no randomness or timing involved — this reproduces exactly every run, on any machine.

Walking through it

now is a parameter, not time.time(). Passing the clock in, rather than reading it internally, is what makes this testable without a real stopwatch. The entire simulation above ran in a fraction of a second, even though it models 20 seconds of traffic.

The shared bucket lets the abusive user steal capacity from the normal one. With one bucket for both, the abusive user's burst can drain tokens moments before the normal user's next request arrives. Normal drops from a perfect 40/40 to 31/40, purely from sharing a budget with someone else's misbehaviour.

Per-user buckets fully protect the normal user — 40/40 — while barely changing the abusive user's outcome. The abusive user was always going to be capped near the same number; isolating buckets does not punish them harder, it only stops their behaviour from spilling onto someone else.

Common mistakes

One global limit for every caller combined. As shown above, this lets a single bad actor degrade service for everyone. Key the limiter by user, API key, or IP address — whatever identifies a caller — not by the service as a whole.

Rejecting with no information. A bare failure forces the caller to guess why. Return a clear status and, where possible, a Retry-After value telling them exactly when to try again.

Setting capacity equal to the sustained rate. A bucket with capacity=3 at rate=3/sec allows no burst at all — every request beyond a perfectly even trickle gets rejected. Real traffic is bursty; give the bucket room to absorb a short spike.

Forgetting the limiter itself must be fast. A rate limiter that takes longer to check than the request itself takes to serve has become the bottleneck it was meant to prevent.

Try it yourself

Add a third user, "burst", sending 10 requests all at t=10.0 — a single instantaneous spike rather than a sustained hammering. Compare how many of those 10 get through against the sustained abusive user's acceptance rate, to see what the bucket's capacity parameter is actually protecting.

What to learn next

Researcher — Mathematics and papers.

Rate limiting algorithm family

  • Fixed window counts requests in discrete time windows (e.g. per calendar minute), resetting the count at each boundary. Simple, but allows up to 2x the intended rate right at a window boundary — a burst just before the reset, followed immediately by a full burst just after.
  • Sliding window log keeps every request timestamp and counts how many fall within the trailing window. Exact, but memory cost grows with request volume.
  • Sliding window counter approximates the sliding log by weighting the current and previous fixed windows, trading a small accuracy loss for constant memory — the approach behind most production API gateways (e.g. Cloudflare's, Kong's).
  • Token bucket (used above) and leaky bucket are mathematically related: token bucket controls burst admission directly; leaky bucket smooths output to a constant rate regardless of input burstiness, better suited to traffic-shaping than to admission control.

Distributed rate limiting

A single-process TokenBucket like the one above works only as long as one process sees every request. A fleet of servers behind a load balancer needs a shared view of each caller's remaining budget. This is typically implemented against a fast shared store (Redis is the common choice), using an atomic increment-and-check operation. Otherwise two servers can each admit a request that, together, exceeds the limit.

The Generic Cell Rate Algorithm (GCRA), used internally by systems like Cloudflare's rate limiter, is a token-bucket-equivalent formulation. It stores only a single timestamp per key, rather than a token count, which is cheaper to keep consistent under concurrent access from multiple servers.

Cost, in complexity terms

A well-implemented distributed token bucket is $O(1)$ per request — one atomic read-modify-write against the shared store. The sliding window log is $O(w)$ in the worst case, where $w$ is the number of requests in the window, both in memory and in the cost of pruning old entries.

References

  • Cloudflare Engineering Blog, How we built rate limiting capable of scaling to millions of domains, 2017 — a practical account of GCRA in production.
  • Brownlee and Claffy, Internet Traffic Flow Measurement, IEEE Internet Computing, 2001 — background on the token-bucket and leaky-bucket traffic models, originally from network traffic shaping rather than API design.

What to learn next