Production and MLOps Section 094
Scaling and Traffic Management
Adding and removing machines as traffic moves, and sending each request to the right one.
10 of 10 lessons published Three reading levels on every lesson
Lessons in order
Work top to bottom. Each lesson assumes the one above it.
- Autoscaling on the right metric
- Autoscaling lag and headroom
- Scale to zero, and what it costs you
- Load balancing inference traffic
- Prefix-aware and sticky routing
- Rate limiting and per-user quotas
- Multi-tenancy and noisy neighbours
- Graceful shutdown and connection draining
- Circuit breakers and fallbacks
- Serving from more than one region