Production and MLOps Section 098

Monitoring Models in Production

Your service is up and answering. This is how you find out whether the answers are still any good.

11 of 11 lessons published Three reading levels on every lesson

Start with “Measuring data drift”

Lessons in order

Work top to bottom. Each lesson assumes the one above it.

  1. Measuring data drift
  2. Concept drift
  3. Training-serving skew
  4. Working with delayed labels
  5. Monitoring when you have no labels
  6. Logging every prediction
  7. Monitoring by segment
  8. Alerts people do not ignore
  9. When model metrics and business metrics disagree
  10. Monitoring a vector index
  11. Feedback loops in production