Machine Learning Section 012

Calibration and Uncertainty

Getting a model to say how sure it is, and having that number mean something.

8 of 8 lessons published Three reading levels on every lesson

Start with “ROC vs precision-recall curves”

Lessons in order

Work top to bottom. Each lesson assumes the one above it.

  1. ROC vs precision-recall curves
  2. Choosing a threshold from costs
  3. Reliability diagrams and calibration error
  4. Platt scaling
  5. Isotonic calibration
  6. Proper scoring rules
  7. Conformal prediction
  8. Prediction intervals for regression