Production and MLOps Section 099

Observability for LLM Applications

Seeing inside a chain of prompts, retrievals and tool calls when a user says the answer was wrong.

9 of 9 lessons published Three reading levels on every lesson

Start with “Tracing an LLM application”

Lessons in order

Work top to bottom. Each lesson assumes the one above it.

  1. Tracing an LLM application
  2. Token and cost telemetry
  3. Versioning prompts
  4. Running evaluations on live traffic
  5. LLM-as-judge in production
  6. Capturing user feedback
  7. Redacting personal data from LLM logs
  8. Replaying production traffic
  9. Observability for agent runs