Language and NLP Section 055

Evaluating Text Systems

How to tell whether your text system got better, when there is no single right answer to compare against.

7 of 7 lessons published Three reading levels on every lesson

Start with “BLEU, chrF and COMET”

Lessons in order

Work top to bottom. Each lesson assumes the one above it.

  1. BLEU, chrF and COMET
  2. ROUGE and what it misses
  3. BERTScore and embedding-based metrics
  4. Exact match and F1 for question answering
  5. Natural language inference
  6. Using an LLM to grade text
  7. Do your labels even agree?