Language and NLP Section 049

Chunking and Long Documents

How to cut a 200-page document into pieces without cutting the answer in half.

8 of 8 lessons published Three reading levels on every lesson

Start with “Chunking strategies compared”

Lessons in order

Work top to bottom. Each lesson assumes the one above it.

  1. Chunking strategies compared
  2. Semantic chunking
  3. Chunking by document structure
  4. Chunking source code and tables
  5. Giving each chunk its context back
  6. Retrieve small, return big
  7. Keeping the trail back to the source
  8. Long context or retrieval?