Language and NLP Section 054
Messy Real-World Text
Real user text is misspelled, half in Hindi, full of emoji and pasted out of a scanner. Here is how to survive it.
10 of 10 lessons published Three reading levels on every lesson
Lessons in order
Work top to bottom. Each lesson assumes the one above it.
- Building a spell checker
- Fuzzy matching names and addresses
- Normalising numbers, dates and currency
- Cleaning social media text
- Fixing OCR errors
- Punctuating a speech transcript
- Finding and masking personal data in text
- Augmenting text data
- When users write 'f r e e m0ney' to dodge your filter
- Labelling data with an LLM and a human reviewer