Learn AI Create the Future — with Pranay
Main site

Messy Real-World Text

Normalising numbers, dates and currency

Normalising numbers, dates and currency turns "12/08/2025", "Rs. 4,999" and "1,20,000" into one consistent, comparable format.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Normalising numbers and dates converts the many ways people write them into one consistent format. A computer can then compare and sort them correctly.

Think about a shopkeeper reading receipts from three customers. One writes "Rs. 500," another "₹500," a third only "500 rupees." All three mean the same amount. Only a person who already understands money can see that instantly.

Software has no such built-in understanding. "12/08/2025," "12 Aug 2025" and "August 12, 2025" all mean the same date. A computer sees three unrelated strings, until something teaches it otherwise.

Why it exists

People type numbers, dates and currency in whatever format feels natural to them. Different countries even disagree on which number comes first in a date. Databases and downstream code need one predictable format to sort, compare and calculate correctly.

Without normalisation, sorting a column of dates from different sources produces garbage. String sorting puts "9 Jan" after "10 Feb." "9" sorts after "1" as a character, not as a number.

How it works

  "1,20,000"   -->  120000            (strip Indian-style grouping commas)
  "Rs. 4,999"  -->  "₹4999"           (spell out the currency consistently)
  "12/08/2025" -->  "2025-08-12"      (one unambiguous date format)

The tricky part is rarely the conversion itself. It is figuring out which format the input was actually written in, before converting it.

Where you have already seen it

  • E-commerce order history, showing dates and prices consistently even though they were entered through different systems over the years.
  • Bank statements, converting dates from a card network's format into the bank's own display format.
  • Spreadsheet imports, where a date column pasted from another country silently swaps day and month.

Remember this

  • Numbers, dates and currency are written many different ways by different people and systems, and normalisation makes them comparable.
  • Date format is genuinely ambiguous sometimes. "12/08" could be 12 August or 8 December, depending on the writer's country.
  • Sorting unnormalised dates or numbers as plain text produces wrong results. Text sorting is not number or date sorting.

What to learn next

Developer — Code and libraries.

Setup

Nothing to install. Pure Python standard library — re and datetime.

Normalising Indian-style numbers, currency and dates

normalise_demo.py
import re
from datetime import datetime

def normalise_indian_number(s):
    # Indian grouping uses commas after the last 3 digits, then every 2 digits.
    # Stripping commas works regardless of which grouping style was used.
    return s.replace(",", "")

def normalise_currency(s):
    s = s.strip()
    s = re.sub(r"^Rs\.?\s*|^INR\s*", "₹", s, flags=re.IGNORECASE)
    return s

DATE_FORMATS = ["%d/%m/%Y", "%d-%m-%Y", "%d %b %Y", "%d %B %Y"]

def normalise_date(s):
    for fmt in DATE_FORMATS:
        try:
            return datetime.strptime(s, fmt).date().isoformat()
        except ValueError:
            continue
    return None

for s in ["1,20,000", "Rs. 4,999", "INR 15,00,000"]:
    print(f"{s:16} -> {normalise_currency(normalise_indian_number(s))}")

print()
for s in ["12/08/2025", "05-01-2026", "9 Aug 2025", "23 September 2025"]:
    print(f"{s:20} -> {normalise_date(s)}")
Output
1,20,000         -> 120000
Rs. 4,999        -> ₹4999
INR 15,00,000    -> ₹1500000

12/08/2025           -> 2025-08-12
05-01-2026           -> 2026-01-05
9 Aug 2025           -> 2025-08-09
23 September 2025    -> 2025-09-23

Line by line

normalise_indian_number only strips commas. It works for both "1,20,000" (Indian lakh-crore grouping) and "120,000" (Western thousand grouping), because in both cases the commas are pure formatting and the digits are unaffected by which grouping style was used.

normalise_date tries several formats in order, keeping the first that parses successfully. %d/%m/%Y reads "12/08/2025" as day-then-month, the convention used across India and most of the world outside the United States.

Every output date lands in ISO 8601 format (YYYY-MM-DD). This is deliberate — ISO dates sort correctly as plain text, unlike almost every other common format, because the most significant part of the date comes first.

Common mistakes

Treating "12/08/2025" as unambiguous. It is not. Read as day-month-year, it is 12 August. Read as month-day-year, the American convention, it is 8 December — a completely different date. normalise_date above assumes day-first, which is correct for Indian input but silently wrong for US-formatted input. Always know which convention your input source actually uses before parsing.

Sorting dates or amounts as plain strings. "9 Aug 2025" sorts after "23 September 2025" as text, because "9" is a larger character than "2." Convert to a real date or numeric type before sorting, never sort the display string.

Assuming one currency symbol per source. Real data mixes "₹," "Rs.," "INR," and sometimes no symbol at all with an implied currency. Normalise all known variants to one canonical form before doing any calculation.

Try it yourself

Add a date string in an ambiguous format — like "03/04/2025" — and check which of the two reasonable readings normalise_date picks. Then add a second parsing attempt using %m/%d/%Y and compare the two results side by side, to see exactly how much a single format assumption can change the outcome.

What to learn next

Researcher — Mathematics and papers.

Why date parsing cannot be fully automatic

A date like "03/04/2025" is genuinely underdetermined without external context — locale convention, surrounding text, or a known data source. Formally, given an ambiguous date string d, resolving it correctly requires either:

  • A known locale prior L, mapping format conventions to regions (day-month-year across most of the world, month-day-year in the United States), or
  • A validity constraint that rules out one reading — "13/04/2025" can only be day-month-year, since no 13th month exists, which is why some ambiguous dates resolve themselves and others do not.

Production date-parsing libraries (dateutil, dateparser) apply heuristics combining both: a configured or detected locale, plus range validation, falling back to explicit ambiguity warnings when neither resolves the case. No purely statistical approach removes the ambiguity for a date like "03/04/2025" specifically — the information needed is not present in the string itself.

Numeral systems and grouping conventions

Indian numbering groups digits as ##,##,##,### (crore, lakh, then thousands) — a comma after the first three digits from the right, then every two digits thereafter. Western numbering groups uniformly in threes: ###,###,###. Both are valid, unambiguous once the convention is known, and indistinguishable from each other by comma position alone once the total digit count is small enough — "1,20,000" (Indian, 120,000) is disambiguated by the position of the first comma alone, but this stops working reliably as numbers shrink.

Devanagari and other Indic scripts additionally have their own native digit glyphs (० १ २ ३ ...) alongside Western Arabic numerals, both in active use depending on context, register and source — a normalisation step for genuinely multilingual Indic text needs a digit-mapping table in addition to the comma-stripping logic shown in the developer block.

Currency normalisation as entity resolution

Currency symbol normalisation is a small, well-defined instance of the same string-to-canonical-form problem covered generally in fuzzy matching names and addresses: many surface forms ("₹," "Rs.," "INR," "Rupees," occasionally no marker at all with currency implied by context) map to one canonical value. Unlike free-form name matching, this mapping is small and enumerable, so exact-match dictionary lookup after light normalisation (case-folding, stripping periods) is standard and sufficient — fuzzy matching is unnecessary overhead here.

Key references

  • ISO 8601:2019. Date and time — Representations for information interchange. International Organization for Standardization. — the YYYY-MM-DD format used throughout the developer block, chosen specifically for its sort-order property.
  • Chang, A. & Manning, C. (2012). SUTime: A library for recognizing and normalizing time expressions. LREC. — a widely-used rule-based system for extracting and normalising natural-language date and time expressions from unstructured text.
  • Strötgen, J. & Gertz, M. (2013). Multilingual and Cross-domain Temporal Tagging. Language Resources and Evaluation. — HeidelTime, covering temporal normalisation across many languages including several Indian languages.

Current state and open problems

Rule-based systems (dateutil, SUTime, HeidelTime) remain the standard for structured date and number normalisation, since the space of real-world formats, while large, is enumerable and the rules are stable over time — this is not a task that benefits much from being reframed as a machine-learning problem.

The genuinely unresolved part is relative and vague date expressions inside free text — "next Tuesday," "last month," "a couple of years ago" — which require both a reference date (when was this text written or spoken) and, for the vaguest expressions, an accepted range rather than one exact date. This shades into the harder problem of temporal reasoning over unstructured text, well beyond simple format conversion.

What to learn next