Learn AI Create the Future — with Pranay
Main site

Scoping an ML Project

Check the data exists before promising the model

A one-day audit of nulls, join coverage, and timestamps tells you whether the promised model can be built from the data that actually exists.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Before promising any model, open the actual tables and verify the data exists, connects, and was knowable at prediction time.

You do not promise biryani to twenty guests before opening the fridge. Maybe there is rice but no meat. Maybe the meat expired. The menu gets promised after the fridge check, not before.

Data works the same way. In every planning meeting, someone says "we have all the transaction data". That sentence has ended more ML projects than any modelling difficulty, because "having data" turns out to mean five different things.

Why it exists

Data can fail you in quiet ways that a meeting never reveals:

  • The column exists but is empty for a third of the rows.
  • The two tables that must join share a key that matches only 60% of the time.
  • The field you need was added last March, so older rows have nothing.
  • The value gets written after the moment you need to predict — so at prediction time, it does not exist yet.

That last one is the killer. A feature recorded after the decision is a message from the future. Train with it and the model looks brilliant; deploy it and the feature is blank.

How it works

The audit is four checks per column you plan to use:

PRESENT   what % of rows have a value?
JOINED    what % of rows survive the join from the other table?
HISTORIC  how far back does this column go?
IN TIME   was the value known BEFORE the prediction moment?

fridge check:  four columns, four checks — one afternoon
alternative:   discover it in month 3

Run it on real tables, not on the schema document. Schemas describe intentions; tables describe reality.

A real example you have seen

Filling a government form, you reach "enter your enrolment number from the previous certificate" — a certificate you never received. The form's designer assumed a document exists for everyone. Data pipelines make the same assumption about columns, and the audit is how you catch it before building on top.

Remember this

  • "We have the data" means nothing until nulls, joins, history, and timing are checked.
  • A feature written after the prediction moment is unusable, however predictive it looks.
  • The audit costs an afternoon and rescues months. Run it before any promise.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install pandas

Outputs verified with pandas 2.2.

The afternoon audit, in code

Loan applications, plus a bureau table someone promised "has the credit history". The audit interrogates both.

fridge_check.py
import pandas as pd

applications = pd.DataFrame({
    "app_id": [1, 2, 3, 4, 5, 6, 7, 8],
    "applied_on": pd.to_datetime(["2025-07-01", "2025-07-02", "2025-07-03",
        "2025-07-05", "2025-07-08", "2025-07-09", "2025-07-11", "2025-07-12"]),
    "monthly_income": [42000, None, 61000, 38000, None, 55000, None, 47000],
})

# A second table someone promised "has the credit history".
bureau = pd.DataFrame({
    "app_id": [1, 3, 4, 6, 8],
    "credit_score": [710, 655, 590, 730, 680],
    "score_pulled_on": pd.to_datetime(["2025-06-28", "2025-07-01", "2025-07-20",
                                       "2025-07-07", "2025-07-10"]),
})

merged = applications.merge(bureau, on="app_id", how="left")

null_income = merged["monthly_income"].isna().mean()
have_score = merged["credit_score"].notna().mean()
# a score pulled AFTER the application date did not exist at decision time
late = (merged["score_pulled_on"] > merged["applied_on"]).sum()

print(f"applications:                {len(merged)}")
print(f"income missing:              {null_income:.0%}")
print(f"credit score join coverage:  {have_score:.0%}")
print(f"scores pulled after the application date: {late}")
Output
applications:                8
income missing:              38%
credit score join coverage:  62%
scores pulled after the application date: 1

The walkthrough

38% missing income is a decision, not a detail. Impute it, drop those rows, or build a separate no-income model path — each choice changes the project's scope. Handling missing data covers the options; the audit's job is surfacing the number now.

62% join coverage means the promised table covers 5 of 8 applicants. Ask why before modelling: are the missing three new-to-credit customers? Then missingness itself correlates with the label, and dropping them biases everything.

One score was pulled on July 20 for a July 5 application. At decision time that score did not exist. Train with it and you leak the future — this exact pattern, features timestamped after the event, is among the most common causes of models that ace validation and fail in production.

how="left" is the honest join. An inner join silently deletes the rows with no match, and the coverage problem disappears from view instead of into a number.

Common mistakes

Auditing the schema instead of the rows. The schema says monthly_income: decimal, not null. The rows say 38% missing, because the constraint was added last year. Trust isna().mean(), not documentation.

Checking coverage overall but not over time. A column added in March is 100% present in recent rows and 0% before. Group the null rate by month — one line of pandas — before believing any average.

Ignoring why values are missing. Income missing because self-employed applicants skip the field is information; the missingness pattern correlates with the outcome. Deleting those rows quietly changes who the model works for.

Auditing once. Pipelines change upstream without notice. The same script, scheduled, becomes your early-warning system — data validation tests grow from exactly this seed.

Try it yourself

Add a region column to applications and compute the null rate of monthly_income per region. Then compute join coverage per week of applied_on. Both cuts routinely reveal problems the overall numbers hide.

What to learn next

Researcher — Mathematics and papers.

Missingness mechanisms

Rubin's taxonomy (Rubin, 1976, Inference and Missing Data, Biometrika) classifies missingness by its dependence structure:

$$ \text{MCAR}: P(M \mid X, Y) = P(M) \qquad \text{MAR}: P(M \mid X, Y) = P(M \mid X_{obs}) \qquad \text{MNAR}: \text{otherwise} $$

Where:

  • $M$ — the missingness indicator matrix.
  • $X_{obs}$ — the observed portion of the features.
  • MCAR / MAR / MNAR — missing completely at random, at random, not at random.

Complete-case analysis is unbiased only under MCAR; most business data is MNAR (income withheld because of its value). The audit's per-segment null rates are a cheap test against MCAR: any structure in missingness rejects it. Under MNAR, the missingness indicator itself is a legitimate, often predictive, feature — with the caveat that it may encode the outcome pathway you are trying to predict.

Point-in-time correctness

The "scores pulled after the application" check is a special case of point-in-time (PIT) correctness: every feature value used for entity $e$ at prediction time $t$ must satisfy $\text{observed_at}(x_{e}) \leq t$. Violations are temporal leakage. Formal treatments live in the feature-store literature: PIT joins ("as-of" joins) reconstruct the feature state as of each historical prediction time. Kapoor and Narayanan (2023), Leakage and the Reproducibility Crisis in ML-based Science (Patterns), survey 294 papers across 17 fields affected by leakage variants, PIT violations prominent among them.

Data documentation as institutional memory

  • Gebru et al. (2021), Datasheets for Datasets (CACM) — a structured questionnaire covering provenance, collection, and known gaps; the audit above is a minimal executable subset.
  • Sambasivan et al. (2021), "Everyone wants to do the model work, not the data work" (CHI) — documents "data cascades": compounding downstream failures from upstream data neglect, observed in 92% of studied AI projects. The strongest empirical argument that this lesson's afternoon is well spent.

Quantifying joinability

Join coverage generalises to record linkage quality. When keys are noisy (names, addresses), exact-join coverage understates true overlap, and probabilistic linkage (Fellegi and Sunter, 1969, JASA) bounds the achievable match rate. The audit should therefore report both exact coverage and, where keys are soft, an estimated linkable fraction — the gap between them is a data-engineering project that belongs in the plan.

What to learn next