Feature stores

The point-in-time join, stated precisely#

Given an event stream $E = {(k, t, v)}$ of keys, timestamps and values, and a set of labelled decision points ${(k_i, t_i, y_i)}$, the correct training feature is:

$$ x_i \;=\; \phi\big({\,e \in E \;:\; e.k = k_i \;\wedge\; e.t \le t_i \,}\big) $$

Where $\phi$ is the aggregation. The naive alternative $\phi({e : e.k = k_i})$ over all time is a temporal leak, and it is undetectable by any check on the data alone — the values are individually correct, and only the join is wrong.

Implemented in SQL this is an AS OF join: for each entity and label timestamp, select the feature row with the greatest timestamp not exceeding it. Naively it is $O(|E| \cdot |L|)$. Sorting both sides by $(k, t)$ and merging reduces it to $O\big((|E| + |L|)\log(|E| + |L|)\big)$, which is why every feature store sorts before joining, and why pandas.merge_asof exists.

The bitemporal subtlety#

A single timestamp is not enough when facts arrive late or are corrected. Two time axes are needed:

• Valid time $t_v$: when the fact was true in the world.
• Transaction time $t_x$: when your system learned it.

The correct training feature uses what was knowable, so the filter is $t_x \le t_i$, not $t_v \le t_i$. A payment that occurred at 10:00 but reached your database at 10:40 was not available for a 10:05 decision, and a training set that uses it overstates what production can do.

Most feature stores model valid time only. Systems where late arrival is common — payments, IoT, anything mobile — need both, and getting this wrong produces a model that is optimistic by exactly the amount of the arrival delay. See streaming data.

Architecture#

The standard design is a dual store fed by one definition:

                    feature definition (one file)
                      /                      \
        batch materialisation            streaming path
                |                               |
         offline store                    online store
      (Parquet / warehouse)            (Redis / DynamoDB)
                |                               |
        point-in-time join                 key lookup
                |                               |
           training set                  live inference

Requirements differ sharply between the two sides:

Consistency between them is the hard engineering problem, and the reason feature stores exist as products rather than as libraries.

Lambda versus streaming-only#

The lambda architecture runs a batch path and a streaming path over the same definition, reconciling them. It guarantees correctness through the batch layer while the streaming layer provides freshness. The cost is two implementations, which is the very problem being solved — the definition is shared, but the execution engines differ and diverge in edge cases.

The kappa architecture (Kreps, 2014) keeps only the streaming path and reprocesses from a retained log to correct history. Simpler in principle, and it requires a log with sufficient retention and deterministic, replayable transformations.

Modern systems increasingly compile one declarative feature definition into both engines, which is the honest solution to a genuinely hard problem rather than a way around it.

Detecting skew in production#

Breck et al. (2019), Data Validation for Machine Learning, SysML, describe the TFX approach: log serving feature vectors, sample them, and compare distributions against the training data. They distinguish three failure classes worth naming separately:

1. Feature skew — the computed value differs between paths, as demonstrated above.
2. Distribution skew — the same code, different input distribution. This is genuine drift and needs a different response.
3. Scoring/serving skew — the model sees a different subset than training assumed, typically because production filters candidates before scoring.

For continuous features, compare quantiles rather than means; the L-infinity distance between empirical CDFs is a robust default. For categorical features, watch for new categories, since an unseen category at serving time is a hard failure in most encoders. Chebyshev distance over category frequencies is what TFDV uses.

Cost model for the online store#

Per prediction request with $f$ features across $g$ entity keys:

• Network round trips: $g$ with batched multi-get, $f$ without. Batching by entity is the single largest latency win.
• Storage: $O(\text{keys} \times f)$ for latest-value-only.
• Freshness lag: the streaming pipeline's end-to-end latency, which must be measured and alerted on, not assumed.

A p99 budget of 10 ms for feature retrieval is typical, and it constrains $f$ and $g$ more than most teams expect when designing the feature set.

When not to build one#

The honest guidance, which vendor material omits: a feature store pays for itself when features are shared across models and teams, or when online and offline paths genuinely differ. With one model, one team and batch scoring, a versioned transformation module invoked by both paths delivers the same correctness at a fraction of the operational cost.

The pattern — one definition, as_of as a parameter, both paths calling it — is the part that matters. The database behind it is an implementation detail you should adopt only when scale demands it.

Reading#

• Zinkevich, Rules of Machine Learning: Best Practices for ML Engineering, Google — rules 29 to 32 cover training/serving skew directly.
• Breck et al., Data Validation for Machine Learning, SysML 2019.
• Kleppmann, Designing Data-Intensive Applications, O'Reilly 2017 — chapter 11 on streams, batch and reprocessing.
• Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015 — on undeclared consumers and data dependencies.
• Feast documentation, Point-in-time correctness — the clearest worked explanation of the AS OF join in an ML context.

What to learn next#

• ETL for machine learning — the pipeline that materialises these features.
• Monitoring and model drift — watching for skew after launch.
• Feature engineering — deciding what the features should be in the first place.