Feature engineering for time series

The reduction#

Feature-based forecasting reduces a sequence problem to supervised regression through a sliding-window transform. Given ${y_t}_{t=1}^{T}$, a lookback $L$ and a horizon $h$, construct

$$ \mathcal{D} = \left{ \left( \mathbf{x}t, \; y{t+h} \right) \right}_{t=L}^{T-h}, \qquad \mathbf{x}_t = \left[ \psi_1(y_{1:t}), \dots, \psi_d(y_{1:t}), \; \mathbf{z}_t \right] $$

• $\psi_j$ — any measurable functional of the history up to and including $t$, and no further.
• $\mathbf{z}_t$ — exogenous covariates whose values at time $t+h$ are known at time $t$.
• $h$ — the horizon; a separate model per $h$ is the direct strategy.

The measurability restriction $\psi_j: \mathcal{F}t \to \mathbb{R}$ is the formal statement of "no leakage". A centred rolling mean is $\mathcal{F}{t+k}$-measurable for $k>0$ and is therefore not an admissible feature, regardless of how the rows are split.

Direct, recursive and hybrid multi-step strategies#

Taieb, Bontempi, Atiya and Sorjamaa (2012), A review and comparison of strategies for multi-step ahead time series forecasting based on the NN5 competition, evaluates all four empirically. No strategy dominates; direct is generally safer at long horizons, recursive at short ones with a well-specified model.

Cross-sectional pooling and hierarchies#

The M5 competition (Makridakis, Spiliotis and Assimakopoulos, 2022) is the reference result for this approach. Walmart data, 42,840 series in a 12-level hierarchy, daily. The top solutions were LightGBM models trained on pooled data across series with series-identity features, calendar features, price features and SNAP-benefit flags.

Pooling is the mechanism that makes this competitive rather than a novelty. A single series of 1,900 daily observations cannot support a large model. Forty thousand series sharing a parameter set can. The implicit assumption is partial exchangeability across series after conditioning on the identity features — an assumption that is testable and often violated across very heterogeneous groups.

Hierarchical coherence — forecasts summing correctly up the hierarchy — is a separate problem. Optimal reconciliation via the MinT estimator (Wickramasuriya, Athanasopoulos and Hyndman, 2019) projects base forecasts onto the coherent subspace using the covariance of base forecast errors, and provably does not increase error under its assumptions.

Loss functions and count data#

Retail demand is non-negative, over-dispersed and intermittent. Squared error is a poor match.

• Tweedie loss with power $p \in (1,2)$ models a compound Poisson-gamma, giving a point mass at zero and a continuous positive part. This was widely used by M5 top entries; LightGBM supports objective="tweedie".
• Poisson loss for pure counts.
• Pinball loss for quantile forecasts, which is what an inventory decision actually needs. $L_\tau(y,\hat{y}) = \max(\tau(y-\hat{y}), (\tau-1)(y-\hat{y}))$ estimates the conditional $\tau$-quantile. The M5 Uncertainty track scored exactly this.
• Croston's method and its Syntetos-Boylan correction remain the classical baselines for intermittent demand and are surprisingly hard to beat on very sparse series.

Feature selection under dependence#

Standard permutation importance is invalid on autocorrelated features: permuting lag_1 breaks its correlation with lag_2, evaluating the model off its training manifold. Options that behave better:

• Blocked permutation — permute contiguous blocks longer than the autocorrelation length rather than individual rows.
• Leave-one-covariate-out refitting — expensive but honest.
• SHAP values with a background sample drawn from the same regime, though the correlation caveat from Lundberg and Lee (2017) still applies. See SHAP and LIME.

Boruta and similar wrappers assume i.i.d. resampling and should not be used unmodified.

Purging and embargo#

When labels span time — forecasting a 7-day-ahead cumulative total, for instance — a training row at $t$ and a validation row at $t+3$ share overlapping label windows. López de Prado (2018), chapter 7, formalises purging (remove training rows whose label window overlaps the validation window) and embargo (additionally drop a gap after the validation window to break residual serial correlation). Omitting these inflates validation scores in a way that survives a naive chronological split.

Global versus local models#

A local model is fitted per series; a global model shares parameters across series. Montero-Manso and Hyndman (2021), Principles and algorithms for forecasting groups of time series, prove a striking result: a global model can achieve equal or lower generalisation error than local models even when the series are generated by unrelated processes, because the effective sample size grows while model complexity is controlled. The practical implication is that pooling is worth trying even on heterogeneous panels, provided the model has enough capacity and identity features.

Reading#

• Makridakis, Spiliotis and Assimakopoulos (2022), M5 accuracy competition: Results, findings and conclusions, IJF 38(4).
• Montero-Manso and Hyndman (2021), Principles and algorithms for forecasting groups of time series: Locality and globality, IJF 37(4) — arxiv.org/abs/2008.00444.
• Taieb et al. (2012), A review and comparison of strategies for multi-step ahead time series forecasting, Expert Systems with Applications 39(8).
• Wickramasuriya, Athanasopoulos and Hyndman (2019), Optimal forecast reconciliation for hierarchical and grouped time series through trace minimization, JASA 114(526).
• López de Prado (2018), Advances in Financial Machine Learning, Wiley, chapter 7.
• Christ, Braun, Neuffer and Kempa-Liehr (2018), Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests (tsfresh), Neurocomputing 307.

What to learn next#

• LSTMs for forecasting — the neural alternative, and when it is worth the trouble.
• Evaluating a forecast — measuring these models without fooling yourself.
• XGBoost — the model family that wins most tabular forecasting competitions.