What is time series data?

Formal setup#

A univariate time series is a realisation of a stochastic process ${Y_t}_{t \in \mathbb{Z}}$ observed at $t = 1, \dots, T$.

• $Y_t$ — the random variable at time index $t$.
• $y_t$ — the observed value, one realisation of $Y_t$.
• $T$ — the number of observations available.
• $h$ — the forecast horizon, the number of steps ahead being predicted.

The forecasting problem is to estimate the conditional distribution

$$ p!\left(Y_{T+h} \mid Y_{1:T} = y_{1:T}\right) $$

and the point forecast $\hat{y}{T+h|T}$ is a functional of it. Under squared-error loss the optimal point forecast is the conditional mean $\mathbb{E}[Y{T+h} \mid \mathcal{F}_T]$, where $\mathcal{F}_T$ is the sigma-algebra generated by observations up to $T$. Under absolute-error loss it is the conditional median. These differ for skewed series, which is why a metric choice is a modelling decision, not a reporting decision.

Why i.i.d. machinery does not transfer#

Standard supervised learning theory assumes the training sample is independent and identically distributed. Time series violate both halves.

Dependence. $\mathrm{Cov}(Y_t, Y_{t-k}) \neq 0$ for small $k$ in almost every real series. The autocovariance function

$$ \gamma(t, t-k) = \mathbb{E}!\left[(Y_t - \mu_t)(Y_{t-k} - \mu_{t-k})\right] $$

is exactly the structure the model is meant to exploit, so it cannot be assumed away.

Non-identical distribution. $\mu_t$ and $\gamma$ typically vary with $t$. This is the subject of stationarity, and stationarity assumptions are what allow $\gamma(t, t-k)$ to collapse to a function $\gamma(k)$ of lag alone.

The consequence for validation is not a matter of taste. K-fold cross-validation estimates generalisation error under exchangeability. Time series are not exchangeable, so the estimator is biased, and the bias is optimistic. Bergmeir and Benítez (2012), On the use of cross-validation for time series predictor evaluation, quantifies this and gives the conditions under which a blocked variant recovers validity.

Rolling-origin evaluation#

Tashman (2000), Out-of-sample tests of forecasting accuracy, formalises the standard protocol. For origins $t_0, t_0+1, \dots, T-h$ produce $\hat{y}{t+h|t}$ using only $y{1:t}$, then aggregate the errors $e_{t+h} = y_{t+h} - \hat{y}_{t+h|t}$.

Two design choices matter and are frequently conflated:

• Expanding window — train on $y_{1:t}$, which grows. Preferred when the data-generating process is stable and data is scarce.
• Sliding window — train on $y_{t-w+1:t}$ for fixed $w$. Preferred under regime change, at the cost of variance.

A purge gap between train and test is required whenever features have look-back windows that would otherwise straddle the boundary, and whenever the label itself spans time. López de Prado (2018), Advances in Financial Machine Learning, chapter 7, gives the purging and embargo construction for overlapping labels.

Decomposing the error floor#

Write the $h$-step error as

$$ \mathbb{E}!\left[(Y_{T+h} - \hat{y}{T+h|T})^2\right] = \underbrace{\mathrm{Var}(Y{T+h} \mid \mathcal{F}T)}{\text{irreducible}} + \underbrace{\left(\mathbb{E}[Y_{T+h}\mid\mathcal{F}T] - \hat{y}{T+h|T}\right)^2}_{\text{reducible}} $$

For a random walk $Y_t = Y_{t-1} + \varepsilon_t$ with $\varepsilon_t \sim \mathrm{WN}(0, \sigma^2)$, the conditional mean is $y_T$ for every $h$, and the irreducible term is $h\sigma^2$. Forecast uncertainty grows linearly in the horizon and no model can shrink it. This is the formal version of the honesty warning in the beginner block, and it is why the naive forecast is a competitive benchmark on financial series rather than a straw man.

Benchmarks that must be beaten#

Any proposed model should be reported against these, on the same rolling origins:

The M-competitions (Makridakis et al., 1982 through M5 in 2020) repeatedly found that simple methods and combinations of them are strong. M4 (2018) was won by a hybrid of exponential smoothing and a recurrent network; the pure-machine-learning entries below it underperformed statistical baselines. Report the benchmark or the result is uninterpretable.

Reading#

• Hyndman and Athanasopoulos, Forecasting: Principles and Practice, 3rd edition — free at otexts.com/fpp3. The standard modern reference.
• Box, Jenkins, Reinsel and Ljung, Time Series Analysis: Forecasting and Control, 5th edition, 2015.
• Bergmeir and Benítez (2012), On the use of cross-validation for time series predictor evaluation, Information Sciences 191.
• Makridakis, Spiliotis and Assimakopoulos (2018), The M4 Competition: Results, findings, conclusion and way forward.
• Tashman (2000), Out-of-sample tests of forecasting accuracy: an analysis and review, International Journal of Forecasting 16.

What to learn next#

• Trend, seasonality and noise — the three parts every series is made of.
• Train, test and validation splits — the ordinary case, so you can see exactly what changes here.
• Evaluating a forecast — how to score one without fooling yourself.