Evaluating a forecast

Notation#

Let $y_t$ be the actual, $\hat{y}_t$ the forecast, $e_t = y_t - \hat{y}_t$ the error, $n$ the number of scored points, and $m$ the seasonal period.

$$ \mathrm{MAE} = \frac{1}{n}\sum |e_t|, \qquad \mathrm{RMSE} = \sqrt{\frac{1}{n}\sum e_t^2}, \qquad \mathrm{MAPE} = \frac{100}{n}\sum \frac{|e_t|}{|y_t|} $$

$$ \mathrm{MASE} = \frac{\frac{1}{n}\sum_{t}|e_t|}{\frac{1}{T-m}\sum_{t=m+1}^{T}|y_t - y_{t-m}|} $$

The denominator is the in-sample mean absolute error of the seasonal naive method over the training set of length $T$. Hyndman and Koehler (2006) proposed it precisely because it is scale-free, defined when $y_t = 0$, and has a finite mean under weaker conditions than percentage errors.

Which functional each loss elicits#

This is the part that decides model rankings and is most often skipped.

A scoring function $S$ is consistent for a statistical functional $\mathcal{T}$ if $\mathbb{E}[S(\mathcal{T}(F), Y)] \le \mathbb{E}[S(x, Y)]$ for all $x$ and all $F$ in the class. Gneiting (2011), Making and evaluating point forecasts, JASA 106(494), sets this out:

The consequence is concrete. On a right-skewed demand distribution the mean exceeds the median, so an MAE-optimal and an RMSE-optimal forecaster produce systematically different numbers from the same predictive distribution. Ranking models by MAE when the decision needs a mean is an error of specification, not of taste.

MAPE is not consistent for any standard functional. Its minimiser depends on the shape of the distribution in a way that has no clean interpretation. Kolassa (2020), Why the "best" point forecast depends on the error or accuracy measure, IJF 36(1), works through examples where MAPE-optimal forecasts for count data are absurd — for low-count Poisson demand the MAPE-optimal point forecast can be zero.

Proper scoring rules for distributions#

Point metrics discard the uncertainty. For a predictive CDF $F$ and observation $y$, the continuous ranked probability score is

$$ \mathrm{CRPS}(F, y) = \int_{-\infty}^{\infty} \left(F(z) - \mathbf{1}{z \ge y}\right)^2 dz $$

CRPS is strictly proper: its expectation is uniquely minimised by the true predictive distribution. It reduces to MAE when $F$ is a point mass, so it is directly comparable with point metrics. It also has a closed form for Gaussian and for empirical-sample predictions, making it cheap on simulated forecast paths.

Gneiting and Raftery (2007), Strictly proper scoring rules, prediction, and estimation, JASA 102(477), is the reference. Gneiting, Balabdaoui and Raftery (2007) add the operational principle: maximise sharpness subject to calibration. Check calibration with a PIT histogram — under a correct predictive distribution the values $F_t(y_t)$ are uniform on $[0,1]$; a U-shape means intervals are too narrow, a hump means too wide.

Comparing two forecasters#

Do not compare means of loss without a test. The Diebold-Mariano statistic (1995) tests $H_0: \mathbb{E}[d_t]=0$ for the loss differential $d_t = L(e_{1t}) - L(e_{2t})$:

$$ \mathrm{DM} = \frac{\bar{d}}{\sqrt{\widehat{\mathrm{LRV}}(d)/n}} $$

where $\widehat{\mathrm{LRV}}$ is a heteroskedasticity- and autocorrelation-consistent long-run variance estimate — necessary because $h$-step errors are MA($h-1$) correlated by construction. Harvey, Leybourne and Newbold (1997) give a small-sample correction and recommend a $t_{n-1}$ reference distribution.

Two constraints are routinely violated: DM is not valid for nested models (use Clark and West, 2007), and it is not valid across many series without a multiple-comparison adjustment. For ranking many methods over many series, the Friedman test followed by the Nemenyi post-hoc procedure (Demšar, 2006) is the standard, and it is what the M4 and M5 analyses used.

Backtest design#

The estimator being computed is $$ \widehat{\mathrm{err}} = \frac{1}{|\mathcal{O}|}\sum_{t \in \mathcal{O}} L!\left(y_{t+h}, \hat{y}_{t+h|t}\right) $$ over a set of origins $\mathcal{O}$. Three design decisions determine its bias and variance:

• Expanding versus sliding window. Expanding uses all history and is preferred under stability; sliding adapts to regime change with higher variance.
• Origin spacing. Overlapping horizons make the fold errors strongly dependent, so the naive standard error across folds understates uncertainty. Spacing origins by at least $h$ (as the code above does) reduces but does not remove this.
• Purge and embargo when features have lookback windows or labels span time. See López de Prado (2018), chapter 7.

Bergmeir, Hyndman and Koo (2018), A note on the validity of cross-validation for evaluating autoregressive time series prediction, establishes a useful and counterintuitive result: for purely autoregressive models with uncorrelated residuals, standard $k$-fold cross-validation is asymptotically valid and more efficient than a single holdout. The validity fails as soon as residuals remain correlated, which is the common case, so rolling origin remains the safe default.

Reading#

• Hyndman and Koehler (2006), Another look at measures of forecast accuracy, IJF 22(4) — the MASE paper.
• Gneiting (2011), Making and evaluating point forecasts, JASA 106(494).
• Gneiting and Raftery (2007), Strictly proper scoring rules, prediction, and estimation, JASA 102(477).
• Diebold and Mariano (1995), Comparing predictive accuracy, JBES 13(3).
• Kolassa (2020), Why the "best" point forecast depends on the error or accuracy measure, IJF 36(1).
• Bergmeir, Hyndman and Koo (2018), A note on the validity of cross-validation for evaluating autoregressive time series prediction, CSDA 120.

What to learn next#

• Anomaly detection — using forecast errors as an alarm rather than a score.
• Monitoring and model drift — keeping these numbers honest after deployment.
• Model evaluation — the general case, so you can see exactly what time adds.