Time Series Backtesting and Rolling-Origin Validation

The operations team has weekly electricity use for the last 156 weeks and believes that seasonal demand and equipment operating patterns may affect the forecast.

Here, seasonal naive (Seasonal Naive) is a benchmark model that forecasts the value at the same position in the previous season as is. If weekly data are assumed to have annual seasonality, using the value from 52 weeks earlier is one example. However, the actual seasonal period should be chosen according to how the data are generated.

The research question is "Using only the data available at the actual time of forecasting, which model forecasts electricity use for the next 4 weeks most reliably?".

So the last 4 weeks, weeks 153–156, are set aside as the final holdout, and the candidate models and settings are decided using weeks 1–152 only.

After running backtests several times within the development period, choose the most suitable model and retrain it on all of weeks 1–152. Then forecast weeks 153–156 for the first time and calculate the final score. If you change the model or settings again after seeing this final result, the holdout is no longer independent evaluation data.

What matters is that the final holdout is not simply data set aside at the end, but data you have committed not to use for model selection.

For example, the first validation could train through week 120 and forecast weeks 121–124, and the second could train through week 128 and forecast weeks 129–132.

Every fold forecasts the 4 weeks after its training data. Weeks 149–152 remain in the development period, so they can be used when retraining the final model, and the final holdout of weeks 153–156 is not included in backtesting.

Having unused weeks between folds is not necessarily wrong. How often to move the forecast origin is decided by considering computational cost, the forecasting cycle and the length of the data. To evaluate forecast performance in finer detail, you can also move the origin every week.

An expanding window lengthens the training period each time new data arrive, as in weeks 1–120, then 1–128. If old information remains useful, it has the advantage of using many observations.

A fixed window, by contrast, moves forward while keeping a constant length, such as the most recent 104 weeks. For example, if the first training period is weeks 17–120, the next can use weeks 25–128. It can be useful when operations have changed a lot recently and old data no longer describe the current situation well.

Neither method is always better. The validation scheme should match how the real forecasting system is retrained each time.

Among the most widely used metrics are mean absolute error (MAE), root mean squared error (RMSE) and mean absolute percentage error (MAPE). MAE is the average of the absolute values of the errors.

RMSE squares the errors, averages them and takes the square root. Because it penalizes large errors relatively more, it is worth consulting when you especially want to weigh situations where a large power shortfall occurs.

MAPE expresses the error as a percentage of the actual value. But it is undefined when the actual value is 0 and can become excessively large when the actual value is close to 0. For this reason it is not an appropriate metric for every time series.

The cold-storage center's electricity use is measured in kWh, so MAE and RMSE are easy to interpret. When comparing several centers of different sizes, you can also consider a scaled metric such as MASE.

It is useful for comparing forecast performance across time series of different scales. For a series with seasonal period m, the seasonal MASE is usually calculated relative to lag-m differences in the training data.

Here, the denominator must be calculated only on the training period of that fold, not on the test period. Calculating the denominator from future test values means using future information in the validation step.

Generally, a MASE below 1 means the forecast errors were smaller than the training-period mean error of the naive benchmark used in the denominator. However, this is not always the same comparison as the score obtained by actually running that benchmark model on the test period, so state which benchmark was used.

The results below are the per-fold 4-week-ahead forecast MAE (kWh).

Because the test periods of the four folds are all 4 weeks long, the arithmetic mean of the fold MAEs equals the MAE over all 16 test observations. SARIMA had the lowest mean MAE at 92 kWh, followed by ETS at 105 kWh and seasonal naive at 120 kWh.

In this example, SARIMA had the smallest error in every fold. But with only four folds, there is no guarantee that the score differences will hold in the future. In particular, you should also check whether errors increase in certain seasons or during equipment maintenance periods.

Uncertainty generally tends to increase as the forecast horizon lengthens, but the actual pattern of errors depends on the model and the data. So in addition to the overall mean MAE, it is good to check errors by forecast lag.

For example, you can collect only the first forecast week of each fold and calculate its MAE, then do the same for the second, third and fourth weeks. This lets the operations team conclude, for example, that "next week's demand forecast" is accurate enough but "the demand forecast 4 weeks out" needs a larger buffer.

Also, if forecasts are updated weekly in real operations, backtesting should mimic forecasts being updated every week. In validation, do not confuse forecasting 4 weeks at once with adding the actual value each week and forecasting again.

A prediction interval (Prediction Interval) provides a range in which a future observation is expected to fall, reflecting a specified level of uncertainty. For example, if next week's electricity forecast is 8,200 kWh and the 90% prediction interval is 7,800–8,650 kWh, the model is presenting that range of uncertainty under certain assumptions. This does not mean the actual use will definitely fall within that interval.

In backtesting you can look at prediction interval coverage (Coverage) and interval width (Width) together. For a 90% prediction interval, check whether the actual values fall inside it about 90% of the time over a sufficiently large number of future observations. But an interval that is too wide may be of little help for actual decisions, so width matters too.

Also, when the sample is small, such as 16 validation observations, the empirical coverage varies a lot, so do not draw firm conclusions about the accuracy of prediction intervals from small differences. If needed, you can also evaluate the interval score (Interval Score) or quantile loss (Pinball Loss).

For example, if you standardize using the mean and standard deviation of all 156 weeks and then run backtesting, information from the future test periods is reflected in the training step.

Imputation of missing values, calculation of outlier thresholds, feature selection and model settings must also be decided only within each fold's training data. Decomposing seasonality using the whole series and then using trend values at future time points as explanatory variables can also be leakage.

The same applies to exogenous variables. If the actual future outdoor temperature could not have been known at the time of forecasting, you must not feed the future observed temperature into the backtest as is. Instead, you need to use the weather forecast that would have been available at the time, or forecast the exogenous variable itself as part of the procedure.

On the other hand, calendar information that is already fixed, such as holidays or days of the week, can be known for future time points and can therefore be used. The key criterion is not "does the value belong to a future time point?" but whether it could actually have been known when the forecast was generated.

Even then, you must not change the model orders or preprocessing settings to fit the final holdout results.

SARIMA also had the smallest MAE in the final evaluation, but at 98 kWh it was higher than its backtest mean of 92 kWh. Such a difference can arise from the demand pattern of the test period or from random variation.

The final holdout is only 4 weeks, so this single value cannot establish long-term operating performance. After deployment, it is good to monitor forecast errors, bias, prediction interval coverage and so on periodically to check whether performance degrades.

If exogenous variables were used, also explain how their future values were obtained at the time of forecasting.

To do this, it is useful to separate training and testing in time order and to run rolling backtests from several past time points.

You should also review not only scores such as MAE or RMSE but also errors by forecast horizon, prediction interval coverage, model stability and real operating conditions together. A final holdout not used for model selection helps confirm performance independently, but results from a short period alone cannot guarantee lasting performance.