Backtest¶
Time-series cross-validation folds + out-of-fold capture — pure.
Backtesting fits on history, validates on a held-out future window, and records the out-of-fold (OOF) predictions the learned ensembler trains on. Two entry points:
make_folds(n, cfg) -> list[Fold]— integer-indexed CV splits overnsorted observations. Folds are anchored from the end: the latest fold validates on the finalhorizonpoints, earlier folds step back bystep.expandinggrows the train window from 0;slidingkeeps a fixedwindow(defaulting tomin_train). A series too short for the requested folds gets as many as it supports (achievable_folds), possibly none — never an exception, because a scoring shortfall must not cost the forecast.backtest_cell(series, model, cfg) -> (oof, fold_metrics, outcome)— features are built once (leakage-free: lags only look backward), then each fold is scored on its validation window.
backtest.scheme decides what a fold's score is a score of, and the three answers are
different numbers rather than cheaper approximations of one number:
expanding(the default) andslidingfit a fresh model per fold, so a score answers "how good is this model when freshly trained?".expanding_frozenfits once on the oldest fold's window and then hands the model the observations that arrived between origins, parameters held fixed — "what does refitting less often cost me?". Ten of the sixteen Python models have that seam; the rest refit and say so.expanding_stalefits once and never tells the model what happened next — "how fast does this decay if nobody touches it?". Every model supports this one, which is what makes it the scheme where a leaderboard compares like with like.
The frozen schemes also run a control arm: the same fitted model walked forward blind, scored
on the same dates, written to yhat_stale and summarised as BacktestOutcome.staleness_gap. It
is a second predict rather than a second fit, so the comparison costs a forecast. The refit
schemes can ask for the same arm with backtest.control_arm, which does cost them one extra fit
per cell — that is what makes "what is refitting buying me?" answerable on the default scheme
without switching to a scheme that answers a different question.
The no-leakage invariant is train_end + gap == val_start for every fold: training data strictly
precedes the validation window, and at the default gap of 0 they are adjacent. Raising gap
opens an embargo — observations that are neither trained on nor scored — which is how you measure a
forecast issued with a reporting lag, where the data for the last fortnight is not in yet.
Freezing is anchored on the oldest surviving fold,
whose training window is a subset of every later fold's, so a frozen model's parameters have never
seen anything a later fold is scored on.
Each fold is scored on the intervals the model already returned, not on the point forecast
alone — so coverage, pinball, interval_score and interval_width are real numbers on
the Python path rather than the NaNs they were for every run before this.
backtest.short_series decides what a series too short for the requested grid gets, and each
branch gives up something different — folds (adapt), fold independence (overlap), training
history (shrink_train), the series (skip) or the run (error). resolve_geometry is the
one place that reasoning lives; make_folds reads its answer and never re-derives it.
Public surface: Fold, FoldGeometry, BacktestOutcome, FitTally, OOF_COLUMNS,
achievable_folds, assert_panel_supports_folds, holdout_fold_id,
hpo_scoring_claim, make_folds, fit_rows, resolve_geometry, suggest_min_train,
training_width, training_window, backtest_cell.
dataclass
¶
BacktestOutcome(
refit_mode,
staleness_gap,
achieved_step=None,
achieved_min_train=None,
first_val_date=None,
last_val_date=None,
)
How the fold loop carried the model between origins — which is not always what was asked.
refit_mode is one of:
"per_fold"— a fresh model was fit for every fold. Whatexpandingandslidingdo."recondition"— fit once, then handed the observations that arrived between origins with its parameters held fixed."extrapolate"— fit once and walked forward blind; the model never saw the newer actuals."unsupported"— a frozen scheme was requested and this model has no seam for it, so the cell refit per fold instead. Recorded rather than silently substituted: someone comparing two models on anexpanding_frozenleaderboard has to be able to see that one of them was not actually frozen, or the comparison is between two different questions.
staleness_gap is loss(blind arm) - loss(primary arm) under the run's
decision_metric, both restated through metrics.loss_of so the sign means the same thing
for coverage and bias as it does for wape. Positive is the ordinary reading: never
refreshing the model costs you that much accuracy. None whenever no control arm ran — a
refit scheme that did not ask for one (backtest.control_arm), a model with no blind seam,
expanding_stale (whose primary arm is the control arm), and any cell where the metric came
back non-finite.
A cell that achieved zero folds still reports the nominal mode for its scheme; nothing was
scored, and backtest_status on the same row already says so.
The remaining four are the geometry this cell was actually scored on, which under
overlap or shrink_train is not the geometry the config asked for. backtest_note
states the substitution in prose; these state it as numbers a query can group by, which is the
difference between a reader noticing that one series was scored on a shrunken step and a reader
having to parse English to find out. achieved_step and achieved_min_train are
resolve_geometry's effective values; first_val_date and last_val_date are the span of
dates the folds actually scored, and they are dates rather than fold counts because fold_id
is an ordinal within one series' plan — on a ragged panel the same fold_id is not the same
window, so two series are only comparable on the dates they were measured over. All four are
None when no fold ran.
dataclass
¶
A running count of .fit() calls and the observations handed to them. Mutable.
Passed down into the fold loop and written to as fits happen, rather than returned up, for
the same reason resources.slot.resource_slot hands its measured/assumed/notes
lists down: the call sites are scattered across branches that already return something else,
and threading a second return value through each of them would obscure what they are for. One
tally per question — the caller decides what it is counting by deciding which tally it passes.
Why counting is not arithmetic. fit_rows estimates this at plan time as
[n, *fold_windows], which assumes a fresh fit per fold. That is right for expanding and
sliding and wrong for everything else: expanding_stale fits once for the whole cell,
expanding_frozen fits twice (the frozen arm and the blind control arm) and then refits only
the folds where reconditioning fell over, and backtest.control_arm adds one fit to a refit
scheme. Six branches of _walk_folds, six different answers. The gap between the plan-time
estimate and this tally is not an error in either — it is the cost of the scheme, and a run
that records both can be asked what freezing actually saved.
train_rows sums the training observations across those fits, because a fold trains on less
history than the final full-history fit does; four years of daily history backtested twice at a
28-day horizon is 2.94 whole-history fits, not 3.
dataclass
¶
One CV split as integer positions into the sorted series (half-open ranges).
dataclass
¶
The grid one series actually gets, after backtest.short_series has had its say.
step and min_train are the effective values, which differ from the authored ones only
under the two policies that buy folds by spending something: overlap spends the step,
shrink_train spends the training requirement. note says what was spent, in the words the
cell writes to forecast_metadata.backtest_note; None means the geometry is exactly what
the config asked for.
How many of the requested folds n observations can actually support (pure).
0 when the series gets no folds — either because it cannot hold one or because
short_series declined to score it — and cfg.backtest.n_folds when it holds them all.
Split out from make_folds because two callers need the count without the folds: the cell
records n_folds_achieved, and a reader deciding whether a run's leaderboard is comparable
needs to know a series was scored on fewer folds than its neighbours.
The count is policy-aware, so it is the number the series really gets rather than the number the
arithmetic would allow. resolve_geometry is where the policy lives; call that instead when you
also need to know what the policy did.
Source code in src/scale_forecasting/backtest.py
Enforce short_series="error" against a measured panel — raise, or return quietly (pure).
A no-op under every other policy, and under error too when every series clears the full fold
grid. The point of the mode is a hard stop for anyone who would rather not discover afterwards
that a third of their leaderboard was scored on two folds instead of five, and a hard stop is
only useful if it lands before the work. So this is called from two places, neither of which
is a cell: the submit path's pre-flight (launch_plan.preflight_short_series, which reads the
panel's series lengths straight out of the warehouse and refuses before anything is
provisioned), and each Python engine's driver once it has the panel in hand (the backstop that
still holds when submit could not reach the warehouse — a staged config, an offline plan, a
launch that skipped the CLI).
Both callers pass the same thing: one observation count per series. The refusal names how many series fall short and by how much, because "some series are too short" is not actionable and "412 of 5,000 series need 89 more observations" is.
Source code in src/scale_forecasting/backtest.py
Run CV for one series and model factory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
series
|
DataFrame
|
one ts_id's raw rows (date/target/exog columns). |
required |
model_factory
|
Callable[[], BaseModel]
|
returns a freshly-constructed model. Called once per fold on the refit schemes, so no fitted state leaks across folds; once or twice for the whole cell on the frozen schemes, where carrying the fit forward is the measurement. |
required |
cfg
|
RunConfig
|
the run config (drives features, fold geometry and |
required |
lam
|
float | None
|
the cell's fitted Box-Cox λ (None for stateless transforms), so the forward transform here matches the inverse the folds' models apply — one λ per cell. |
None
|
tally
|
FitTally | None
|
a |
None
|
owns_covariate_lags
|
bool
|
the model's |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
|
list[dict[str, float]]
|
concatenated across folds, |
BacktestOutcome
|
fold order) for the primary arm, and |
tuple[DataFrame, list[dict[str, float]], BacktestOutcome]
|
the model was carried between origins. The registry later augments this frame with |
tuple[DataFrame, list[dict[str, float]], BacktestOutcome]
|
|
tuple[DataFrame, list[dict[str, float]], BacktestOutcome]
|
it (see |
tuple[DataFrame, list[dict[str, float]], BacktestOutcome]
|
The control arm is never scored into |
tuple[DataFrame, list[dict[str, float]], BacktestOutcome]
|
selection, calibration, the leaderboard — reads that panel, and a second set of numbers in |
tuple[DataFrame, list[dict[str, float]], BacktestOutcome]
|
it would be picked up as if it were a second model. It lives in |
tuple[DataFrame, list[dict[str, float]], BacktestOutcome]
|
summarised once as |
Source code in src/scale_forecasting/backtest.py
933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 | |
Training-row count of every fit one cell performs on a series of n observations (pure).
[n, *fold_windows] — the final full-history fit the shipped forecast comes from, then one
entry per achieved fold, in fold order. A cell with backtesting off does one fit, so the list
is [n].
This is the arithmetic behind config.estimate_workload, and it exists so that arithmetic
cannot drift from the geometry: the fold windows are read off make_folds rather than
re-derived from horizon/step/min_train. A count that re-derived them would go
quietly wrong the first time the geometry changed — and fold geometry has changed twice
already (clamping, then oldest-first dropping).
Why the row counts and not just the fit count: a fold trains on less history than the final
fit, so n_folds + 1 overstates the cost of backtesting. Four years of daily history
backtested twice at a 28-day horizon is 2.94 whole-history fits, not 3.
Source code in src/scale_forecasting/backtest.py
The fold id reserved from every fit — the newest one, n_folds - 1 (pure).
The invariant this exists to state: the fold with the smallest step-back is never used to fit
anything. Not stacker weights, not inverse_error weights, not hyperparameters. Everything
that learns from the backtest learns from the inner folds; the newest fold is what those
learned things are then judged on. Without it, a learned ensemble's leaderboard number is an
in-sample fit statistic sitting in the same column as the base models' out-of-fold numbers, and
a per-series hyperparameter search is scored on the exact folds it optimised against.
It is a pure function of cfg — not of the series — and that is what makes it usable as a
join key rather than a per-cell lookup. Two facts make it safe: make_folds keeps a survivor's
original fold_id and drops the oldest folds first, so every series that achieved any
fold at all achieved this one; and engines.bigquery_sql.fold_plan numbers folds identically,
so the two runtimes agree on which fold is the holdout without having to compare dates.
A series that achieved exactly one fold has only the holdout, so it has nothing to fit on --
callers fall back to using every fold and record ensemble_scoring / hpo_scoring as
'in_sample' rather than pretending. Same at n_folds == 1, where that is true of the
whole run.
Source code in src/scale_forecasting/backtest.py
The run-level answer to "did the hyperparameter search reserve the newest fold?" (pure).
"off" when no search runs at all, "holdout" when the run's fold geometry leaves an
inner fold for one to score on, "in_sample" when n_folds == 1 and it cannot. Written
once onto the run header's job_telemetry under $.scoring.hpo so the claim is legible
for the whole run without reading a per-cell column — the per-cell hpo_scoring still says
what happened for an individual short series under per-series tuning, which this cannot.
Lives here rather than in hpo because main needs it on the submit path, and importing
hpo there would drag the whole model stack in behind models.get_model.
Source code in src/scale_forecasting/backtest.py
Build the CV folds for n observations — as many as the series supports.
Uses cfg.backtest: n_folds, horizon, step, min_train, scheme, gap,
window, short_series, min_folds and min_train_floor.
The shortfall is resolve_geometry's to answer, not this function's. step and the
training floor used below are the effective ones it returns, which under overlap or
shrink_train are not the authored ones. Everything else here — the anchoring, the numbering,
the holdout — is unchanged by the policy, and that is deliberate: a policy decides how much
evidence a short series contributes, never where a fold sits relative to its neighbours.
gap is an embargo, and it moves the training end, not the validation window. The folds
stay anchored where they were — fold k still validates on the same dates — and the fit
simply stops gap observations earlier, so train_end + gap == val_start. That is the
right way round for the thing an embargo models: a forecast issued with a reporting lag knows
the world up to gap periods before the window it is judged on. Anchoring the validation
off the training end instead would make every fold of a gap=14 run score a different
fortnight than the same fold of a gap=0 run, and the two runs would stop being comparable.
window is the sliding scheme's training width, defaulting to min_train so an
unset window reproduces the previous geometry exactly. Splitting them frees min_train to
mean only what its name says — the feasibility floor, how much history a fold needs before it
is worth scoring — while window says how much of that history the model may look at. They
were one field, and a forecaster who wanted a 90-day sliding window was also telling the fold
planner that 90 days was enough history to score on.
Clamps rather than raises — under every policy, including error. A series too short for
the requested folds used to raise ConfigError, which run_cell caught as a cell error — so
the forecast was thrown away over a scoring shortfall, and short history became the single
largest error class in the registry. The fit itself was never in question. Now the shortest
series in a panel returns the folds it can support, possibly none, and the caller still fits
and forecasts it. short_series="error" refuses the run from assert_panel_supports_folds,
before any cell exists; it never turns this function into the raising version again.
Survivors keep their fold_id from the full plan; the OLDEST folds are the ones dropped.
Both halves matter. Dropping the oldest keeps every series scored on the most recent window it
can reach, which is the window a leaderboard is about. Keeping the original numbering means
fold_id compares across series: renumbering the survivors 0..k would make MAX(fold_id)
meaningless in a panel of mixed-length series, and would silently align a short series' fold 0
against a long series' fold 0 covering a completely different date range. Fold identity is
anchored on the date, not the ordinal — see ensembler._pivot_oof.
Dropping the oldest is also what makes the holdout universal: fold n_folds - 1 survives for
every series that achieved any fold at all, so holdout_fold_id can be a function of the config
instead of a per-series lookup. Each fold carries that verdict as role.
Source code in src/scale_forecasting/backtest.py
420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 | |
Apply backtest.short_series to a series of n observations (pure).
A series long enough for the full grid never reaches a policy branch — the policies exist only for the shortfall, and each one answers it by giving up something different:
adapt(the default) gives up folds. The geometry is held exactly and the oldest folds are dropped, so the series is still scored on the most recent window it can reach, on the same dates as its longer neighbours.overlapgives up fold independence. The fold count is held and the step shrinks to buy it, so validation windows share observations. That is a real technique and a real cost: the per-fold scores are correlated, so their mean is more confident than the evidence warrants. The step is reduced only as far as it has to be, so the overlap is the least that works.shrink_traingives up training history. The count and the step are held and the training requirement drops instead, never pastmin_train_floor(which is why that field is required with this mode). Also reduced only as far as needed.skipgives up the series. Anything short of the full grid is left unscored, so every series on the leaderboard was measured on identical geometry — the comparable slice, enforced when the run happens rather than reconstructed when it is read.errorgives up the run, and does it somewhere else:assert_panel_supports_foldschecks the panel before any cell starts. Here it behaves asadapt, deliberately, so that a cell reached by any other path still forecasts — a shortfall must never cost a forecast.
min_folds is the give-up floor the first three respect. Below it the series is left unscored
rather than ranked on evidence too thin to rank it — one fold of five is not a fifth of an
answer. At its default of 1 it is exactly today's behaviour.
Source code in src/scale_forecasting/backtest.py
326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 | |
The largest backtest.min_train that still gives target_share of series all folds.
None when no positive min_train reaches the target — the panel is simply too short for
the requested fold geometry and the fix is fewer folds or a shorter horizon, not a smaller
min_train.
A series gets the full n_folds exactly when
min_train <= n - horizon - gap - (n_folds - 1) * step (rearranged from achievable_folds),
so each series has its own ceiling and the answer is the target_share quantile of those
ceilings, taken from the long end. The embargo is subtracted here for the same reason
achievable_folds subtracts it: it consumes history no fold can use, so advice that ignored it
would suggest a min_train that then achieves fewer folds than promised. Purely advisory: it
reports what the geometry permits and changes nothing, because min_train is a config field
and moving it moves the run_id.
Source code in src/scale_forecasting/backtest.py
How many observations a sliding fold trains on — window, or min_train (pure).
One function because three places need the same answer and they are in three different
languages: make_folds slices integer positions, training_window slices the history a
scale-free metric divides by, and engines.bigquery_sql._train_window_where renders a SQL date
bound. They disagreed once before over what a sliding window meant, and a MASE whose denominator
came from a different window than the fit is a number nobody can act on.
window defaults to None rather than to min_train's value so that an unset window
keeps meaning "whatever the floor is" even if the floor is later changed — and so the two ideas
stay separable in the serialized config, where a reader can see which one the author actually
chose.
Source code in src/scale_forecasting/backtest.py
The slice of a series' own history one fold trained on, given that fold's cutoff (pure).
MASE and RMSSE divide by the mean step of the training data, so which history goes in is not
a detail — it is the number. backtest_cell hands each fold its own slice and always has. The
two paths that score from a separate history read did not: the BigQuery-native engine and the
ensemble scorer both passed the whole series, including the very window being scored. That
makes a native model's MASE and a Python model's MASE for the same series answers to different
questions, which is exactly the comparison the leaderboard exists to support. This is the one
rule all of them now apply.
ds must be datetime64 and sorted ascending, paired positionally with y — the callers
read history with ORDER BY ts_id, ds, so it arrives that way. The window is every
observation at or before cutoff, which is what engines.bigquery_sql._fit_filter trains on
and what fold.train_end slices to, narrowed to the last training_width observations under
the sliding scheme, whose window is fixed-width by definition.
Counting that sliding window in observations rather than in dates is deliberate: it is what
make_folds does, so the engines agree. On a series with gaps the native SQL's date-space bound
would take slightly fewer rows — a real difference, and a smaller one than scoring against a
denominator that has seen the future.
Returns the whole of y when the cutoff is missing. That is the behaviour every run had
before the cutoff was recorded, and it is the only honest answer for an OOF frame that never
wrote one down.