Run in Colab Enterprise

Run in Colab Enterprise: click the badge to import this notebook, pick a runtime, and Run all. The Terraform-deployed templates already carry the SF_* run identity in their env, so there's no environment cell to fill in. Runs on the sf-main runtime template (Python 3.11). See docs/notebook_runtimes.md for the per-notebook template mapping and the headless acceptance harness.

08 · The Multi-Family Parallel Master Workflow: Multi-Engine DAG, Foundation Models, Explainability, Ensembles & Inventory Policy¶

This notebook demonstrates the flagship architecture of scale-forecasting: a single RunConfig that routes active model families in parallel across their optimal cloud runtimes (dag.py & router.py — mixing spark, vertex, gce, gke, ray, vertex_automl, and bigquery, or running all Python families on a single shared gke cluster with dedicated CPU and GPU node pools), tears down each family's compute the instant it finishes (zero-idle per-family compute), joins all families in a downstream stacked ensemble stage, and runs the full post-run decision workbench (FVA, Two-Tier Explainability, Anomalies & Change Points, and Operational Inventory Replenishment Policies).

At a glance¶

Dimension	Details
Learning outcomes	Dispatch a single RunConfig across parallel per-family jobs (statistical, ml, deep_learning including [models-foundation] chronos_bolt, native, and optional automl), inspect v_run_jobs provisioning vs. execution telemetry and version parity, blend all families in a stacked ensemble, and run FVA, TreeSHAP attributions, anomaly/change-point detection, and $(s, S)$ inventory replenishment policies.
Prerequisites	Deployed scale-forecasting Google Cloud project (SF_* env vars) + scale-forecasting[gcp].
Estimated time	~8–15 minutes for the 4-family run (+ ~45–75 minutes if optional vertex_automl Tabular Workflow is enabled).
Estimated run cost	Estimated ~\$0.25–\$1.00 USD for statistical + ml + deep_learning + native (Dataproc / Vertex AI / GCE / GKE CPU & GPU minutes + BigQuery). Enabling optional vertex_automl adds an estimated ~\$3–\$22 USD (Vertex AI Pipelines, Dataflow, and AutoML training/batch prediction node-hours).
Ongoing cost	Zero compute when idle (each family's batch, VM, or pod exits immediately upon finishing its models).

flowchart LR
    CFG["Single RunConfig
1 Deterministic run_id"] --> DAG["dag.plan_dag()
Per-Family Parallel Router"]
    subgraph Parallel["Concurrent Family Execution (Zero-Idle Compute Across 5 Families)"]
        F1["statistical
theta · holtwinters
(Spark, Vertex, GCE, or GKE CPU)"]
        F2["ml
lightgbm · xgboost
(Vertex or GKE CPU · Two-Tier TreeSHAP)"]
        F3["deep_learning
tide · chronos_bolt
(Vertex, GKE, or Ray CPU/GPU)"]
        F4["automl (Optional)
vertex_tide · vertex_l2l
(Vertex AI AutoML Tabular Workflow)"]
        F5["native
arima_plus · timesfm
(BigQuery Serverless SQL)"]
    end
    DAG --> F1 & F2 & F3 & F4 & F5 --> ENS["ensemble Stage
inverse_error · nnls · ridge · champion"]
    ENS --> LB["v_model_leaderboard + Decision Workbench
FVA · Attributions · Anomalies · Inventory Policy"]

Act 1 · Cloud Bootstrap & Deployment Resolution¶

On a managed cloud notebook (Colab Enterprise, Vertex AI Workbench) the bootstrap cell below clones or updates the repository and installs the locked dependency set (uv.lock) into an isolated .colab-deps prefix so import scale_forecasting resolves against the exact package versions every cloud worker uses. In a local clone with .venv, this cell is a fast no-op.

In [1]:

# Cloud bootstrap: clone + install the LOCKED dependency set (uv.lock) so
# `import scale_forecasting` resolves against the exact versions every other surface runs.
import importlib.util
import os
import shutil
import subprocess
import sys

REPO_URL = os.environ.get("SF_REPO_URL", "https://github.com/statmike/scale-forecasting.git")
REPO_DIR = os.environ.get("SF_REPO_DIR", "scale-forecasting")
EXTRAS = ['gcp']

if importlib.util.find_spec("scale_forecasting") is None:
    if not os.path.isdir(REPO_DIR):
        subprocess.run(["git", "clone", "--depth", "1", REPO_URL, REPO_DIR], check=True)
    uv = shutil.which("uv")
    if uv is None:
        subprocess.run([sys.executable, "-m", "pip", "install", "-q", "uv"], check=True)
        uv = shutil.which("uv") or "uv"
    reqs = os.path.abspath(os.path.join(REPO_DIR, "colab-requirements.txt"))
    extra_flags = [f for e in EXTRAS for f in ("--extra", e)]
    subprocess.run(
        [uv, "export", "--frozen", "--no-emit-project", "--no-hashes", "--no-dev", *extra_flags, "-o", reqs],
        cwd=REPO_DIR,
        check=True,
    )
    target = os.path.abspath(os.path.join(REPO_DIR, ".colab-deps"))
    subprocess.run(
        [uv, "pip", "install", "--python", sys.executable, "--target", target, "-r", reqs],
        check=True,
    )
    sys.path.insert(0, target)
    sys.path.insert(0, os.path.join(REPO_DIR, "src"))
    print("installed scale_forecasting from", REPO_DIR)
else:
    print("scale_forecasting already importable — skipping bootstrap")

# Cloud bootstrap: clone + install the LOCKED dependency set (uv.lock) so # `import scale_forecasting` resolves against the exact versions every other surface runs. import importlib.util import os import shutil import subprocess import sys REPO_URL = os.environ.get("SF_REPO_URL", "https://github.com/statmike/scale-forecasting.git") REPO_DIR = os.environ.get("SF_REPO_DIR", "scale-forecasting") EXTRAS = ['gcp'] if importlib.util.find_spec("scale_forecasting") is None: if not os.path.isdir(REPO_DIR): subprocess.run(["git", "clone", "--depth", "1", REPO_URL, REPO_DIR], check=True) uv = shutil.which("uv") if uv is None: subprocess.run([sys.executable, "-m", "pip", "install", "-q", "uv"], check=True) uv = shutil.which("uv") or "uv" reqs = os.path.abspath(os.path.join(REPO_DIR, "colab-requirements.txt")) extra_flags = [f for e in EXTRAS for f in ("--extra", e)] subprocess.run( [uv, "export", "--frozen", "--no-emit-project", "--no-hashes", "--no-dev", *extra_flags, "-o", reqs], cwd=REPO_DIR, check=True, ) target = os.path.abspath(os.path.join(REPO_DIR, ".colab-deps")) subprocess.run( [uv, "pip", "install", "--python", sys.executable, "--target", target, "-r", reqs], check=True, ) sys.path.insert(0, target) sys.path.insert(0, os.path.join(REPO_DIR, "src")) print("installed scale_forecasting from", REPO_DIR) else: print("scale_forecasting already importable — skipping bootstrap")

scale_forecasting already importable — skipping bootstrap

Resolve the Live GCP Deployment (Settings.resolve())¶

Settings.resolve() reads the SF_* environment variables (SF_PROJECT_ID, SF_REGION, SF_DATASET_ID, SF_WAREHOUSE_URI, SF_CONNECTION, SF_COMPUTE_SA) pre-populated by Terraform in the sf-main runtime template (or your local shell).

In [2]:

from google.cloud import bigquery
from scale_forecasting.settings import Settings

settings = Settings.resolve()
client = bigquery.Client(project=settings.project_id)
DATASET = settings.dataset_ref
print(f"Deployment: {DATASET} | Region: {settings.region} | Bucket: {settings.warehouse_uri}")

from google.cloud import bigquery from scale_forecasting.settings import Settings settings = Settings.resolve() client = bigquery.Client(project=settings.project_id) DATASET = settings.dataset_ref print(f"Deployment: {DATASET} | Region: {settings.region} | Bucket: {settings.warehouse_uri}")

Deployment: statmike-scale-forecasting.scale_forecasting | Region: us-central1 | Bucket: gs://statmike-scale-forecasting-warehouse/warehouse

Act 2 · Configure the Multi-Family Master DAG & Inspect forecaster.explain()¶

Look at how forecaster.explain() decomposes our single RunConfig into independent family execution rows, each with its own runtime (spark, vertex, gce, gke, ray, vertex_automl, or bigquery), training mode (local vs. global), hardware (cpu vs. gpu), and sizing. (When routing families to runtime='gke', shared_clusters.py provisions a single shared GKE cluster with fit-for-purpose CPU and GPU node pools per family and scales down individual GPU nodes as each deep_learning model pod finishes.)

In [3]:

import time
from scale_forecasting import Forecaster, RunConfig

# === Parameters — edit me ===============================================
RUN_NAME = f"nb08 master workflow {int(time.time())}"
SOURCE_TABLE = "source_series_iceberg"
MODELS = ["theta", "holtwinters", "lightgbm", "xgboost", "tide", "chronos_bolt", "arima_plus", "timesfm"]
HORIZON = 28
SERIES_LIMIT = 15
N_FOLDS = 2
ENSEMBLE_STRATEGIES = ["mean", "inverse_error", "nnls", "ridge", "champion"]
# ========================================================================

cfg = RunConfig(
    run_name=RUN_NAME,
    python_runtime="spark",
    data={"source_table": SOURCE_TABLE, "horizon": HORIZON, "series_limit": SERIES_LIMIT},
    models=MODELS,
    model_params={
        "tide": {"training_mode": "global", "max_epochs": 15},
    },
    compute={
        "families": {
            "statistical": {"runtime": "spark"},
            "ml": {"runtime": "vertex", "machine_type": "n2-standard-4", "workers": 1},
            "deep_learning": {"runtime": "vertex", "hardware": "cpu", "machine_type": "n2-standard-4", "workers": 1},
        }
    },
    features={"holidays": ["US"]},
    backtest={"enabled": True, "n_folds": N_FOLDS, "horizon": HORIZON, "step": HORIZON},
    ensemble={"enabled": True, "strategies": ENSEMBLE_STRATEGIES},
)

forecaster = Forecaster(cfg, settings=settings)
print("Master run_id:", forecaster.run_id)
display(forecaster.explain())

# Also inspect the DAG node topology directly via forecaster.dag()
for node in forecaster.dag():
    print(f"  DAG node: {node.family:<15} runtime={node.runtime:<10} hardware={str(node.hardware):<5} models={list(node.models)} depends_on={list(node.depends_on)}")

# Preview how adding an 'automl' model (e.g. vertex_tide) expands the DAG to all 5 families:
five_family_preview = cfg.model_copy(update={"models": [*MODELS, "vertex_tide"]})
print("\n--- 5-Family Execution Plan Preview (adding vertex_tide on vertex_automl) ---")
display(Forecaster(five_family_preview, settings=settings).explain(validate_data=False))

import time from scale_forecasting import Forecaster, RunConfig # === Parameters — edit me =============================================== RUN_NAME = f"nb08 master workflow {int(time.time())}" SOURCE_TABLE = "source_series_iceberg" MODELS = ["theta", "holtwinters", "lightgbm", "xgboost", "tide", "chronos_bolt", "arima_plus", "timesfm"] HORIZON = 28 SERIES_LIMIT = 15 N_FOLDS = 2 ENSEMBLE_STRATEGIES = ["mean", "inverse_error", "nnls", "ridge", "champion"] # ======================================================================== cfg = RunConfig( run_name=RUN_NAME, python_runtime="spark", data={"source_table": SOURCE_TABLE, "horizon": HORIZON, "series_limit": SERIES_LIMIT}, models=MODELS, model_params={ "tide": {"training_mode": "global", "max_epochs": 15}, }, compute={ "families": { "statistical": {"runtime": "spark"}, "ml": {"runtime": "vertex", "machine_type": "n2-standard-4", "workers": 1}, "deep_learning": {"runtime": "vertex", "hardware": "cpu", "machine_type": "n2-standard-4", "workers": 1}, } }, features={"holidays": ["US"]}, backtest={"enabled": True, "n_folds": N_FOLDS, "horizon": HORIZON, "step": HORIZON}, ensemble={"enabled": True, "strategies": ENSEMBLE_STRATEGIES}, ) forecaster = Forecaster(cfg, settings=settings) print("Master run_id:", forecaster.run_id) display(forecaster.explain()) # Also inspect the DAG node topology directly via forecaster.dag() for node in forecaster.dag(): print(f" DAG node: {node.family:<15} runtime={node.runtime:<10} hardware={str(node.hardware):<5} models={list(node.models)} depends_on={list(node.depends_on)}") # Preview how adding an 'automl' model (e.g. vertex_tide) expands the DAG to all 5 families: five_family_preview = cfg.model_copy(update={"models": [*MODELS, "vertex_tide"]}) print("\n--- 5-Family Execution Plan Preview (adding vertex_tide on vertex_automl) ---") display(Forecaster(five_family_preview, settings=settings).explain(validate_data=False))

Master run_id: nb08-master-workflow-1792005806-e997855397a2

	run_id	family	job_key	runtime	machine_type	workers	hardware	gpu_type	models	training_modes	n_series	n_folds	expected_cells	covariates	depends_on
0	nb08-master-workflow-1792005806-e997855397a2	statistical	sf-nb08-master-workflow-1792005806-e997855397a...	spark	None	NaN	cpu	None	theta, holtwinters	theta:local, holtwinters:local	15	2	30	none
1	nb08-master-workflow-1792005806-e997855397a2	ml	sf-nb08-master-workflow-1792005806-e997855397a...	vertex	n2-standard-4	1.0	cpu	None	lightgbm, xgboost	lightgbm:local, xgboost:local	15	2	30	none
2	nb08-master-workflow-1792005806-e997855397a2	deep_learning	sf-nb08-master-workflow-1792005806-e997855397a...	vertex	n2-standard-4	2.0	cpu	None	tide, chronos_bolt	tide:global, chronos_bolt:local	15	2	30	none
3	nb08-master-workflow-1792005806-e997855397a2	native	sf-nb08-master-workflow-1792005806-e997855397a...	bigquery	sql	1.0	sql	None	arima_plus, timesfm	arima_plus:local, timesfm:local	15	2	30	none
4	nb08-master-workflow-1792005806-e997855397a2	ensemble	sf-nb08-master-workflow-1792005806-e997855397a...	bigquery	sql	1.0	sql	None	ensemble_mean, ensemble_inverse_error, ensembl...	barrier	15	2	75	none	sf-nb08-master-workflow-1792005806-e997855397a...

  DAG node: statistical     runtime=spark      hardware=cpu   models=['theta', 'holtwinters'] depends_on=[]
  DAG node: ml              runtime=vertex     hardware=cpu   models=['lightgbm', 'xgboost'] depends_on=[]
  DAG node: deep_learning   runtime=vertex     hardware=cpu   models=['tide', 'chronos_bolt'] depends_on=[]
  DAG node: native          runtime=bigquery   hardware=None  models=['arima_plus', 'timesfm'] depends_on=[]
  DAG node: ensemble        runtime=bigquery   hardware=None  models=[] depends_on=['sf-nb08-master-workflow-1792005806-e997855397a2-statistical-a1', 'sf-nb08-master-workflow-1792005806-e997855397a2-ml-a1', 'sf-nb08-master-workflow-1792005806-e997855397a2-deep_learning-a1', 'sf-nb08-master-workflow-1792005806-e997855397a2-native-a1']

--- 5-Family Execution Plan Preview (adding vertex_tide on vertex_automl) ---

	run_id	family	job_key	runtime	machine_type	workers	hardware	gpu_type	models	training_modes	n_series	n_folds	expected_cells	covariates	depends_on
0	nb08-master-workflow-1792005806-99ec064b6593	statistical	sf-nb08-master-workflow-1792005806-99ec064b659...	spark	None	NaN	cpu	None	theta, holtwinters	theta:local, holtwinters:local	15	2	30	none
1	nb08-master-workflow-1792005806-99ec064b6593	ml	sf-nb08-master-workflow-1792005806-99ec064b659...	vertex	n2-standard-4	1.0	cpu	None	lightgbm, xgboost	lightgbm:local, xgboost:local	15	2	30	none
2	nb08-master-workflow-1792005806-99ec064b6593	deep_learning	sf-nb08-master-workflow-1792005806-99ec064b659...	vertex	n2-standard-4	2.0	cpu	None	tide, chronos_bolt	tide:global, chronos_bolt:local	15	2	30	none
3	nb08-master-workflow-1792005806-99ec064b6593	automl	sf-nb08-master-workflow-1792005806-99ec064b659...	vertex_automl	None	NaN	cpu	None	vertex_tide	vertex_tide:global	15	2	15	none
4	nb08-master-workflow-1792005806-99ec064b6593	native	sf-nb08-master-workflow-1792005806-99ec064b659...	bigquery	sql	1.0	sql	None	arima_plus, timesfm	arima_plus:local, timesfm:local	15	2	30	none
5	nb08-master-workflow-1792005806-99ec064b6593	ensemble	sf-nb08-master-workflow-1792005806-99ec064b659...	bigquery	sql	1.0	sql	None	ensemble_mean, ensemble_inverse_error, ensembl...	barrier	15	2	75	none	sf-nb08-master-workflow-1792005806-99ec064b659...

Act 3 · Launch & Monitor All Families Concurrently (forecaster.run_live())¶

forecaster.run_live() launches the parallel family threads, displays real-time per-family progress bars, escalates to live platform probes (probe=True) if any engine is quiet for >300s, and runs the final ensemble join.

In [4]:

result = forecaster.run_live(poll_s=15.0, probe=True)
print(f"Master workflow {result.run_id} finished: status={result.status}, wall_clock={result.runtime_seconds:.1f}s")

result = forecaster.run_live(poll_s=15.0, probe=True) print(f"Master workflow {result.run_id} finished: status={result.status}, wall_clock={result.runtime_seconds:.1f}s")

Master workflow nb08-master-workflow-1792005806-e997855397a2 finished: status=COMPLETED, wall_clock=1141.9s

Act 4 · Comprehensive Post-Run Review (Leaderboard, FVA, Cohorts, Provisioning Telemetry, Calibration, Ensemble Weights, Two-Tier Feature Attributions, Anomalies, Change Points & Inventory Policy)¶

Let's run the complete diagnostic & decision workbench suite across all active model families and the stacked ensembles:

1. leaderboard_df(), plot_leaderboard(), plot_metric_distribution(), and fva_df() (Forecast Value Added & Diebold–Mariano $p$-values)
2. cohorts_df() and jobs_df() (including provision_seconds vs. execute_seconds and launcher_version / worker_version parity)
3. plot_calibration() and plot_ensemble_weights()
4. Two-Tier Feature Attributions: attributions_df() and plot_attributions() (Tier 1 global feature importance + Tier 2 per-horizon-step TreeSHAP attributions)
5. plot_forecasts(), plot_forecast_explanation(), and plot_trace()
6. Universal Anomaly & Structural Change-Point Detection (detect_anomalies_df() & detect_change_points_df())
7. Operational Inventory & Replenishment Policy (inventory_policy_df() & plot_inventory_policy())

In [5]:

import matplotlib.pyplot as plt
from scale_forecasting import (
    plot_leaderboard,
    plot_metric_distribution,
    plot_trace,
)

display(forecaster.leaderboard_df())

review = forecaster.review_run()
plot_leaderboard(review)
plt.show()

plot_metric_distribution(review)
plt.show()

print("--- Forecast Value Added (fva_df) ---")
display(forecaster.fva_df())

display(forecaster.cohorts_df())
display(forecaster.jobs_df())

forecaster.plot_calibration()
plt.show()

forecaster.plot_ensemble_weights()
plt.show()

print("--- Two-Tier Feature Attributions (forecaster.attributions_df & plot_attributions) ---")
attr_df = forecaster.attributions_df(model_type="xgboost", level="global")
if not attr_df.empty:
    display(attr_df.head(10))
    forecaster.plot_attributions(model_type="xgboost", level="global", title="Tier 1 Global TreeSHAP Importance (xgboost)")
    plt.show()
    forecaster.plot_attributions(model_type="xgboost", level="local", title="Tier 2 Local Per-Step TreeSHAP Attributions (xgboost)")
    plt.show()

forecaster.plot_forecasts(history_tail=120)
plt.show()

forecaster.plot_forecast_explanation(history_tail=120)
plt.show()

trace_df = forecaster.trace()
display(trace_df)
plot_trace(trace_df)
plt.show()

print("--- Universal Anomaly & Structural Change-Point Detection (ensemble_nnls) ---")
display(forecaster.detect_anomalies_df(model_name="ensemble_nnls", anomalies_only=True).head(10))
display(forecaster.detect_change_points_df(model_name="ensemble_nnls").head(10))

print("--- Operational Inventory & Replenishment Policy (Continuous Review (s, S) @ 95% CSL) ---")
inv_df = forecaster.inventory_policy_df(
    model_name="ensemble_nnls",
    lead_time=7.0,
    service_level=0.95,
    on_hand=150.0,
    holding_cost=1.0,
    stockout_cost=9.0,
)
display(
    inv_df[
        [
            "ts_id",
            "model_type",
            "lead_time_demand",
            "error_autocorr_rho",
            "safety_stock",
            "reorder_point",
            "recommended_order_qty",
            "expected_total_cost",
        ]
    ].head(10)
)
forecaster.plot_inventory_policy(
    model_name="ensemble_nnls",
    lead_time=7.0,
    service_level=0.95,
    on_hand=150.0,
    max_rows=10,
)
plt.show()

import matplotlib.pyplot as plt from scale_forecasting import ( plot_leaderboard, plot_metric_distribution, plot_trace, ) display(forecaster.leaderboard_df()) review = forecaster.review_run() plot_leaderboard(review) plt.show() plot_metric_distribution(review) plt.show() print("--- Forecast Value Added (fva_df) ---") display(forecaster.fva_df()) display(forecaster.cohorts_df()) display(forecaster.jobs_df()) forecaster.plot_calibration() plt.show() forecaster.plot_ensemble_weights() plt.show() print("--- Two-Tier Feature Attributions (forecaster.attributions_df & plot_attributions) ---") attr_df = forecaster.attributions_df(model_type="xgboost", level="global") if not attr_df.empty: display(attr_df.head(10)) forecaster.plot_attributions(model_type="xgboost", level="global", title="Tier 1 Global TreeSHAP Importance (xgboost)") plt.show() forecaster.plot_attributions(model_type="xgboost", level="local", title="Tier 2 Local Per-Step TreeSHAP Attributions (xgboost)") plt.show() forecaster.plot_forecasts(history_tail=120) plt.show() forecaster.plot_forecast_explanation(history_tail=120) plt.show() trace_df = forecaster.trace() display(trace_df) plot_trace(trace_df) plt.show() print("--- Universal Anomaly & Structural Change-Point Detection (ensemble_nnls) ---") display(forecaster.detect_anomalies_df(model_name="ensemble_nnls", anomalies_only=True).head(10)) display(forecaster.detect_change_points_df(model_name="ensemble_nnls").head(10)) print("--- Operational Inventory & Replenishment Policy (Continuous Review (s, S) @ 95% CSL) ---") inv_df = forecaster.inventory_policy_df( model_name="ensemble_nnls", lead_time=7.0, service_level=0.95, on_hand=150.0, holding_cost=1.0, stockout_cost=9.0, ) display( inv_df[ [ "ts_id", "model_type", "lead_time_demand", "error_autocorr_rho", "safety_stock", "reorder_point", "recommended_order_qty", "expected_total_cost", ] ].head(10) ) forecaster.plot_inventory_policy( model_name="ensemble_nnls", lead_time=7.0, service_level=0.95, on_hand=150.0, max_rows=10, ) plt.show()

	rank	model_type	base_model	family	scope_capability	training_mode	cov_profile	is_univariate	covariates_used	model_spec_hash	...	mean_mae	mean_rmse	mean_coverage	mean_interval_score	mean_fit_seconds	median_fit_seconds	no_artifact_rate	n_predictions	lift_vs_best_base	lift_pct
0	1	ensemble_champion	ensemble_champion	ensemble	None	None	None	None	None	None	...	9.910560	13.698536	0.680952	52.391319	NaN	NaN	1.0	420	0.006832	0.018908
1	2	timesfm	timesfm	native	None	None	None	None	None	None	...	11.005670	15.294330	0.761905	50.668533	NaN	NaN	1.0	420	NaN	NaN
2	3	ensemble_ridge	ensemble_ridge	ensemble	None	None	None	None	None	None	...	11.007550	15.129316	0.682143	56.190352	NaN	NaN	0.0	420	-0.020416	-0.056505
3	4	ensemble_nnls	ensemble_nnls	ensemble	None	None	None	None	None	None	...	11.039538	15.034472	0.697619	60.387998	NaN	NaN	0.0	420	-0.024919	-0.068968
4	5	holtwinters	holtwinters	statistical	None	None	None	None	None	None	...	12.934460	17.531664	0.698810	65.546850	2.945893	1.160322	1.0	420	NaN	NaN
5	6	ensemble_inverse_error	ensemble_inverse_error	ensemble	None	None	None	None	None	None	...	11.039874	14.959812	0.675000	63.562790	NaN	NaN	1.0	420	-0.026689	-0.073867
6	7	chronos_bolt	chronos_bolt	deep_learning	None	None	None	None	None	None	...	12.646913	16.922808	0.736905	57.914674	2.346027	0.953497	1.0	420	NaN	NaN
7	8	ensemble_mean	ensemble_mean	ensemble	None	None	None	None	None	None	...	11.978062	16.051136	0.648810	69.446867	NaN	NaN	1.0	420	-0.040236	-0.111364
8	9	arima_plus	arima_plus	native	None	None	None	None	None	None	...	11.596456	15.484153	0.617857	56.040234	NaN	NaN	1.0	420	NaN	NaN
9	10	lightgbm	lightgbm	ml	None	None	None	None	None	None	...	13.821479	18.199157	0.282143	99.018758	3.772185	3.541315	1.0	420	NaN	NaN
10	11	theta	theta	statistical	None	None	None	None	None	None	...	14.614160	18.970049	0.995238	262.555296	4.102680	1.434090	1.0	420	NaN	NaN
11	12	xgboost	xgboost	ml	None	None	None	None	None	None	...	14.319092	19.378332	0.210714	116.923408	5.469811	5.914349	1.0	420	NaN	NaN
12	13	tide	tide	deep_learning	multi_scope	global	cov_none	True		d4804fdf4a4fc1cf	...	17.900452	22.432844	0.771429	90.109112	1.200056	1.200056	1.0	420	NaN	NaN

13 rows × 34 columns

--- Forecast Value Added (fva_df) ---

	fva_stage_rank	fva_stage	model_type	base_model	demand_class	n_series	n_points	metric	mean_metric	pooled_wape	...	univariate_baseline_model	covariate_fva_abs	covariate_fva_pct	covariate_p_value_dm	covariate_significant_at_05	local_baseline_model	pooling_fva_abs	pooling_fva_pct	pooling_p_value_dm	pooling_significant_at_05
0	1	Baseline	arima_plus	arima_plus	all	15	840	wape	0.403000	0.135709	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
1	2	Statistical / SQL	timesfm	timesfm	all	15	840	wape	0.352233	0.128795	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
2	2	Statistical	holtwinters	holtwinters	all	15	840	wape	0.381727	0.151367	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
3	2	Statistical	theta	theta	all	15	840	wape	0.432903	0.171024	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
4	3	Machine Learning	lightgbm	lightgbm	all	15	840	wape	0.433242	0.161747	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
5	3	Machine Learning	xgboost	xgboost	all	15	840	wape	0.435188	0.167571	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
6	4	Deep Learning	chronos_bolt	chronos_bolt	all	15	840	wape	0.381411	0.148002	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
7	4	Deep Learning	tide	tide	all	15	840	wape	0.476087	0.209482	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
8	6	Ensemble	ensemble_champion	ensemble_champion	all	15	840	wape	0.344703	0.115979	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
9	6	Ensemble	ensemble_ridge	ensemble_ridge	all	15	840	wape	0.374328	0.128817	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
10	6	Ensemble	ensemble_nnls	ensemble_nnls	all	15	840	wape	0.376687	0.129191	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
11	6	Ensemble	ensemble_inverse_error	ensemble_inverse_error	all	15	840	wape	0.377716	0.129195	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False
12	6	Ensemble	ensemble_mean	ensemble_mean	all	15	840	wape	0.390780	0.140174	...	None	NaN	NaN	NaN	False	None	NaN	NaN	NaN	False

13 rows × 30 columns

	model_type	family	is_ensemble	n_series	score	pooled_wape	n_comparable_series	n_full	n_reduced	n_unscored	n_failed	n_not_requested	fold_histogram	refit_modes	staleness_gap
0	ensemble_champion	ensemble	True	15	0.354474	0.111546	15	0	0	0	0	15	none	per_fold:15	None
1	timesfm	native	False	15	0.361305	0.114332	15	15	0	0	0	0	2f:15	per_fold:15	None
2	ensemble_ridge	ensemble	True	15	0.381721	0.121881	15	0	0	0	0	15	none	per_fold:15	None
3	ensemble_nnls	ensemble	True	15	0.386224	0.116385	15	0	0	0	0	15	none	per_fold:15	None
4	holtwinters	statistical	False	15	0.387183	0.135680	15	15	0	0	0	0	2f:15	per_fold:15	None
5	ensemble_inverse_error	ensemble	True	15	0.387994	0.117580	15	0	0	0	0	15	none	per_fold:15	None
6	chronos_bolt	deep_learning	False	15	0.397640	0.126006	15	15	0	0	0	0	2f:15	per_fold:15	None
7	ensemble_mean	ensemble	True	15	0.401542	0.123605	15	0	0	0	0	15	none	per_fold:15	None
8	arima_plus	native	False	15	0.413395	0.125555	15	15	0	0	0	0	2f:15	per_fold:15	None
9	lightgbm	ml	False	15	0.441840	0.151010	15	15	0	0	0	0	2f:15	per_fold:15	None
10	theta	statistical	False	15	0.443389	0.158292	15	15	0	0	0	0	2f:15	per_fold:15	None
11	xgboost	ml	False	15	0.448197	0.139390	15	15	0	0	0	0	2f:15	per_fold:15	None
12	tide	deep_learning	False	15	0.489635	0.199350	15	15	0	0	0	0	2f:15	per_fold:15	None

	family	job_key	system_job_id	runtime	status	attempt	hardware	gpu_type	spark_mode	runtime_seconds
0	deep_learning	sf-nb08-master-workflow-1792005806-e997855397a...	projects/307701787156/locations/us-central1/cu...	vertex	COMPLETED	1	cpu	None	None	605.298630
1	ensemble	sf-nb08-master-workflow-1792005806-e997855397a...	sf-nb08-master-workflow-1792005806-e997855397a...	bigquery	COMPLETED	1	None	None	None	34.258376
2	ml	sf-nb08-master-workflow-1792005806-e997855397a...	projects/307701787156/locations/us-central1/cu...	vertex	COMPLETED	1	cpu	None	None	930.308667
3	native	sf-nb08-master-workflow-1792005806-e997855397a...	sf-nb08-master-workflow-1792005806-e997855397a...	bigquery	COMPLETED	1	None	None	None	104.410528
4	statistical	sf-nb08-master-workflow-1792005806-e997855397a...	sf-nb08-master-workflow-1792005806-e997855397a...	spark	COMPLETED	1	cpu	None	serverless	1041.232298

--- Two-Tier Feature Attributions (forecaster.attributions_df & plot_attributions) ---

	ts_id	model_type	compute_engine	feature	importance
0	s_000000	xgboost	vertex	lag_7	0.739868
1	s_000000	xgboost	vertex	lag_14	0.139584
2	s_000000	xgboost	vertex	is_holiday	0.048060
3	s_000000	xgboost	vertex	doy	0.018106
4	s_000000	xgboost	vertex	month	0.017481
5	s_000000	xgboost	vertex	dow	0.012837
6	s_000000	xgboost	vertex	lag_28	0.007346
7	s_000000	xgboost	vertex	lag_1	0.006607
8	s_000000	xgboost	vertex	lag_3	0.003971
9	s_000000	xgboost	vertex	dom	0.003362

	kind	lane	label	start	end	duration_s	status	runtime	model_type	ts_id
0	job	deep_learning	deep_learning	2026-10-14 19:23:56.165027+00:00	2026-10-14 19:34:04.286297+00:00	608.121270	COMPLETED	vertex	None	None
1	job	ensemble	ensemble	2026-10-14 19:41:27.872693+00:00	2026-10-14 19:42:07.236214+00:00	39.363521	COMPLETED	bigquery	None	None
2	job	ml	ml	2026-10-14 19:23:56.055640+00:00	2026-10-14 19:39:29.432988+00:00	933.377348	COMPLETED	vertex	None	None
3	job	native	native	2026-10-14 19:23:56.075212+00:00	2026-10-14 19:25:43.296497+00:00	107.221285	COMPLETED	bigquery	None	None
4	job	statistical	statistical	2026-10-14 19:23:56.117744+00:00	2026-10-14 19:41:19.864800+00:00	1043.747056	COMPLETED	spark	None	None
...	...	...	...	...	...	...	...	...	...	...
90	cell	gdpic-srvls-batch-34a28466-b2b4-4e78-bca9-8565...	theta:s_000006	2026-10-14 19:38:49.021501+00:00	2026-10-14 19:38:51.565893+00:00	2.544392	None	spark	theta	s_000006
91	cell	gdpic-srvls-batch-34a28466-b2b4-4e78-bca9-8565...	holtwinters:s_000014	2026-10-14 19:38:51.160778+00:00	2026-10-14 19:38:52.489405+00:00	1.328627	None	spark	holtwinters	s_000014
92	cell	gdpic-srvls-batch-34a28466-b2b4-4e78-bca9-8565...	holtwinters:s_000003	2026-10-14 19:38:52.490370+00:00	2026-10-14 19:38:53.569617+00:00	1.079247	None	spark	holtwinters	s_000003
93	cell	gdpic-srvls-batch-34a28466-b2b4-4e78-bca9-8565...	holtwinters:s_000007	2026-10-14 19:38:53.570444+00:00	2026-10-14 19:38:54.563601+00:00	0.993157	None	spark	holtwinters	s_000007
94	cell	gdpic-srvls-batch-34a28466-b2b4-4e78-bca9-8565...	holtwinters:s_000000	2026-10-14 19:38:54.564600+00:00	2026-10-14 19:38:55.711098+00:00	1.146498	None	spark	holtwinters	s_000000

95 rows × 10 columns

--- Universal Anomaly & Structural Change-Point Detection (ensemble_nnls) ---

	ts_id	model_type	ds	y	yhat	is_anomaly	anomaly_probability	lower_bound	upper_bound
0	s_000000	ensemble_nnls	2024-11-11	216.978	189.744888	True	0.994167	170.384864	209.104913
1	s_000000	ensemble_nnls	2024-12-25	204.093	175.845783	True	0.995759	156.485758	195.205807
2	s_000001	ensemble_nnls	2024-11-26	17.494	1.420978	True	0.999989	-5.747598	8.589555
3	s_000001	ensemble_nnls	2024-12-18	8.790	1.380567	True	0.957216	-5.788009	8.549143
4	s_000001	ensemble_nnls	2024-12-29	9.121	1.638665	True	0.959219	-5.529912	8.807241
5	s_000002	ensemble_nnls	2024-12-12	172.680	153.399286	True	0.952744	134.352116	172.446456
6	s_000002	ensemble_nnls	2024-12-25	178.345	154.564380	True	0.985596	135.517210	173.611550
7	s_000002	ensemble_nnls	2024-12-28	205.475	182.051491	True	0.984060	163.004321	201.098660
8	s_000002	ensemble_nnls	2024-12-29	214.760	187.032046	True	0.995672	167.984876	206.079216
9	s_000003	ensemble_nnls	2024-11-25	77.486	119.505885	True	0.988916	87.082387	151.929382

	ts_id	model_type	begin_date	end_date	window_count	window_avg	window_min	window_max	window_stddev	gap_edge_missing_steps	is_gap_edge	status
0	s_000002	ensemble_nnls	2024-12-02	2024-12-08	7	170.627571	150.312	188.021	14.580563	0	False
1	s_000002	ensemble_nnls	2024-12-19	2024-12-25	7	178.292714	157.452	204.105	15.098823	0	False
2	s_000003	ensemble_nnls	2024-11-16	2024-11-22	7	71.684714	58.930	85.839	9.627494	0	False
3	s_000003	ensemble_nnls	2024-11-30	2024-12-06	7	80.028429	71.174	93.643	8.161444	0	False
4	s_000003	ensemble_nnls	2024-12-21	2024-12-27	7	89.104286	84.609	92.511	2.615408	0	False
5	s_000004	ensemble_nnls	2024-11-14	2024-11-20	7	67.413000	15.531	109.185	37.721467	0	False
6	s_000004	ensemble_nnls	2024-11-21	2024-11-27	7	86.360571	17.381	151.681	51.084085	0	False
7	s_000004	ensemble_nnls	2024-12-01	2024-12-07	7	40.787429	0.000	88.440	39.625923	0	False
8	s_000004	ensemble_nnls	2024-12-18	2024-12-24	7	80.608429	10.730	150.836	54.203511	0	False
9	s_000006	ensemble_nnls	2024-11-09	2024-11-15	7	1.408571	0.000	3.587	1.764598	0	False

--- Operational Inventory & Replenishment Policy (Continuous Review (s, S) @ 95% CSL) ---

	ts_id	model_type	lead_time_demand	error_autocorr_rho	safety_stock	reorder_point	recommended_order_qty	expected_total_cost
0	s_000000	ensemble_nnls	1488.009142	0.128558	85.870706	1573.879848	1423.879848	96.778017
1	s_000001	ensemble_nnls	11.562551	0.146052	25.330190	36.892741	0.000000	28.547635
2	s_000002	ensemble_nnls	1250.672051	0.548822	93.845595	1344.517646	1194.517646	105.765878
3	s_000003	ensemble_nnls	611.436321	0.572329	101.813658	713.249979	563.249979	114.746047
4	s_000004	ensemble_nnls	394.701115	0.645650	338.841210	733.542325	583.542325	381.880879
5	s_000005	ensemble_nnls	1744.746765	0.205336	44.041540	1788.788305	1638.788305	49.635703
6	s_000006	ensemble_nnls	7.992938	0.053293	10.896748	18.889686	0.000000	12.280855
7	s_000007	ensemble_nnls	1103.165184	0.335967	112.424026	1215.589210	1065.589210	126.704145
8	s_000008	ensemble_nnls	299.902609	-0.047925	258.942785	558.845395	408.845395	291.833743
9	s_000009	ensemble_nnls	471.285449	0.780054	734.378997	1205.664446	1055.664446	827.659944

---

Where next¶

• Next in Track 4 (Master Workflow & Operations): 10_registry_operations_and_scale.ipynb — Day-2 registry health (doctor), safe repair (retry), all 5 launch pathways (including Cloud Composer 3 DAGs), and cross-platform run review.
• Extend the platform: 09_custom_models_and_metrics.ipynb — add custom 1-file models and metrics.
• Documentation: Choosing a runtime · Cost estimates & controls · Validation ledger