Run in Colab Enterprise

Run in Colab Enterprise: click the badge to import this notebook, pick a runtime, and Run all. The Terraform-deployed templates already carry the SF_* run identity in their env, so there's no environment cell to fill in. Runs on the sf-main runtime template (Python 3.11). See docs/notebook_runtimes.md for the per-notebook template mapping and the headless acceptance harness.

02 · Serverless Single-VM, Vertex AI & GKE Indexed Jobs (gce, vertex & gke Runtimes)¶

For workloads that do not require an Apache Spark or Ray cluster coordinator, scale-forecasting provides three direct container execution runtimes powered by the shared engines/vertex_engine.py kernel:

1. GCE Ephemeral Single-VM (runtime = 'gce'): Boots a single Container-Optimized OS (COS) VM with local ThreadPoolExecutor LPT concurrency and triple-redundant zero-orphan cleanup (maxRunDuration + instanceTerminationAction='DELETE', guest trap cleanup EXIT self-delete, and launcher try...finally deletion).
2. Vertex AI CustomJob (runtime = 'vertex'): Launches 1 to $N$ serverless worker replicas (workerPoolSpecs). When workers > 1, each replica pushes its contiguous [start_id, end_id] boundary directly into the BigQuery Storage Read API (row_restriction) and schedules chunks longest-first (Longest Processing Time / LPT).
3. Google Kubernetes Engine Indexed Jobs (runtime = 'gke', gke_mode = 'job'): Launches workers = 1..N pods (JOB_COMPLETION_INDEX) on fit-for-purpose per-family CPU or GPU node pools. When multiple deep_learning models run together, workers = 1 auto-expands to len(models) (1 dedicated GPU pod per model), and each orchestrated pod exits independently as soon as its model finishes so the GKE Cluster Autoscaler (minNodeCount = 1, maxNodeCount = K) scales down completed GPU nodes immediately.

At a glance¶

Dimension	Details
Learning outcomes	Execute Python statistical, tree, and GPU deep-learning models across self-deleting Compute Engine single VMs (gce), multi-worker Vertex AI CustomJob (vertex), and GKE Indexed Jobs (gke).
Prerequisites	Deployed scale-forecasting Google Cloud project (SF_* env vars) + scale-forecasting[gcp].
Estimated time	~6–12 minutes (includes container startup and VM/pod provisioning).
Estimated run cost	Estimated ~\$0.10–\$0.60 USD (Compute Engine / Vertex AI Training / GKE vCPU-hours and optional L4/T4 GPU-hours + BigQuery Storage Read/Write API).
Ongoing cost	Zero compute when idle for vertex, gce (3-way self-deletion), and ephemeral GKE clusters. If you enabled a standing GKE cluster (create_gke = true), node pools scale to 0 at rest while the GKE cluster management fee applies until create_gke = false.

flowchart LR
    subgraph Router["dag.py · resources/sizing.py"]
        CFG["RunConfig
python_runtime = vertex, gce, or gke"] --> SZ["Auto-Sizing Catalog
n2-standard-* · g2-standard-* (L4)
a2-highgpu-* (A100)"]
    end
    subgraph Vertex["Vertex AI CustomJob & GKE Indexed Job (workers >= 1)"]
        W0["Worker / Pod 0
BQ Read row_restriction [id_0..id_k]"]
        W1["Worker / Pod 1..N-1
BQ Read row_restriction [id_k+1..id_M]
Independent Pod Exit & Node Scale-Down on GKE"]
    end
    subgraph GCE["GCE Single-VM (workers == 1)"]
        VM["COS VM + LPT ThreadPool
Triple-Redundant Auto-Delete"]
    end
    SZ --> W0 & W1 & VM --> BQ["BigQuery Storage Write API
forecast_metadata · forecast_predictions"]

Act 1 · Cloud Bootstrap & Deployment Resolution¶

On a managed cloud notebook (Colab Enterprise, Vertex AI Workbench) the bootstrap cell below clones or updates the repository and installs the locked dependency set (uv.lock) into an isolated .colab-deps prefix so import scale_forecasting resolves against the exact package versions every cloud worker uses. In a local clone with .venv, this cell is a fast no-op.

In [1]:

# Cloud bootstrap: clone + install the LOCKED dependency set (uv.lock) so
# `import scale_forecasting` resolves against the exact versions every other surface runs.
import importlib.util
import os
import shutil
import subprocess
import sys

REPO_URL = os.environ.get("SF_REPO_URL", "https://github.com/statmike/scale-forecasting.git")
REPO_DIR = os.environ.get("SF_REPO_DIR", "scale-forecasting")
EXTRAS = ['gcp']

if importlib.util.find_spec("scale_forecasting") is None:
    if not os.path.isdir(REPO_DIR):
        subprocess.run(["git", "clone", "--depth", "1", REPO_URL, REPO_DIR], check=True)
    uv = shutil.which("uv")
    if uv is None:
        subprocess.run([sys.executable, "-m", "pip", "install", "-q", "uv"], check=True)
        uv = shutil.which("uv") or "uv"
    reqs = os.path.abspath(os.path.join(REPO_DIR, "colab-requirements.txt"))
    extra_flags = [f for e in EXTRAS for f in ("--extra", e)]
    subprocess.run(
        [uv, "export", "--frozen", "--no-emit-project", "--no-hashes", "--no-dev", *extra_flags, "-o", reqs],
        cwd=REPO_DIR,
        check=True,
    )
    target = os.path.abspath(os.path.join(REPO_DIR, ".colab-deps"))
    subprocess.run(
        [uv, "pip", "install", "--python", sys.executable, "--target", target, "-r", reqs],
        check=True,
    )
    sys.path.insert(0, target)
    sys.path.insert(0, os.path.join(REPO_DIR, "src"))
    print("installed scale_forecasting from", REPO_DIR)
else:
    print("scale_forecasting already importable — skipping bootstrap")

# Cloud bootstrap: clone + install the LOCKED dependency set (uv.lock) so # `import scale_forecasting` resolves against the exact versions every other surface runs. import importlib.util import os import shutil import subprocess import sys REPO_URL = os.environ.get("SF_REPO_URL", "https://github.com/statmike/scale-forecasting.git") REPO_DIR = os.environ.get("SF_REPO_DIR", "scale-forecasting") EXTRAS = ['gcp'] if importlib.util.find_spec("scale_forecasting") is None: if not os.path.isdir(REPO_DIR): subprocess.run(["git", "clone", "--depth", "1", REPO_URL, REPO_DIR], check=True) uv = shutil.which("uv") if uv is None: subprocess.run([sys.executable, "-m", "pip", "install", "-q", "uv"], check=True) uv = shutil.which("uv") or "uv" reqs = os.path.abspath(os.path.join(REPO_DIR, "colab-requirements.txt")) extra_flags = [f for e in EXTRAS for f in ("--extra", e)] subprocess.run( [uv, "export", "--frozen", "--no-emit-project", "--no-hashes", "--no-dev", *extra_flags, "-o", reqs], cwd=REPO_DIR, check=True, ) target = os.path.abspath(os.path.join(REPO_DIR, ".colab-deps")) subprocess.run( [uv, "pip", "install", "--python", sys.executable, "--target", target, "-r", reqs], check=True, ) sys.path.insert(0, target) sys.path.insert(0, os.path.join(REPO_DIR, "src")) print("installed scale_forecasting from", REPO_DIR) else: print("scale_forecasting already importable — skipping bootstrap")

scale_forecasting already importable — skipping bootstrap

Resolve the Live GCP Deployment (Settings.resolve())¶

Settings.resolve() reads the SF_* environment variables (SF_PROJECT_ID, SF_REGION, SF_DATASET_ID, SF_WAREHOUSE_URI, SF_CONNECTION, SF_COMPUTE_SA) pre-populated by Terraform in the sf-main runtime template (or your local shell).

In [2]:

from google.cloud import bigquery
from scale_forecasting.settings import Settings

settings = Settings.resolve()
client = bigquery.Client(project=settings.project_id)
DATASET = settings.dataset_ref
print(f"Deployment: {DATASET} | Region: {settings.region} | Bucket: {settings.warehouse_uri}")

from google.cloud import bigquery from scale_forecasting.settings import Settings settings = Settings.resolve() client = bigquery.Client(project=settings.project_id) DATASET = settings.dataset_ref print(f"Deployment: {DATASET} | Region: {settings.region} | Bucket: {settings.warehouse_uri}")

Deployment: statmike-scale-forecasting.scale_forecasting | Region: us-central1 | Bucket: gs://statmike-scale-forecasting-warehouse/warehouse

Act 2 · Compare gce Single-VM vs. vertex & gke Multi-Worker Plans with explain()¶

Before launching our live run, let's use Forecaster.explain(validate_data=False) to compare how the resource catalog (resources/catalog.py & resources/sizing.py) sizes:

• A GCE Single-VM run (python_runtime='gce')
• A Vertex AI CustomJob or GKE Indexed Job (python_runtime='vertex' or python_runtime='gke', compute={'gke_mode': 'job'}) mixing CPU families (statistical, ml) with a GPU deep_learning family (hardware='gpu', gpu_type='L4'), where multi-model deep_learning (['tide', 'tsmixer']) automatically assigns 1 dedicated GPU VM/pod per model (workers: 1 -> 2).

In [3]:

import time
from scale_forecasting import Forecaster, RunConfig

gce_preview = RunConfig(
    run_name="preview-gce-single-vm",
    python_runtime="gce",
    data={"source_table": "source_series_iceberg", "horizon": 28, "series_limit": 200},
    models=["theta", "holtwinters", "lightgbm"],
    compute={"machine_type": "n2-standard-8", "workers": 1},
)
print("--- GCE Single-VM Plan ---")
display(Forecaster(gce_preview, settings=settings).explain(validate_data=False))

vertex_gpu_preview = RunConfig(
    run_name="preview-vertex-multi-family",
    python_runtime="vertex",
    data={"source_table": "source_series_iceberg", "horizon": 28, "series_limit": 200},
    models=["theta", "xgboost", "tide", "tsmixer"],
    compute={
        "machine_type": "n2-standard-8",
        "workers": 2,
        "families": {
            "deep_learning": {
                "runtime": "vertex",
                "hardware": "gpu",
                "gpu_type": "L4",
                "accelerator_count": 1,
                "workers": 2,
            }
        },
    },
)
print("--- Vertex AI Multi-Worker CPU + L4 GPU Plan ---")
display(Forecaster(vertex_gpu_preview, settings=settings).explain(validate_data=False))

gke_gpu_preview = RunConfig(
    run_name="preview-gke-multi-family",
    python_runtime="gke",
    data={"source_table": "source_series_iceberg", "horizon": 28, "series_limit": 200},
    models=["theta", "xgboost", "tide", "tsmixer"],
    compute={
        "gke_mode": "job",
        "machine_type": "n2-standard-8",
        "workers": 2,
        "families": {
            "deep_learning": {
                "runtime": "gke",
                "gke_mode": "job",
                "hardware": "gpu",
                "gpu_type": "L4",
                "accelerator_count": 1,
                "workers": 1,  # auto-expands 1 -> 2 dedicated GPU pods (1 per model)
            }
        },
    },
)
print("--- GKE Indexed Job CPU + L4 GPU Plan (Auto-Expanded 1 -> 2 GPU Pods) ---")
display(Forecaster(gke_gpu_preview, settings=settings).explain(validate_data=False))

import time from scale_forecasting import Forecaster, RunConfig gce_preview = RunConfig( run_name="preview-gce-single-vm", python_runtime="gce", data={"source_table": "source_series_iceberg", "horizon": 28, "series_limit": 200}, models=["theta", "holtwinters", "lightgbm"], compute={"machine_type": "n2-standard-8", "workers": 1}, ) print("--- GCE Single-VM Plan ---") display(Forecaster(gce_preview, settings=settings).explain(validate_data=False)) vertex_gpu_preview = RunConfig( run_name="preview-vertex-multi-family", python_runtime="vertex", data={"source_table": "source_series_iceberg", "horizon": 28, "series_limit": 200}, models=["theta", "xgboost", "tide", "tsmixer"], compute={ "machine_type": "n2-standard-8", "workers": 2, "families": { "deep_learning": { "runtime": "vertex", "hardware": "gpu", "gpu_type": "L4", "accelerator_count": 1, "workers": 2, } }, }, ) print("--- Vertex AI Multi-Worker CPU + L4 GPU Plan ---") display(Forecaster(vertex_gpu_preview, settings=settings).explain(validate_data=False)) gke_gpu_preview = RunConfig( run_name="preview-gke-multi-family", python_runtime="gke", data={"source_table": "source_series_iceberg", "horizon": 28, "series_limit": 200}, models=["theta", "xgboost", "tide", "tsmixer"], compute={ "gke_mode": "job", "machine_type": "n2-standard-8", "workers": 2, "families": { "deep_learning": { "runtime": "gke", "gke_mode": "job", "hardware": "gpu", "gpu_type": "L4", "accelerator_count": 1, "workers": 1, # auto-expands 1 -> 2 dedicated GPU pods (1 per model) } }, }, ) print("--- GKE Indexed Job CPU + L4 GPU Plan (Auto-Expanded 1 -> 2 GPU Pods) ---") display(Forecaster(gke_gpu_preview, settings=settings).explain(validate_data=False))

--- GCE Single-VM Plan ---

	run_id	family	job_key	runtime	machine_type	workers	hardware	gpu_type	models	training_modes	n_series	n_folds	expected_cells	covariates	depends_on
0	preview-gce-single-vm-5cc5a953ef14	statistical	sf-preview-gce-single-vm-5cc5a953ef14-statisti...	gce	n2-standard-8	1	cpu	None	theta, holtwinters	theta:local, holtwinters:local	200	1	400	none
1	preview-gce-single-vm-5cc5a953ef14	ml	sf-preview-gce-single-vm-5cc5a953ef14-ml-a1	gce	n2-standard-8	1	cpu	None	lightgbm	lightgbm:local	200	1	200	none

--- Vertex AI Multi-Worker CPU + L4 GPU Plan ---

	run_id	family	job_key	runtime	machine_type	workers	hardware	gpu_type	models	training_modes	n_series	n_folds	expected_cells	covariates	depends_on
0	preview-vertex-multi-family-c2c9611e2a47	statistical	sf-preview-vertex-multi-family-c2c9611e2a47-st...	vertex	n2-standard-8	2	cpu	None	theta	theta:local	200	1	200	none
1	preview-vertex-multi-family-c2c9611e2a47	ml	sf-preview-vertex-multi-family-c2c9611e2a47-ml-a1	vertex	n2-standard-8	2	cpu	None	xgboost	xgboost:local	200	1	200	none
2	preview-vertex-multi-family-c2c9611e2a47	deep_learning	sf-preview-vertex-multi-family-c2c9611e2a47-de...	vertex	g2-standard-8	2	gpu	L4	tide, tsmixer	tide:local, tsmixer:local	200	1	400	none

--- GKE Indexed Job CPU + L4 GPU Plan (Auto-Expanded 1 -> 2 GPU Pods) ---

	run_id	family	job_key	runtime	machine_type	workers	hardware	gpu_type	models	training_modes	n_series	n_folds	expected_cells	covariates	depends_on
0	preview-gke-multi-family-c0bf6931d7a9	statistical	sf-preview-gke-multi-family-c0bf6931d7a9-stati...	gke	n2-standard-8	2	cpu	None	theta	theta:local	200	1	200	none
1	preview-gke-multi-family-c0bf6931d7a9	ml	sf-preview-gke-multi-family-c0bf6931d7a9-ml-a1	gke	n2-standard-8	2	cpu	None	xgboost	xgboost:local	200	1	200	none
2	preview-gke-multi-family-c0bf6931d7a9	deep_learning	sf-preview-gke-multi-family-c0bf6931d7a9-deep_...	gke	g2-standard-8	2	gpu	L4	tide, tsmixer	tide:local, tsmixer:local	200	1	400	none

Act 3 · Configure, Launch & Monitor a Live Vertex / GCE Run¶

Now let's configure and run a fast multi-family workload using python_runtime='vertex' (or switch RUNTIME = 'gce' to test ephemeral GCE VM execution) with statistical + ml models and in-run ensembling.

In [4]:

# === Parameters — edit me ===============================================
RUN_NAME = f"nb02 vertex gce {int(time.time())}"
RUNTIME = "vertex"  # "vertex" (Vertex AI CustomJob) or "gce" (Ephemeral Single-VM)
SOURCE_TABLE = "source_series_iceberg"
MODELS = ["theta", "naive_seasonal", "lightgbm"]
HORIZON = 28
SERIES_LIMIT = 15
N_FOLDS = 2
# ========================================================================

cfg = RunConfig(
    run_name=RUN_NAME,
    python_runtime=RUNTIME,
    data={"source_table": SOURCE_TABLE, "horizon": HORIZON, "series_limit": SERIES_LIMIT},
    models=MODELS,
    features={"holidays": ["US"]},
    backtest={"enabled": True, "n_folds": N_FOLDS, "horizon": HORIZON, "step": HORIZON},
    ensemble={"enabled": True, "strategies": ["mean", "inverse_error"]},
)

forecaster = Forecaster(cfg, settings=settings)
print("run_id:", forecaster.run_id)
display(forecaster.explain())

result = forecaster.run_live(poll_s=15.0, probe=True)
print(f"Completed {result.run_id}: status={result.status}, runtime={result.runtime_seconds:.1f}s")

# === Parameters — edit me =============================================== RUN_NAME = f"nb02 vertex gce {int(time.time())}" RUNTIME = "vertex" # "vertex" (Vertex AI CustomJob) or "gce" (Ephemeral Single-VM) SOURCE_TABLE = "source_series_iceberg" MODELS = ["theta", "naive_seasonal", "lightgbm"] HORIZON = 28 SERIES_LIMIT = 15 N_FOLDS = 2 # ======================================================================== cfg = RunConfig( run_name=RUN_NAME, python_runtime=RUNTIME, data={"source_table": SOURCE_TABLE, "horizon": HORIZON, "series_limit": SERIES_LIMIT}, models=MODELS, features={"holidays": ["US"]}, backtest={"enabled": True, "n_folds": N_FOLDS, "horizon": HORIZON, "step": HORIZON}, ensemble={"enabled": True, "strategies": ["mean", "inverse_error"]}, ) forecaster = Forecaster(cfg, settings=settings) print("run_id:", forecaster.run_id) display(forecaster.explain()) result = forecaster.run_live(poll_s=15.0, probe=True) print(f"Completed {result.run_id}: status={result.status}, runtime={result.runtime_seconds:.1f}s")

run_id: nb02-vertex-gce-1791391131-8a67037d28fd

	run_id	family	job_key	runtime	machine_type	workers	hardware	gpu_type	models	training_modes	n_series	n_folds	expected_cells	covariates	depends_on
0	nb02-vertex-gce-1791391131-8a67037d28fd	statistical	sf-nb02-vertex-gce-1791391131-8a67037d28fd-sta...	vertex	n2-standard-8	1	cpu	None	theta, naive_seasonal	theta:local, naive_seasonal:local	15	2	30	none
1	nb02-vertex-gce-1791391131-8a67037d28fd	ml	sf-nb02-vertex-gce-1791391131-8a67037d28fd-ml-a1	vertex	n2-standard-8	1	cpu	None	lightgbm	lightgbm:local	15	2	15	none
2	nb02-vertex-gce-1791391131-8a67037d28fd	ensemble	sf-nb02-vertex-gce-1791391131-8a67037d28fd-ens...	bigquery	sql	1	sql	None	ensemble_mean, ensemble_inverse_error	barrier	15	2	30	none	sf-nb02-vertex-gce-1791391131-8a67037d28fd-sta...

Completed nb02-vertex-gce-1791391131-8a67037d28fd: status=COMPLETED, runtime=475.4s

Act 4 · Review Leaderboard, Forecast Intervals, Two-Tier Feature Attributions, Job Sizing & Cohort Health¶

Every Vertex AI CustomJob, GCE Single-VM, and GKE Indexed Job family job records both its planned sizing ($.sizing) and actual executed hardware ($.sizing_executed) in run_jobs.details (forecaster.jobs_df()), exposes backtest cohort coverage via forecaster.cohorts_df(), and persists Two-Tier Feature Attributions (forecaster.attributions_df() and forecaster.plot_attributions()).

In [5]:

import matplotlib.pyplot as plt
from scale_forecasting import plot_leaderboard, plot_trace

display(forecaster.leaderboard_df())

review = forecaster.review_run()
plot_leaderboard(review)
plt.show()

forecaster.plot_forecasts(history_tail=90)
plt.show()

print("--- Two-Tier Feature Attributions (forecaster.attributions_df & plot_attributions) ---")
attr_df = forecaster.attributions_df(level="global")
if not attr_df.empty:
    display(attr_df.head(10))
    forecaster.plot_attributions(level="global", title="Tier 1 Global Feature Attributions")
    plt.show()
    forecaster.plot_attributions(level="local", title="Tier 2 Local Per-Step Feature Attributions")
    plt.show()

trace_df = forecaster.trace()
display(trace_df)
plot_trace(trace_df)
plt.show()

print("--- Per-Family Job Sizing & Telemetry (forecaster.jobs_df) ---")
display(forecaster.jobs_df())

print("--- Backtest Cohort Health & Comparable Holdout Leaderboard (forecaster.cohorts_df) ---")
display(forecaster.cohorts_df())

import matplotlib.pyplot as plt from scale_forecasting import plot_leaderboard, plot_trace display(forecaster.leaderboard_df()) review = forecaster.review_run() plot_leaderboard(review) plt.show() forecaster.plot_forecasts(history_tail=90) plt.show() print("--- Two-Tier Feature Attributions (forecaster.attributions_df & plot_attributions) ---") attr_df = forecaster.attributions_df(level="global") if not attr_df.empty: display(attr_df.head(10)) forecaster.plot_attributions(level="global", title="Tier 1 Global Feature Attributions") plt.show() forecaster.plot_attributions(level="local", title="Tier 2 Local Per-Step Feature Attributions") plt.show() trace_df = forecaster.trace() display(trace_df) plot_trace(trace_df) plt.show() print("--- Per-Family Job Sizing & Telemetry (forecaster.jobs_df) ---") display(forecaster.jobs_df()) print("--- Backtest Cohort Health & Comparable Holdout Leaderboard (forecaster.cohorts_df) ---") display(forecaster.cohorts_df())

	rank	model_type	family	is_ensemble	ensemble_id	compute_engine	n_series	decision_metric	score	pooled_wape	...	mean_mae	mean_rmse	mean_coverage	mean_interval_score	mean_fit_seconds	median_fit_seconds	no_artifact_rate	n_predictions	lift_vs_best_base	lift_pct
0	1	ensemble_inverse_error	ensemble	True	55cf2eed0555	ensemble	15	wape	0.426720	0.137570	...	13.097401	17.180223	NaN	NaN	NaN	NaN	1.0	420	0.015121	0.034222
1	2	lightgbm	ml	False	None	vertex	15	wape	0.441840	0.151010	...	13.821479	18.199157	0.282143	99.018758	2.286257	1.386402	1.0	420	NaN	NaN
2	3	theta	statistical	False	None	vertex	15	wape	0.443389	0.158292	...	14.614160	18.970049	0.995238	262.555232	6.527285	5.076216	1.0	420	NaN	NaN
3	4	ensemble_mean	ensemble	True	55cf2eed0555	ensemble	15	wape	0.445685	0.147642	...	14.492586	19.324221	NaN	NaN	NaN	NaN	1.0	420	-0.003845	-0.008702
4	5	naive_seasonal	statistical	False	None	vertex	15	wape	0.562872	0.195826	...	19.792555	28.264536	0.694048	108.083731	1.245784	1.117355	1.0	420	NaN	NaN

5 rows × 25 columns

--- Two-Tier Feature Attributions (forecaster.attributions_df & plot_attributions) ---

	ts_id	model_type	compute_engine	feature	importance
0	s_000000	lightgbm	vertex	lag_7	0.154222
1	s_000000	lightgbm	vertex	lag_1	0.137333
2	s_000000	lightgbm	vertex	lag_28	0.120444
3	s_000000	lightgbm	vertex	doy	0.120333
4	s_000000	lightgbm	vertex	lag_14	0.111778
5	s_000000	lightgbm	vertex	lag_2	0.109556
6	s_000000	lightgbm	vertex	lag_3	0.107222
7	s_000000	lightgbm	vertex	dom	0.079222
8	s_000000	lightgbm	vertex	dow	0.033111
9	s_000000	lightgbm	vertex	is_holiday	0.014444

	kind	lane	label	start	end	duration_s	status	runtime	model_type	ts_id
0	job	ensemble	ensemble	2026-10-07 16:46:01.972173+00:00	2026-10-07 16:46:26.291391+00:00	24.319218	COMPLETED	bigquery	None	None
1	job	ml	ml	2026-10-07 16:39:17.830371+00:00	2026-10-07 16:45:37.397149+00:00	379.566778	COMPLETED	vertex	None	None
2	job	statistical	statistical	2026-10-07 16:39:17.890613+00:00	2026-10-07 16:45:54.199515+00:00	396.308902	COMPLETED	vertex	None	None
3	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000014	2026-10-07 16:44:16.628112+00:00	2026-10-07 16:44:34.334460+00:00	17.706348	None	vertex	lightgbm	s_000014
4	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000004	2026-10-07 16:44:16.630049+00:00	2026-10-07 16:44:34.294712+00:00	17.664663	None	vertex	lightgbm	s_000004
5	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000003	2026-10-07 16:44:20.019205+00:00	2026-10-07 16:44:41.498731+00:00	21.479526	None	vertex	theta	s_000003
6	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000013	2026-10-07 16:44:20.019849+00:00	2026-10-07 16:44:40.847823+00:00	20.827974	None	vertex	theta	s_000013
7	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000004	2026-10-07 16:44:20.022294+00:00	2026-10-07 16:44:40.922204+00:00	20.899910	None	vertex	theta	s_000004
8	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000014	2026-10-07 16:44:20.022849+00:00	2026-10-07 16:44:41.035190+00:00	21.012341	None	vertex	theta	s_000014
9	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000012	2026-10-07 16:44:34.296676+00:00	2026-10-07 16:44:35.632304+00:00	1.335628	None	vertex	lightgbm	s_000012
10	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000003	2026-10-07 16:44:34.336614+00:00	2026-10-07 16:44:35.827326+00:00	1.490712	None	vertex	lightgbm	s_000003
11	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000013	2026-10-07 16:44:35.635653+00:00	2026-10-07 16:44:37.104924+00:00	1.469271	None	vertex	lightgbm	s_000013
12	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000000	2026-10-07 16:44:35.828657+00:00	2026-10-07 16:44:37.237700+00:00	1.409043	None	vertex	lightgbm	s_000000
13	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000010	2026-10-07 16:44:37.108276+00:00	2026-10-07 16:44:38.335382+00:00	1.227106	None	vertex	lightgbm	s_000010
14	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000009	2026-10-07 16:44:37.238803+00:00	2026-10-07 16:44:38.487324+00:00	1.248521	None	vertex	lightgbm	s_000009
15	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000011	2026-10-07 16:44:38.339199+00:00	2026-10-07 16:44:39.898557+00:00	1.559358	None	vertex	lightgbm	s_000011
16	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000002	2026-10-07 16:44:38.489586+00:00	2026-10-07 16:44:39.876151+00:00	1.386565	None	vertex	lightgbm	s_000002
17	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000001	2026-10-07 16:44:39.878196+00:00	2026-10-07 16:44:41.007136+00:00	1.128940	None	vertex	lightgbm	s_000001
18	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000007	2026-10-07 16:44:39.900663+00:00	2026-10-07 16:44:41.149203+00:00	1.248540	None	vertex	lightgbm	s_000007
19	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000013	2026-10-07 16:44:40.848697+00:00	2026-10-07 16:44:41.653082+00:00	0.804385	None	vertex	naive_seasonal	s_000013
20	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000004	2026-10-07 16:44:40.923046+00:00	2026-10-07 16:44:41.588385+00:00	0.665339	None	vertex	naive_seasonal	s_000004
21	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000008	2026-10-07 16:44:41.009947+00:00	2026-10-07 16:44:42.462651+00:00	1.452704	None	vertex	lightgbm	s_000008
22	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000014	2026-10-07 16:44:41.035991+00:00	2026-10-07 16:44:41.964038+00:00	0.928047	None	vertex	naive_seasonal	s_000014
23	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000006	2026-10-07 16:44:41.150512+00:00	2026-10-07 16:44:42.352283+00:00	1.201771	None	vertex	lightgbm	s_000006
24	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000003	2026-10-07 16:44:41.499731+00:00	2026-10-07 16:44:43.511082+00:00	2.011351	None	vertex	naive_seasonal	s_000003
25	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000012	2026-10-07 16:44:41.589219+00:00	2026-10-07 16:44:45.980543+00:00	4.391324	None	vertex	theta	s_000012
26	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000011	2026-10-07 16:44:41.653878+00:00	2026-10-07 16:44:49.047002+00:00	7.393124	None	vertex	theta	s_000011
27	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000000	2026-10-07 16:44:41.964832+00:00	2026-10-07 16:44:47.041181+00:00	5.076349	None	vertex	theta	s_000000
28	cell	cmle-training-workerpool0-86f32d14b8-0-mltww:1	lightgbm:s_000005	2026-10-07 16:44:42.355072+00:00	2026-10-07 16:44:43.483685+00:00	1.128613	None	vertex	lightgbm	s_000005
29	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000001	2026-10-07 16:44:43.511832+00:00	2026-10-07 16:44:50.711730+00:00	7.199898	None	vertex	theta	s_000001
30	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000012	2026-10-07 16:44:45.981283+00:00	2026-10-07 16:44:47.998964+00:00	2.017681	None	vertex	naive_seasonal	s_000012
31	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000000	2026-10-07 16:44:47.042065+00:00	2026-10-07 16:44:48.865258+00:00	1.823193	None	vertex	naive_seasonal	s_000000
32	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000010	2026-10-07 16:44:47.999933+00:00	2026-10-07 16:44:51.343293+00:00	3.343360	None	vertex	theta	s_000010
33	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000009	2026-10-07 16:44:48.866280+00:00	2026-10-07 16:44:51.451162+00:00	2.584882	None	vertex	theta	s_000009
34	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000011	2026-10-07 16:44:49.047826+00:00	2026-10-07 16:44:50.552646+00:00	1.504820	None	vertex	naive_seasonal	s_000011
35	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000007	2026-10-07 16:44:50.553731+00:00	2026-10-07 16:44:52.829937+00:00	2.276206	None	vertex	theta	s_000007
36	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000001	2026-10-07 16:44:50.712510+00:00	2026-10-07 16:44:51.809328+00:00	1.096818	None	vertex	naive_seasonal	s_000001
37	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000010	2026-10-07 16:44:51.344034+00:00	2026-10-07 16:44:52.582044+00:00	1.238010	None	vertex	naive_seasonal	s_000010
38	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000009	2026-10-07 16:44:51.451903+00:00	2026-10-07 16:44:52.569407+00:00	1.117504	None	vertex	naive_seasonal	s_000009
39	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000008	2026-10-07 16:44:51.810101+00:00	2026-10-07 16:44:56.120586+00:00	4.310485	None	vertex	theta	s_000008
40	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000002	2026-10-07 16:44:52.570177+00:00	2026-10-07 16:44:57.370126+00:00	4.799949	None	vertex	theta	s_000002
41	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000006	2026-10-07 16:44:52.582819+00:00	2026-10-07 16:44:58.824577+00:00	6.241758	None	vertex	theta	s_000006
42	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000007	2026-10-07 16:44:52.833641+00:00	2026-10-07 16:44:55.081490+00:00	2.247849	None	vertex	naive_seasonal	s_000007
43	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	theta:s_000005	2026-10-07 16:44:55.082213+00:00	2026-10-07 16:44:58.547910+00:00	3.465697	None	vertex	theta	s_000005
44	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000008	2026-10-07 16:44:56.121373+00:00	2026-10-07 16:44:57.893384+00:00	1.772011	None	vertex	naive_seasonal	s_000008
45	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000002	2026-10-07 16:44:57.370763+00:00	2026-10-07 16:44:58.484951+00:00	1.114188	None	vertex	naive_seasonal	s_000002
46	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000005	2026-10-07 16:44:58.548804+00:00	2026-10-07 16:44:58.782504+00:00	0.233700	None	vertex	naive_seasonal	s_000005
47	cell	cmle-training-workerpool0-752bc3eade-0-wjcks:1	naive_seasonal:s_000006	2026-10-07 16:44:58.825263+00:00	2026-10-07 16:44:58.940271+00:00	0.115008	None	vertex	naive_seasonal	s_000006

--- Per-Family Job Sizing & Telemetry (forecaster.jobs_df) ---

	family	job_key	system_job_id	runtime	status	attempt	hardware	gpu_type	spark_mode	runtime_seconds
0	ensemble	sf-nb02-vertex-gce-1791391131-8a67037d28fd-ens...	sf-nb02-vertex-gce-1791391131-8a67037d28fd-ens...	bigquery	COMPLETED	1	None	None	None	21.111446
1	ml	sf-nb02-vertex-gce-1791391131-8a67037d28fd-ml-a1	projects/307701787156/locations/us-central1/cu...	vertex	COMPLETED	1	cpu	None	None	376.602836
2	statistical	sf-nb02-vertex-gce-1791391131-8a67037d28fd-sta...	projects/307701787156/locations/us-central1/cu...	vertex	COMPLETED	1	cpu	None	None	393.162084

--- Backtest Cohort Health & Comparable Holdout Leaderboard (forecaster.cohorts_df) ---

	model_type	family	is_ensemble	n_series	score	pooled_wape	n_comparable_series	n_full	n_reduced	n_unscored	n_failed	n_not_requested	fold_histogram	refit_modes	staleness_gap
0	ensemble_inverse_error	ensemble	True	15	0.426720	0.137570	15	0	0	0	0	15	none	per_fold:15	None
1	lightgbm	ml	False	15	0.441840	0.151010	15	15	0	0	0	0	2f:15	per_fold:15	None
2	theta	statistical	False	15	0.443389	0.158292	15	15	0	0	0	0	2f:15	per_fold:15	None
3	ensemble_mean	ensemble	True	15	0.445685	0.147642	15	0	0	0	0	15	none	per_fold:15	None
4	naive_seasonal	statistical	False	15	0.562872	0.195826	15	15	0	0	0	0	2f:15	per_fold:15	None

---

Where next¶

• Next in Track 2 (Cloud Runtimes): 03_spark_serverless_and_connect.ipynb — fan out across Dataproc Serverless Spark batches, Dataproc clusters, and interactive Spark Connect.
• Compare runtimes: Choosing a runtime · Cost estimates & controls
• Documentation: Runtimes reference · Quota, sizing & scale proof