Interactive Notebooks (`notebooks/`)¶

A Complete Interactive Curriculum for Enterprise Scale Forecasting on Google Cloud
From local zero-GCP prototyping across all 37 models and 21 metrics to distributed execution across BigQuery ML, GCE, Vertex AI, GKE (Indexed Jobs & Ray on GKE), Vertex AI AutoML Tabular Workflows, Spark, and Ray — plus 3-tier covariates, two-tier explainability, 8-method FPP3 reconciliation, cross-run ensembling, custom 1-file plugins, and 100,000-series operations.

flowchart TD
    subgraph Track1["Track 1: Local Sandbox & Custom Plugins (Zero GCP Required)"]
        NB00["00_model_playground.ipynb<br/>37 Models · 21 Metrics · Local/Global/Hybrid<br/>3-Tier Covariates · 8 FPP3 Reconciliation"]
        NB09["09_custom_models_and_metrics.ipynb<br/>1-File BaseModel & BaseMetric Plugins<br/>+ Zero-Rebuild Dynamic Code Shipping"]
    end

    subgraph Track2["Track 2: Cloud Runtimes & Distributed Engines"]
        NB01["01_bigquery_native_sql.ipynb<br/>Serverless BigQuery SQL (ARIMA_PLUS · ARIMA_PLUS_XREG · TimesFM 2.5/3.0)"]
        NB02["02_vertex_and_gce_vms.ipynb<br/>GCE Single-VM, Vertex AI & GKE Indexed Jobs (CPU + L4 GPU)"]
        NB03["03_spark_serverless_and_connect.ipynb<br/>Dataproc Serverless Batch, Cluster & Spark Connect"]
        NB04["04_ray_on_vertex_gpu.ipynb<br/>Autoscaling Ray on Vertex AI & GKE + Fractional GPU Packing"]
    end

    subgraph Track3["Track 3: Covariates, Hierarchy, HPO & Cross-Run Ensembles"]
        NB05["05_covariates_and_global_models.ipynb<br/>3-Tier Covariates (Static · Future · Past) + Global ML"]
        NB06["06_hierarchical_reconciliation.ipynb<br/>Coherent Hierarchy Rollups (MinT Shrinkage · WLS · Bottom-Up)"]
        NB07["07_hpo_backtesting_and_ensembles.ipynb<br/>Optuna HPO + In-Run, Post-Run & Cross-Run Ensembling"]
    end

    subgraph Track4["Track 4: Master Multi-Family DAG, Registry Ops & Run Review"]
        NB08["08_multi_engine_master_workflow.ipynb<br/>Multi-Family Parallel DAG + Foundation Models + Stacked Ensembles"]
        NB10["10_registry_operations_and_scale.ipynb<br/>Registry Doctor, Live Probes, Composer DAG & Run Review"]
    end

    NB00 --> NB09
    Track1 --> Track2 --> Track3 --> Track4

Eleven Focused & Combination Workflow Notebooks¶

Every cloud workflow notebook (01–08) is a complete, self-contained 5-act journey: 1. Setup & Architecture Context: Bootstrap, Settings.resolve(), and visual Mermaid architecture map. 2. Configure & Explain Plan: Author RunConfig and inspect the pre-flight execution table via forecaster.explain(). 3. Launch & Monitor Live: Execute asynchronously with live in-place progress bars and infrastructure probes via forecaster.run_live(). 4. Retrieve & Review Results: Inspect forecaster.leaderboard_df(), plot_leaderboard(), plot_metric_distribution(), forecaster.plot_forecasts(), forecaster.cohorts_df(), forecaster.jobs_df(), forecaster.attributions_df(), forecaster.plot_attributions(), and forecaster.trace(). 5. Deep Dive / Follow-Up Action: Inspect 4-panel forecast decomposition (forecaster.plot_forecast_explanation()), two-tier feature attributions (forecaster.attributions_df(), forecaster.plot_attributions()), empirical interval calibration by horizon step (forecaster.plot_calibration()), learned ensemble stacking weights (forecaster.plot_ensemble_weights()), post-run reensemble(), cross-run ensemble_runs(), or direct Spark/Ray engine embedding (run_group, chunk_cells).

Track 1: Foundations, Local Prototyping & Extensibility (Zero GCP Setup)¶

Notebook	Target Environment	What You Will Learn
`00_model_playground.ipynb`	Local Python (Offline)	Explore the full 37-model (5-family) and 21-metric catalogs (`model_catalog()`, `metric_catalog()`), generate synthetic panels (`sample_data()`), run 3-fold rolling-origin backtesting and conformal interval calibration (`run_model()`, `bakeoff()`), decompose forecasts into trend, regime level shift, seasonality, and exogenous covariate attribution (`explain_forecast_frame()`, `plot_forecast_explanation()`), inspect Two-Tier Feature Attributions (`build_attributions_frame()`, `plot_attributions()`), compare `local` vs. `global` vs. `hybrid` training modes, run all 8 FPP3 hierarchical reconciliation methods, and evaluate the Forecaster Workbench (`fva_df()`, `horizon_decay_df()`, `bias_report_df()`, `rollup_df()`, `inventory_policy_df()`, `simulate_scenarios()`) with zero cloud credentials.
`09_custom_models_and_metrics.ipynb`	Local Python (Offline)	Author a custom 1-file `BaseModel` subclass (`@register`) and custom `BaseMetric` subclass, test them immediately in `run_model()` and `bakeoff()`, and inspect automatic BigQuery schema migration (`render_migrations`) and zero-rebuild `src.zip` shipping.

Track 2: Cloud Runtimes & Distributed Engines¶

Notebook	Target Environment	What You Will Learn
`01_bigquery_native_sql.ipynb`	Serverless BigQuery ML	Execute `arima_plus` (`ARIMA_PLUS`), `arima_plus_xreg` (`ARIMA_PLUS_XREG` + `ML.EXPLAIN_FORECAST` Two-Tier attributions), and Google Research `timesfm@univariate` (`TimesFM 2.5`) vs. `timesfm` (`TimesFM 3.0` multivariate covariates via `AI.FORECAST`) directly inside BigQuery SQL with zero cluster provisioning, blend them into stacked ensembles, evaluate Covariate FVA (`fva_df()`), detect anomalies and structural change points (`detect_anomalies_df()`, `detect_change_points_df()`), and compute $(s, S)$ inventory replenishment policies (`inventory_policy_df()`, `plot_inventory_policy()`).
`02_vertex_and_gce_vms.ipynb`	GCE Single-VM, Vertex AI CustomJob & GKE Indexed Jobs	Compare all three direct container runtimes powered by `vertex_engine.py`: ephemeral GCE Single-VM (`runtime="gce"`, triple-redundant auto-delete), multi-worker Vertex AI CustomJob (`runtime="vertex"`), and Google Kubernetes Engine Indexed Jobs (`runtime="gke", gke_mode="job"`, per-model GPU pod auto-expansion and independent node scale-down), plus inspect Two-Tier Feature Attributions (`attributions_df()`, `plot_attributions()`).
`03_spark_serverless_and_connect.ipynb`	Dataproc Spark (`serverless` / `cluster` / `connect`)	Run the distributed Arrow-backed `groupBy(bucket).applyInPandas` fan-out on Dataproc Serverless in parallel with BigQuery SQL, compare with Dataproc Standard Clusters and interactive Spark Connect (`DataprocSparkSession`), and embed the pure Spark UDF directly via `spark_io.run_group()`.
`04_ray_on_vertex_gpu.ipynb`	Ray on Vertex AI & Ray on GKE (CPU & GPU)	Provision ephemeral autoscaling Ray clusters on Vertex AI (`ray_mode="vertex"`) or GKE (`ray_mode="gke"` / `runtime="gke", gke_mode="ray"`), pack PyTorch deep learning tasks fractionally onto GPUs (`gpu_fraction=0.25`), compare `local`, `global`, and `hybrid` neural network regimes, and embed Ray task chunking directly via `ray_io.chunk_cells()`.

Track 3: Advanced Modeling — Covariates, Hierarchy, HPO & Cross-Run Ensembles¶

Notebook	Target Environment	What You Will Learn
`05_covariates_and_global_models.ipynb`	Vertex AI / Spark + Exog Table	Configure all 3 exogenous covariate tiers (`static_covariates`, `future_covariates`, lag-shifted `past_covariates`) on `source_series_covariates_native`, train global cross-series deep learning (`tide`, `tsmixer`) and ML models (`lightgbm`, `xgboost`), and visualize both structural exogenous covariate attribution (`plot_forecast_explanation`) and Two-Tier TreeSHAP feature attributions (`attributions_df`, `plot_attributions`).
`06_hierarchical_reconciliation.ipynb`	Vertex AI / Spark + Hierarchy	Configure multi-level business hierarchies (`region` $\rightarrow$ `category` $\rightarrow$ `ts_id`), reconcile forecasts across `bottom_up`, `wls_struct`, and `mint_shrink`, and verify exact coherence across levels with `forecaster.hierarchy_df()` and `forecaster.plot_hierarchy()`.
`07_hpo_backtesting_and_ensembles.ipynb`	BigQuery + Vertex AI	Tune hyperparameters with Optuna (`hpo.enabled=True`, `forecaster.best_params_df()`), inspect backtest cohort health (`cohorts_df()`), run in-run ensembling, add new stacked strategies in seconds with post-run re-ensembling (`forecaster.reensemble()`), combine base models across two separate runs using cross-run ensembling (`forecaster.ensemble_runs()`), and plot learned base-model weights (`plot_ensemble_weights()`).

Track 4: Master Multi-Family Workflow, Registry Operations & Run Review¶

Notebook	Target Environment	What You Will Learn
`08_multi_engine_master_workflow.ipynb`	Spark $\parallel$ Vertex / GKE / AutoML $\parallel$ BigQuery	Dispatch active model families (`statistical`, `ml`, `deep_learning` including `[models-foundation]` `chronos_bolt`, and `native`) concurrently across their optimal runtimes from a single `RunConfig` (zero-idle per-family compute — including a 5-family execution plan preview with `automl` on `vertex_automl`), join them in a stacked ensemble, and run the complete diagnostic, explainability, anomaly/change-point, and inventory replenishment workbench (`leaderboard_df`, `fva_df`, `cohorts_df`, `jobs_df`, `attributions_df`, `plot_attributions`, `plot_calibration`, `plot_ensemble_weights`, `plot_forecasts`, `plot_forecast_explanation`, `plot_trace`, `detect_anomalies_df`, `detect_change_points_df`, `inventory_policy_df`, `plot_inventory_policy`).
`10_registry_operations_and_scale.ipynb`	BigQuery Registry & Ops	Zero-SQL registry health audit (`reg.doctor()`, `reg.runs_df()`), historical run reattachment (`Forecaster.from_run_id()`, `attributions_df()`), pre-flight fold feasibility (`feasibility()`), preview-safe Day-2 verbs (`retry`, `settle`, `cancel`), all 5 production launch pathways (including live execution of the Cloud Composer 3 `airflow_tasks` DAG sequence against a GCS-staged config), and a cross-platform review of every completed run in the registry (the 100,000-series benchmark `run_id`s themselves are recorded in Quota & Scale).

How to Run the Notebooks¶

Option 1: 1-Click in Google Cloud Colab Enterprise (Recommended)¶

When you deploy the platform via Terraform (create_colab_runtimes = true, enabled by default), the sf-main Python 3.11 runtime template is pre-configured with all required environment variables (SF_PROJECT_ID, SF_DATASET_ID, SF_CONNECTION, SF_WAREHOUSE_URI, SF_CODE_BUCKET, SF_CONTAINER_IMAGE, SF_COMPUTE_SA, SF_SUBNETWORK_URI).

Open any notebook and click the Run in Colab Enterprise badge at the top.
Select the sf-main runtime template.
Click Run all — the environment bootstrap cell automatically sets up the locked dependencies from uv.lock and executes smoothly.

Option 2: Local Jupyter / VS Code¶

Install the pinned environment and register the Jupyter kernel:

uv sync --all-extras
uv run python -m ipykernel install --user --name scale-forecasting

Export your deployment's environment variables (not required for 00_model_playground.ipynb or 09_custom_models_and_metrics.ipynb):
```
eval "$(cd terraform/main && terraform output -raw sf_env_exports)"
```
Open any notebook in your local IDE and select the scale-forecasting kernel.

Automated Verification¶

All eleven notebooks are verified end-to-end using the headless acceptance test runner:

uv run python -m scale_forecasting.notebook_acceptance --tier all

For technical details on all 7 cloud compute runtimes and 4-tier scaling, see docs/runtimes_reference.md. For Colab Enterprise runtime templates and interpreter configurations, see docs/notebook_runtimes.md.

Interactive Notebooks (notebooks/)¶