Skip to content

Version matrix — one Python everywhere, and why

This page is the single source of truth for which Python, Spark, and Ray versions run where, and why the whole system is pinned to Python 3.11. If you're debugging a PYTHON_VERSION_MISMATCH, a Ray JobSubmissionClient handshake hang, or a runtime-version choice, start here.

TL;DR

Python 3.11 is the single supported version across every surface — the uv project kernel, the custom container image, every Dataproc batch, every Ray cluster + client, the Spark Connect session, and every Colab template. This is not a coincidence to preserve casually: it's the one constraint that keeps four independent driver↔worker boundaries in parity. Deviating on any one of them breaks that surface.

The pin is enforced at the patch level, not just the minor: .python-version names the exact interpreter (3.11.15) and uv installs that build for local dev, CI, the runtime container, and the Dataproc-cluster packed-venv — so the interpreter is byte-identical everywhere uv controls it. (Colab and the Dataproc base runtimes provide their own 3.11 patch; parity is a minor-version property, so that's fine — the patch pin is about reproducibility where we build the environment ourselves.)

flowchart TD
    Pin[".python-version (3.11.15)<br/>pyproject.toml (>=3.11,<3.13 — package)"]
    Pin --> Kernel["Local uv Kernel &<br/>Colab Enterprise sf-main (py311)"]
    Pin --> Container["Custom Container (/opt/venv)<br/>& Packed Venv Archive (3.11.15)"]
    Kernel <-->|"1. applyInPandas Pickle Parity"| Connect["Dataproc Spark Connect 2.3<br/>(Python 3.11 Workers)"]
    Kernel <-->|"2. JobSubmissionClient REST API<br/>(Client Ray >=2.52 -> Cluster 2.47.1 + py311)"| VertexRay["Vertex AI Ray Cluster<br/>(Python 3.11 / Ray 2.47.1)"]
    Container <-->|"3. Driver ↔ Executor / Pod Parity"| Batch["Dataproc · Vertex CustomJob · GCE Single-VM<br/>& GKE (Indexed Job & Ray on GKE)"]

The matrix

Surface Where it runs Runtime version Base runtime Python Effective Python Spark / Ray How Python is set
Project / kernel local uv, Colab sf-main — — 3.11.15 Ray >=2.52 (client) .python-version (3.11.15); Colab template PATCHed to py311. (pyproject.toml requires-python = ">=3.11,<3.13" is the package's install range, see below)
Custom container attached to batch, Spark Connect, Vertex, GCE & GKE — — 3.11.15 Spark 3.5.x / Ray 2.59.0 docker/Dockerfile — uv installs 3.11.15 (from .python-version) into /opt/venv on debian:12-slim
Dataproc batch (explode) Serverless 2.2 (default) 3.12 3.11 ← container wins Spark 3.5.3 container image attached on every submit (submit.py), overriding base
Vertex CustomJob · GCE Single-VM · GKE (job & ray) Vertex AI, GCE COS, GKE Autopilot/Standard — — 3.11.15 Ray 2.59.0 (when gke_mode="ray") runs /opt/venv/bin/python directly inside the shared runtime container image
Spark Connect (nb03, interactive) Serverless session 2.3 3.11 3.11 Spark 3.5.3 runtime 2.3 base is already 3.11 and container attached
Ray on Vertex (ray_mode="vertex") Vertex Ray cluster Ray 2.47 3.11 3.11 Ray 2.47.1 (cluster) ray_infra.py pins python_version="3.11", ray_version="2.47"; client uses ray>=2.52 over REST Jobs API
Vertex AI AutoML (vertex_automl) Vertex AI Pipelines & Training Managed Managed 3.11.15 (launcher) / Managed (pipeline) — launcher runs in 3.11.15; remote AutoML stages execute inside Google-managed Kubeflow / AutoML containers
BigQuery-native (arima_plus, timesfm) BigQuery engine — — n/a (SQL) — no client Python on the compute path

"Effective Python" is what your code actually executes on. For batch it's the container's 3.11, not the base runtime's 3.12 — the attached image replaces the runtime's interpreter for both driver and executors. That's why batch can sit on runtime 2.2 (base 3.12) and still run 3.11 cleanly.

Why 3.11 everywhere — the four parity boundaries

Each of these is a place where two processes exchange pickled Python objects and therefore must share a Python minor version (3.11 ≠ 3.12 for pickle/applyInPandas/cloudpickle purposes):

  1. Dataproc batch: container ↔ itself. The container sets both driver and executor Python, so this is self-consistent by construction — the base runtime (2.2 = 3.12) never touches your code because the image overrides it. This is why the batch default runtime version is unremarkable: the container, not the runtime, decides.

  2. Spark Connect: Colab kernel ↔ session executors. The applyInPandas fan-out pickles the group-runner closure on the notebook kernel (sf-main, py3.11) and unpickles it on the session executors. So the executors must be 3.11. Runtime 2.3's workers are Python 3.11; runtime 3.0's are Python 3.12 → PYTHON_VERSION_MISMATCH. This is the whole reason nb01 uses 2.3, not 3.0.

  3. Ray on Vertex: Python 3.11 cluster image + REST Jobs API. Vertex offers prebuilt Ray cluster images only for a fixed set (2.9.3 / 2.33.0 / 2.42.0 / 2.47.1), and on Python 3.11 only 2.42 or 2.47 are available. ray_infra.py defaults ray_version="2.47" and python_version="3.11" for the cluster image, while the client-side [ray] extra installs ray[default]>=2.52 (for security remediation) and submits jobs via the HTTP REST JobSubmissionClient over PSC-I (with ray excluded from the on-cluster uv runtime_env via code_delivery._CLUSTER_PROVIDED).

  4. Same code locally and under Composer. .python-version pins the uv project to 3.11.15, so a developer's local kernel, the CI kernel, and the Composer runner all resolve the same interpreter — the code that runs locally is byte-identical to what runs in the cloud.

The package installs on 3.11–3.12; the cloud stack runs 3.11

These are two different contracts, kept apart on purpose. requires-python = ">=3.11,<3.13" says what the package installs on: the pure layer (models, metrics, backtests, the playground) and the client libraries all run on 3.11 and 3.12, every dependency in uv.lock resolves for both, and the CI core-install job proves the bare install on each. (3.13 waits on the numpy<2 pin neuralprophet still needs — numpy 1.26 ships no cp313 wheel — and widens the day that pin lifts.) That is what lets a 3.12 notebook pip install scale-forecasting and run the playground, or build a RunConfig and call --dry-run, without a second toolchain. .python-version says what the cloud stack runs on, and the four parity boundaries above are why that stays 3.11: a driver that submits to Spark Connect or Ray on Vertex must match the workers, and the runtime container and packed-venv are built from the 3.11.15 lock. Run those submit paths from a 3.11 interpreter; run everything else from either.

The Spark Connect 2.x vs 3.x decision (the "Connect 2 / Connect 3" question)

Both Dataproc runtime 2.3 and 3.0 support Spark Connect. We use 2.3. The trade:

Runtime 2.3 ✅ (chosen) Runtime 3.0 ❌
Spark 3.5.3 4.0.x
Base worker Python 3.11 3.12
Parity with sf-main kernel (py3.11) ✅ holds ❌ PYTHON_VERSION_MISMATCH on applyInPandas
Parity with custom container (py3.11) ✅ ❌
Needs a new py3.12 container + py3.12 kernel? no yes — a whole second toolchain

Runtime 3.0 (Spark 4) is the newer line and drops Jupyter in favor of Connect-only sessions — but adopting it would force the entire stack to Python 3.12, which is blocked anyway: Vertex Ray caps at Python 3.11 (no 3.12 cluster image), and the client must match the cluster. So going to 3.0 would split the toolchain (3.12 for Spark/batch, 3.11 for Ray) — reintroducing the two-container, two-template mess that the single sf-main template deliberately removed. 3.11-everything is the only configuration that unifies all tracks under one container and one template — 2.3 is a consequence of that, not a compromise.

Revisit only if Vertex Ray ships a Python 3.12 cluster image; until then 3.11 is load-bearing.

Where these numbers live in the code

Value File
requires-python = ">=3.11,<3.13" (the package's install range) pyproject.toml
exact interpreter 3.11.15 (uv installs it) .python-version
uv-installed 3.11.15 into /opt/venv, debian:12-slim docker/Dockerfile
Batch default runtime "2.2" src/scale_forecasting/batch_infra.py
Container attached on submit src/scale_forecasting/submit.py
Spark Connect session version = "2.3" + container notebooks/03_spark_serverless_and_connect.ipynb (Act 5)
Ray python_version="3.11", ray_version="2.47" src/scale_forecasting/ray_infra.py
Colab template pinned to py311 (REST PATCH) terraform/main/modules/colab/main.tf

Sources

See also: notebook_runtimes.md (per-notebook Python + template mapping) · architecture.md (the compute tracks) · deploying_on_gcp.md (the Terraform).