Runtime dependencies — how each compute surface gets its packages¶

The same forecasting code runs across eight Python compute surfaces — Colab Enterprise runtimes, Dataproc Serverless batches, Dataproc clusters, Ray on Vertex, Vertex AI CustomJob, Compute Engine (gce) Single-VM, Google Kubernetes Engine (gke Indexed Jobs & Ray on GKE), and Vertex AI AutoML (vertex_automl) — and each has its own native way of installing software. This page is the single source of truth for how the dependencies reach each surface, so that a fit running on any of them imports the identical scale_forecasting package and the identical third-party stack (statsmodels, lightgbm, torch, …).

This is about dependencies (the third-party libraries + the interpreter). Your source code ships separately at runtime — see editing_code_without_rebuilding.md. For the Python version each surface runs, see version_matrix.md.

The one source of truth: `uv.lock` + `.python-version`¶

Every mechanism below resolves back to a single locked environment, managed end-to-end by uv:

uv.lock — the fully-resolved, hashed version of every dependency (core + all extras + dev), and .python-version — the exact interpreter (3.11.15). Together they pin the whole stack.
docker/Dockerfile — uses uv to install that interpreter and uv sync --frozen that lock (core + the gcp, models, and ray extras) into an isolated venv at /opt/venv on debian:12-slim, plus the handful of system libraries the wheels link against (e.g. libgomp1 for lightgbm). The result is the shared runtime image.
docker/requirements.txt — a derived, human-readable export of the lock (uv export …, regenerated by make lock). It's kept in git so there's a stable file to diff in review and to content-address the packed-venv archive by, but it is not the source of truth — the lock is. CI fails if it drifts from the lock (see updating dependencies).

Everything else is a delivery detail. The image is the locked environment; the packed-venv archive is built from that same image; the notebook bootstrap installs the same lock. There is one place to bump when a dependency changes (pyproject.toml → make lock), and it is content-addressed — the image and the archive both rebuild only when the locked deps change, never on a source edit.

flowchart LR
    Lock["pyproject.toml +<br/>.python-version (3.11.15)<br/>↓<br/>uv.lock"] --> Req["docker/requirements.txt<br/>(Derived Export)"]
    Lock --> Img["Shared Runtime Container<br/>(/opt/venv on debian:12-slim)"]
    Img --> Sless["Dataproc Serverless<br/>& Spark Connect"]
    Img --> Vtx["Vertex AI CustomJob · GCE Single-VM<br/>& GKE (Indexed Job & Ray on GKE)"]
    Img -->|"tar /opt/venv"| Venv["Packed Venv Archive<br/>(gs://.../envs/<hash>.tar.gz)"]
    Venv --> Cluster["Dataproc GCE Cluster<br/>(--archives=...#env)"]
    Req --> RayEnv["Vertex AI Ray<br/>(runtime_env uv plugin)"]
    Lock --> Colab["Colab Enterprise (sf-main)<br/>& Local uv sync --frozen"]

Dependency extras¶

pyproject.toml layers the package so that a bare install is the pure forecasting layer and every client library is an opt-in extra. The layering is what lets pip install scale-forecasting run the playground with no project, billing, or credentials, and it is what keeps a Composer worker or a thin Cloud Shell client from carrying PySpark's JARs or PyTorch.

Extra	Adds	Composed from
(none)	pydantic, pandas, NumPy, pyarrow, statsmodels, SciPy, scikit-learn, threadpoolctl, Optuna, holidays — every model's metadata, the 15 models that fit on this stack alone, all 21 metrics, backtesting, calibration, reconciliation, ensembling, HPO, the playground, `--dry-run` and `--emit-airflow`	—
`gcp`	every Google Cloud client the package imports by name: BigQuery + Storage Read API, Cloud Storage, Dataproc, Vertex AI (`aiplatform`), `google-auth`, `google-api-core`, `db-dtypes`, `protobuf`	—
`notebook`	`ipykernel`, `matplotlib` (every `plot_*` helper in `review.py` and `sdk.plot_trace`)	—
`spark`	PySpark + `dataproc-spark-connect` (the interactive Spark Connect client; submitting a batch needs only `gcp`)	`gcp`
`ray`	`ray[default]` (Ray's `JobSubmissionClient`, the one launch client that is not a Google library) + `immutabledict`	`gcp`
`submit`	the thin launch client for Cloud Shell and CI — every launch client, no PySpark	`ray`
`models-stats`, `models-trees`, `models-prophet`, `models-dl`, `models-foundation`	statsforecast; xgboost + lightgbm + catboost; prophet; neuralprophet + neuralforecast + torch; chronos-forecasting + timesfm[torch]	—
`models-automl`	`google-cloud-pipeline-components` (the Tabular Workflow compiler behind the four `vertex_*` models)	`gcp`
`models`	the six family extras	`models-*`
`all`	everything — what `uv sync --all-extras` and `make sync` install	every extra

Each surface installs a named subset of this table: the shared runtime image and the docker/requirements.txt export carry gcp + models + ray (the Makefile EXPORT_ARGS, the Dockerfile, and the CI lock-check job spell the identical flag set); the notebook bootstrap cells install gcp (plus spark for the Spark Connect notebook) so a Colab run resolves the locked client versions rather than whatever the runtime image happens to ship; the dev dependency group includes gcp and notebook so a bare uv sync still yields a venv that can reach BigQuery and plot.

A core-only install that calls something cloud-side does not fail with a ModuleNotFoundError six frames down. main.run, every main verb except --dry-run and --emit-airflow, each submit CLI, and every plotting helper call errors.require_extra(...) first, which raises MissingExtraError (both an ImportError and a ScaleForecastError) whose message names the purpose, the extra, and the exact pip install "scale-forecasting[<extra>]" line. The offline plan (--dry-run) degrades instead: it still returns the run id, fan-out, and runtime split, and logs that the registry verdict and launch commands were skipped. tests/unit/test_packaging_extras.py keeps the table above, the four surfaces, and every install line in the docs in agreement with pyproject.toml; tests/unit/test_core_install.py and the CI core-install job prove the bare install in a fresh interpreter.

The matrix¶

Surface	Mechanism	What carries the deps	Set by
Dataproc Serverless (`explode` / `multi` batches)	Custom container	the shared runtime image, attached on every submit	`submit.py` (`runtime_config.container_image`)
Vertex AI CustomJob (`runtime="vertex"`)	Custom container	the same shared runtime image (`/opt/venv/bin/python`), attached to each `WorkerPoolSpec`	`vertex_submit.py` (`container_spec.image_uri`)
Compute Engine Single-VM (`runtime="gce"`)	Container on COS VM	the same shared runtime image launched via `docker run` on Container-Optimized OS	`vertex_submit.py` (`_build_gce_startup_script`)
GKE Indexed Job & Ray on GKE (`runtime="gke"`)	Custom container	the same shared runtime image (`/opt/venv/bin/python` + bundled `ray==2.59.0`) on GKE Autopilot or Standard pods	`gke_submit.py` / `gke_ray_infra.py`
Vertex AI AutoML (`runtime="vertex_automl"`)	Managed Vertex pipelines / training containers	launcher uses local/container `google-cloud-aiplatform` + `google-cloud-pipeline-components`; remote AutoML stages run inside Google-managed Kubeflow Pipeline / AutoML containers	`automl_submit.py` (`automl_mode="tabular_workflow" \\| "training_job"`)
Spark Connect (interactive, nb03) — the same Serverless product, session API not batch API	Custom container + artifacts	the same image, pinned on the session; code via `addArtifacts`. Covers the executors only — the driver is your local process	notebook session cell
Ray on Vertex (`runtime="ray"`, `ray_mode="vertex"`)	`uv` runtime_env	`uv` installs the frozen lock into the per-job venv on Vertex's prebuilt Ray image	`code_delivery.py` (`build_runtime_env`)
Dataproc cluster (`spark_mode="cluster"`)	Self-contained venv archive	a tar of the image's `/opt/venv` (interpreter bundled) attached to the job (`#env`)	`cluster_deps.py` + `compute.spark_deps`
Colab Enterprise (all notebooks)	`uv`-from-lock install	`uv` installs the frozen lock in the runtime (bootstrap or post-startup)	notebook bootstrap cell / template
Local dev	`uv` project env	`uv sync --frozen` installs the same lock	`uv.lock`

The rest of this page walks each row: why that mechanism, and what to know operationally.

Why more than one mechanism?¶

The obvious question about the table above is why it isn't one row. It can't be, and the reasons are platform facts rather than preferences. Four of them decide the whole shape:

A venv carries an interpreter but not system libraries. The archive bundles Python, so it needs no Python on the node — but it cannot bundle libgomp1 (which lightgbm links against), jemalloc, or an NVIDIA kernel driver. Something else has to supply those: the container installs them with apt; every other surface inherits them from its base image.
spark.archives reaches executors, not a client-mode driver. Job archives are localized into the executors' working directories, so a relative ./env/bin/python fails for the driver. On a Dataproc cluster we work around this with an init action that lands the venv at an absolute path on master and workers alike. Dataproc Serverless has no init actions — so the archive route is not a drop-in replacement for the container there, which is why the container is the default.
Ray-on-Vertex accepts neither an image nor an archive. A custom node image fails Vertex Ray GPU-node provisioning, and runtime_env offers pip / uv / conda / py_modules with no venv-tarball hook. Ray therefore must install at job start; there is no mechanism it shares with Spark.
Installing at job start makes a package index a run-time dependency. The container (Artifact Registry) and the archive (Cloud Storage) are both fetched over Google-internal paths and need no internet. uv pip install needs PyPI and the PyTorch CUDA index reachable from the subnet at the moment the job runs, for a set that includes a ~1 GB CUDA torch wheel. Ray pays this; nothing else should have to.

Constraints 2 and 3 together set the ceiling: at most two mechanisms, because the surface that must install at job start (Ray) and the surfaces that must not (Serverless, Vertex CustomJob, GCE, and GKE) share none. What is single is the thing that matters — one uv.lock, one build, one bump.

The Artifact-Registry-free fallback (`SF_SERVERLESS_DEPS`)¶

Because constraint 2 is about localization rather than support, spark.archives is a supported Serverless property, and the batch path can be switched to it:

export SF_SERVERLESS_DEPS=packed_venv     # default: container
export SF_VENV_ARCHIVE="$(terraform output -raw venv_archive_uri)"
# SF_CONTAINER_IMAGE is then not required

The batch is submitted with no container image and three properties instead: spark.archives with the archive under #env, and PYSPARK_PYTHON repointed at ./env/bin/python for both the driver (spark.dataproc.driverEnv.*) and the executors (spark.executorEnv.*). Which envelope ran is recorded on the run header — container_image and venv_archive in the batch telemetry, exactly one of them set.

This is a fallback, not a recommendation. It exists for a deployment with no Artifact Registry, and to make constraint 2 measurable on Serverless rather than assumed. It has not been proven live: if the driver doesn't get the archive localized, the job fails at startup with a missing interpreter. The default stays container, which installs nothing at launch and keeps the batch's Python ours regardless of which Python the runtime version ships.

The full alignment picture — substrate, Python, and GPU driver¶

The matrix above covers one layer: the Python environment. A running fit actually stands on three layers, and each is aligned independently:

Compute substrate — the VM/OS (or its managed equivalent) the job runs on.
Python runtime — the interpreter + locked third-party stack (the matrix above; everything traces back to uv.lock).
GPU driver — the host NVIDIA kernel driver, needed only by the deep-learning family on GPU hardware. This is a host/kernel layer, distinct from the Python torch wheel (which bundles its own CUDA userspace); a wheel can't supply the kernel driver.

This chart shows how each service satisfies all three — it is the map of which software reaches which service and how they stay aligned:

Service	Compute substrate	Python runtime	GPU driver
Dataproc Serverless	Google-managed	shared container image (from `uv.lock`)	Google-managed
Vertex AI CustomJob	Google-managed	shared container image (from `uv.lock`)	Google-managed
Compute Engine Single-VM (`gce`)	Container-Optimized OS (`cos-stable`)	shared container image (from `uv.lock`)	COS `cos-extensions install gpu`
GKE (`job` & `ray`)	GKE Autopilot or Standard node pools	shared container image (from `uv.lock`)	GKE-managed NVIDIA device plugin
Vertex AI AutoML (`vertex_automl`)	Google-managed Vertex AI / Dataflow	Google-managed pipeline containers (launched via `uv.lock` client)	Google-managed
Ray on Vertex	Google-managed	Vertex prebuilt image + `uv` runtime_env (same lock)	Google-managed
Dataproc cluster — CPU	stock `2.2-debian12` VM image	packed-venv archive (same lock)	— (no GPU)
Dataproc cluster — GPU	stock VM image, pinned to `2.2.85-debian12`	packed-venv archive (same lock)	provided by us, installed by an init action at cluster create

Two invariants keep the whole picture aligned:

One Python source of truth across every service. uv.lock builds the container image and the requirements.txt export; the cluster archive is a tar of that same image's /opt/venv. Dataproc Serverless, Vertex AI CustomJob, GCE Single-VM, and GKE (job and ray) consume the container directly; Ray on Vertex installs the same locked deps with uv into the per-job venv on Vertex's prebuilt image (byte-aligned with the container, no custom image needed); clusters can't take a container, so they carry the byte-identical environment as an archive. There is no second dependency definition anywhere — see the one source of truth.
The GPU driver is the one layer managed services get for free and clusters do not. Serverless, Vertex AI CustomJob, GKE, Vertex AI AutoML, and Ray on Vertex run on Google-managed substrate with the driver already present (and GCE Single-VM installs it cleanly via COS's built-in cos-extensions install gpu). A Dataproc cluster is a plain set of VMs, so the driver is ours to supply — and by default it is installed by an init action on each cluster create, which costs a few minutes of boot time and needs no artifact of ours at all. Because that install compiles NVIDIA's kernel modules against the node's running kernel, GPU clusters pin the image to a sub-minor (2.2.85-debian12) rather than taking the floating alias; CPU clusters, which run no driver install, keep tracking the alias. There is also an opt-in pre-baked GPU image that trades a build-time artifact for faster creates — both paths, and why the slower one is the default, are in GPU clusters — the driver init action below.

Only the deep-learning family on GPU hardware touches the driver layer. Statistical and ML families, and everything running on CPU, ignore it entirely.

Dataproc Serverless, Vertex AI CustomJob, GCE Single-VM & GKE — custom container¶

Serverless batches, Vertex AI CustomJobs, GCE Single-VMs, and GKE pods all accept our custom container image, so the cleanest path is to hand them the shared runtime image directly. submit.py attaches it on every batch via runtime_config.container_image, vertex_submit.py attaches it on every worker pool via container_spec.image_uri (or docker run on GCE COS), and gke_submit.py / gke_ray_infra.py attach it on every Kubernetes pod (invoking /opt/venv/bin/python directly). Nothing is installed at launch — the image is already the environment — so startup is fast and the executed environment is byte-for-byte the one that was built and tested. It also decouples our Python from the runtime version's: runtime 3.0 ships 3.12, we need 3.11 (Ray's ceiling), and the image makes that a non-issue.

This is the default Spark path and needs no per-run configuration beyond SF_CONTAINER_IMAGE (populated from terraform output). The envelope is chosen by the deployment, not by the run — SF_SERVERLESS_DEPS (default container; see the fallback section above), not compute.spark_deps, which is the Dataproc-cluster knob. Keeping it out of the config is deliberate: both envelopes deliver the identical locked environment, so folding the choice into the run config would fold it into the run_id and make one experiment two runs.

Spark Connect (interactive notebook) — container + shipped artifacts¶

This is not a fourth compute surface — it is Dataproc Serverless. A Connect session is a dataproc_v1.Session carrying the same runtime_config.container_image a Batch carries; only the API differs (interactive session vs. batch submit). It gets its own section for one reason:

the container covers the executors, but not the driver. In a batch, driver and executors both run in the cloud inside our image, so the environment question is closed. Under Connect the driver is your local Python process — the notebook kernel — and Connect refuses to run a driver whose Python minor doesn't match the workers'. That is why the session pins runtime 2.3 (workers already 3.11) rather than 3.0 (workers 3.12): our environment is 3.11 because Ray ceilings it there. See version_matrix.md.

The interactive Connect session (nb01) reuses the same container image — pinned on the Session so its executors carry the deps — and additionally ships the code with spark.addArtifacts(zip, pyfile=True), because an interactive session runs a closure pickled on the notebook kernel. The dep story is identical to Serverless; only the code-delivery step differs. Full detail lives in notebook_runtimes.md.

Ray on Vertex — uv runtime_env¶

A Vertex Ray cluster could take a custom worker image, but ours does not: a custom node image fails Vertex Ray GPU-node provisioning ("An internal error occurred on your cluster"), so the Ray path runs on Vertex's prebuilt Ray image and code_delivery.py's build_runtime_env installs the deps at job submit via Ray's runtime_env uv plugin. uv installs the same pinned requirements.txt export (the container is built from) into a per-job virtualenv, so a Ray task imports the identical stack a Spark task does — byte-aligned, no second resolve. The CUDA-12.6 torch build the GPU path needs comes from an extra index (--extra-index-url + --index-strategy unsafe-best-match, mirroring docker/Dockerfile), and Ray drops the image's pinned Ray version from the install list so the client↔cluster Ray stays matched. Ray 2.47's uv plugin self-bootstraps uv into the prebuilt image, so nothing preinstalls it. Because client↔cluster Ray parity is strict, the prebuilt image's Python (3.11) and the pinned Ray version are load-bearing — see version_matrix.md.

Dataproc cluster — self-contained venv archive¶

A Dataproc cluster cannot use a custom container (that is a Serverless-only feature). So the cluster path delivers the same locked environment a different way: a self-contained venv archive.

The image's /opt/venv is built to be fully self-contained — uv's interpreter (a python-build-standalone build with its own libpython + stdlib and relative RPATHs) is bundled in-tree, with bin/python pointing at it via a relative symlink. A normal venv only symlinks out to a base interpreter, so a packed copy would dangle on a bare cluster node; bundling the interpreter removes any dependency on the node having a compatible Python of its own.

How it is produced and consumed:

Pack (build time). After the shared image is built, the build tars its /opt/venv into env.tar.gz and uploads it to the code bucket at envs/<requirements-hash>.tar.gz. Because the venv is already self-contained and relocatable, a plain tar is all that's needed (no venv-pack; the image is intentionally pip-less). Building the archive from the image guarantees it is the exact same environment — no second resolve, no drift. (See docker/cloudbuild.yaml; Terraform runs this automatically on apply and exposes the URI as the venv_archive_uri output.)
Attach (submit time). cluster_submit.py attaches the archive to the PySpark job with --archives=<uri>#env and points Spark at the unpacked interpreter via spark.pyspark.python / spark.pyspark.driver.python = ./env/bin/python. Dataproc unpacks the tarball into ./env on every node before the job starts; ./env/bin/python resolves through the in-tree relative symlink to the bundled interpreter, so it runs from wherever it lands with no Python on the node.

Config + identity:

compute.spark_deps selects the mechanism. On a cluster it must be "packed_venv" (the default); "container" is rejected on a cluster because clusters can't take a container.
The archive URI is resolved from SF_VENV_ARCHIVE (or BatchInfra.venv_archive_uri / terraform output venv_archive_uri). If spark_deps="packed_venv" and no archive URI is available, submission fails fast with a clear error rather than running against the bare cluster.

Why not a cluster init-action pip install? It would re-resolve deps on every cluster create (slow, and not guaranteed identical to the tested image), and it would need network egress to PyPI from the cluster. The archive is a single content-addressed artifact that is provably the same environment as the container — one build, reused everywhere.

GPU clusters — the driver init action¶

The venv archive delivers the Python stack (including the CUDA torch wheel), but the deep-learning family on GPU hardware also needs the host NVIDIA kernel driver, which a wheel can't provide. On the managed services Google supplies it; on a cluster it is ours to install.

A GPU cluster boots a stock image and installs the driver at create time. That is the supported path, it is what a fresh deploy gets with no extra steps, and it is what smoke 06 is validated on. It costs a few minutes of driver install on every cluster create.

The image is pinned to 2.2.85-debian12, and that pin is not cosmetic. Google's install_gpu_driver.sh compiles NVIDIA's open kernel modules from source unless a prebuilt tarball for that exact kernel is already cached in the deployment's own Dataproc temp bucket. NVIDIA's source calls pci_resize_resource with three arguments; the Debian 6.1.0-52 headers that later 2.2 sub-minors boot declare a fourth, so the compile stops and the GPU workers fail to come up. Every released driver fails the same way, and so does Debian's own packaged nvidia-open-kernel-dkms, so neither the gpu-driver-version nor the gpu-driver-provider metadata knob is a way out. 2.2.85 boots 6.1.0-49, where the build succeeds. The pin lives in _GPU_IMAGE_VERSION in dataproc_cluster.py with the full reasoning; drop it once NVIDIA or Debian closes the gap. If Google retires 2.2.85 before then, the create fails with a message that names the pin and says where to change it — the version is in our code, not in your config.

There is also a pre-baked image, and it is an opt-in optimisation rather than the default. Set Terraform's build_gpu_image = true and a separate Cloud Build (docker/cloudbuild-gpu-image.yaml) produces a custom 2.2-debian12 VM image with the driver already in it, content-addressed on the customization script + the base Dataproc version rather than on the lock — deliberately, because it carries a kernel driver and must not rebuild when a dependency changes. dataproc_cluster.py picks it up through the gpu_image_uri Terraform output when one exists.

Why the fast path is not the default. A custom Dataproc image ages out. Google retires the base version it was built from, and after that the image cannot create clusters at all — the failure is Selected software image version … can no longer be used to create new clusters, and it lands on whoever next asks for a GPU cluster, in a message about image versions rather than about anything they did. The init-action path carries a sub-minor pin of its own, so it is not quite expiry-free either — but recovering from a retired pin is editing one constant, while recovering from a retired custom image requires rebuilding an image artifact. Take the optimisation if you create GPU clusters often enough for the boot-time install to matter, and expect to rebuild the image when the base version moves.

Colab Enterprise — `uv`-from-lock install¶

A Colab runtime is a general VM, but it installs from the same uv.lock as everything else rather than a fresh resolve. Colab Enterprise ships uv (the bootstrap installs it if a runtime doesn't have it); the install exports the frozen lock to a requirements file, uv pip installs exactly that set into the runtime's Python, then editable-installs the package with --no-deps. Two timings:

Bootstrap cell (default). Each notebook's first cell does git clone + the uv-from-lock install on first run. Self-contained; a little slower on a cold runtime.
Post-startup script (opt-in). Set install_via_post_startup = true to run the same install at runtime-creation time for a faster cold start. Off by default — the field is deprecated and some orgs block it.

The runtime's Python is pinned to 3.11 to match every other surface. Full Colab detail (templates, baked SF_* env, headless acceptance) is in notebook_runtimes.md.

Local dev — the `uv` project env¶

Locally, uv sync --frozen installs uv.lock (the same lock the clusters and the container are built from) into a project venv. This is what the notebooks and the offline test gate run on. See running_and_reviewing.md.

Updating dependencies¶

Because every surface traces back to uv.lock, there is exactly one update loop:

Edit pyproject.toml — change a floor, add a package, adjust an extra.
make lock — re-resolves uv.lock and regenerates the derived docker/requirements.txt. Commit both.
CI enforces it (.github/workflows/ci.yml): uv lock --check fails if the lock is stale vs pyproject.toml, and a re-export + git diff fails if docker/requirements.txt drifted from the lock. So the derived file can never silently diverge from the source of truth.
Re-deploy — rebuild the image (terraform apply runs Cloud Build), which re-syncs the lock into /opt/venv and re-tars the archive; both are content-addressed on the requirements hash, so they rebuild only when the locked deps actually changed. Local envs and notebook kernels pick up the new lock on their next uv sync / bootstrap.

Never pin a dependency in only one place (a bare pip install in a cell, a hand-edited requirements.txt); that reintroduces the drift this design exists to prevent — and CI will reject the hand-edit anyway.

Known limits and gotchas¶

The interpreter is in the archive, but system libraries are not. The archive bundles the Python interpreter (so it needs no Python on the node), but not the OS packages the container installs via apt (e.g. libgomp1, which lightgbm links against). Serverless bakes these into the custom container; Ray relies on Vertex's prebuilt Ray image providing them (it does — the live CPU/GPU smokes fit fine); a Dataproc cluster relies on its base image providing them. If a cluster fit fails with a missing .so (e.g. libgomp.so.1), the fix is a cluster init action that apt-get installs the library — the tarred venv can't carry a system lib.
The archive is large. The venv includes the CUDA-12.6 torch build (the GPU deep-learning path) plus the bundled interpreter, on the order of a gigabyte. That is fine for a cluster job (staged once to nodes) but is the reason the archive is content-addressed and cached in the code bucket rather than rebuilt per run.
The tar runs as root. /opt/venv is root-owned while the image's default user is the spark user (UID 1099), so the build tars the venv as root (docker run --user root …). This is a build-time detail; nothing about the running job changes.
spark.archives on Serverless is unproven. The SF_SERVERLESS_DEPS=packed_venv fallback is offline-tested (the batch message and the emitted gcloud command both carry the archive and the two PYSPARK_PYTHON properties) but has never run live. The open question is exactly constraint 2 above: whether the Serverless driver gets the archive localized. If it doesn't, the batch fails at startup with a missing interpreter — and unlike a cluster, there is no init action to fix it with.
One version knob for all surfaces. Because every mechanism traces back to uv.lock, edit pyproject.toml → make lock → re-apply (or rebuild): the image, the archive, and — after a uv sync / Colab reinstall — the notebook and local envs all move together. See updating dependencies; don't pin a dependency in only one place.

Sources: python-build-standalone · uv · Dataproc Serverless custom containers · Dataproc: Python environment on clusters

See also: architecture.md (the compute tracks) · version_matrix.md (the Python version per surface) · editing_code_without_rebuilding.md (how code ships at runtime) · deploying_on_gcp.md (the Terraform that builds the image + archive).