Troubleshooting¶

The single home for known issues seen building and running this at scale, each as symptom → cause → fix / where it's handled. Most are already handled in code or config; this doc explains why the handling exists so you recognize the symptom fast.

Grouped by where it bites: In-flight runs · Capacity · Ray · Spark · Registry / writes · Install / extras · Versions · Deploy / Terraform · Notebooks.

In-flight runs — probe, settle, cancel¶

The rest of this page is symptom-keyed. This section is the tool you reach for when the symptom is "I can't tell what this run is doing" — a bar that has stopped moving, a job that may or may not still exist, a run you want to stop without losing what it already produced.

Why the registry alone can't answer. A run_jobs row is written by the job. A job that dies without writing — a preempted VM, a torn-down cluster, a driver OOM — leaves its row RUNNING forever. That row is visually identical to a slow but healthy job. The probe exists to break that tie by asking the runtime itself.

flowchart TD
    Watch["monitor_run(run_id)<br/>Check quiet_seconds"] -->|"Suspiciously quiet"| Probe["main --probe / monitor(probe=True)<br/>Query Dataproc · Vertex Ray · Vertex CustomJob · GCE · GKE · Vertex AutoML · BigQuery"]
    Probe -->|"RUNNING_CONFIRMED"| Wait["Healthy & Running<br/>Keep waiting (or --cancel --force)"]
    Probe -->|"STALE_REGISTRY or<br/>LIKELY_COMPLETED"| SettleOK["Work Landed<br/>main --settle --force → COMPLETED"]
    Probe -->|"LOST or<br/>ABANDONED_WAIT"| SettleFail["Job Gone / Walk Abandoned<br/>main --settle --force → FAILED"]
    SettleOK & SettleFail --> Close["All Job Rows Terminal<br/>registry.ops close-runs --yes"]

Probe a run¶

python -m scale_forecasting.main --config CONFIG --probe            # whole run
python -m scale_forecasting.main --config CONFIG --probe --job ml   # one family

Forecaster(cfg).probe()  # the same report
Forecaster(cfg).monitor(probe=True)  # progress bars *with* the reconciliation attached

The probe reads the registry, then escalates every non-terminal family to its own runtime (Dataproc / Vertex Ray / Vertex CustomJob / Compute Engine / GKE / Vertex AutoML / BigQuery) using the probe_handle recorded in run_jobs.job_telemetry at launch. Three properties are worth knowing before you put it in a loop:

It is advisory. Every native call is capped (20 s) and any error degrades that family to UNKNOWN. A probe never raises and never hangs, so it cannot take down the thing that called it.
A terminal run short-circuits. Nothing is escalated and no native call is made, so probing a finished run is free.
It is not a poll. quiet_seconds (free, in every monitor()) is the cheap signal; the probe is the deliberate escalation when an age has grown suspicious. Reach for it by exception.

Read the verdict¶

Each family comes back with one of seven verdicts. The two columns that matter are what the registry says and what the runtime says; disagreement is set only when they contradict each other.

Verdict	Means	What to do
`TRUST_REGISTRY`	Terminal, or never launched — nothing to reconcile.	Nothing.
`RUNNING_CONFIRMED`	The runtime confirms the job is alive.	Wait. The bar is slow, not dead.
`STALE_REGISTRY`	The runtime is terminal but the row never caught up.	The run is over; `--settle` writes that down.
`LIKELY_COMPLETED`	Job gone and every expected cell landed.	It finished; the finalize write was lost. `--settle`.
`LOST`	Job gone, artifacts missing, past the startup grace.	The job died. `--settle` finalizes it `FAILED`, then re-run.
`ABANDONED_WAIT`	Still `AWAITING_CAPACITY` past any walk's own budget.	The driver that was waiting is gone. `--settle`, then re-run.
`UNKNOWN`	Couldn't tell — no handle, or the probe degraded.	Re-probe; treat the registry as authoritative meanwhile.

Some distinctions in that table are deliberate and easy to misread:

LIKELY_COMPLETED vs LOST is decided by artifact evidence, not by the runtime. Both are "the job is gone"; the landed-vs-expected cell count is what separates a run that finished from one that died. A family with no known denominator (n_expected unavailable) can never be LIKELY_COMPLETED — it degrades to LOST rather than claiming success it can't prove.
A young job that 404s is UNKNOWN, not LOST. The RUNNING row is written before the native job exists, so a probe fired seconds after launch legitimately finds nothing. Only a family quiet past the startup grace (15 min, stale_after_s to override) is judged LOST. An EMITTED row — a staged command whose job id has been handed out but which nobody has run yet — gets the same grace for the same reason.
A native family reporting SUCCEEDED with incomplete artifacts is UNKNOWN, not stale. A BigQuery family's statements go DONE one at a time, so an all-DONE reading mid-run is a lull between statements. The probe declines to overrule the registry on that.
ABANDONED_WAIT is the one verdict with no runtime behind it. A family still waiting for capacity never created a runtime job, so there is nothing to ask; the witness is the clock. A live capacity walk ends itself at its budget (CapacityPolicy.max_wall_seconds, 3600 s) by writing FAILED/CAPACITY_EXHAUSTED, so a row still AWAITING_CAPACITY at twice that (2 h, abandoned_after_s to override) was written by a walk that stopped existing — a Ctrl-C, a closed shell, a restarted kernel. Under that window the wait is legitimate and reads TRUST_REGISTRY.
A capacity wait is asked, and usually ignored. Every job row carries the runtime id it was going to submit under from the moment it is written, so the probe can address the job during the launch window rather than only after it. A side effect is that a family still hopping regions gets asked about a job that does not exist yet. Its answer — "no such job" — is word-for-word what an orphaned launcher's vanished job gives, so it is not allowed to change the row: only the runtime naming a job it actually has, or a complete set of landed cells, can overrule an AWAITING_CAPACITY status. Without that rule every healthy capacity wait in the fleet would report UNKNOWN.

Settle a stale row¶

The probe knows the deep-learning family finished forty minutes ago. Settle is the verb that writes that down — a probe that writes, at the job tier, taking the probe's own per-family verdict.

python -m scale_forecasting.main --config CONFIG --settle                      # preview only
python -m scale_forecasting.main --config CONFIG --settle --force \
    --reason "driver OOMed after the last cell landed"                         # execute

Forecaster(cfg).settle()  # preview
Forecaster(cfg).settle(yes=True, reason="…")  # execute

Six readings settle; everything else is refused:

Verdict	Registry says	Runtime says	Cells	Settles to
`STALE_REGISTRY`	any	`SUCCEEDED`	all landed	`COMPLETED`
`STALE_REGISTRY`	any	`FAILED`	any	`FAILED`, `failure_reason=RUNTIME_FAILED`
`LIKELY_COMPLETED`	any	job gone	all landed	`COMPLETED`
`LOST`	`EMITTED`	job gone	missing	`FAILED`, `failure_reason=NEVER_LAUNCHED`
`LOST`	anything else	job gone	missing	`FAILED`, `failure_reason=RUNTIME_LOST`
`ABANDONED_WAIT`	`AWAITING_CAPACITY`	never launched	missing	`FAILED`, `failure_reason=CAPACITY_ABANDONED`

Refusal is the feature. RUNNING_CONFIRMED is live, TRUST_REGISTRY is already terminal or deliberately waiting, and UNKNOWN — the probe degraded, no handle was recorded, the runtime claims SUCCEEDED but the cells have not landed — never writes. The preview lists the families it will not touch and why, because "left alone: verdict UNKNOWN" is the line that means go look, and a count of what was settled is exactly how the one row that mattered gets missed.

Six more things worth knowing:

NEVER_LAUNCHED is not RUNTIME_LOST, and it is the same distinction in the other direction. Both rows read "job gone, work missing". RUNTIME_LOST means a job ran and vanished, so there is something to go and look for — a driver log, an OOM, a cancelled batch. NEVER_LAUNCHED means nothing ever ran: the row was filed by stage to reserve the job id it printed, and nobody pasted the command. There is nothing to investigate; just run it, or drop the run. See A submit is refused for why the row exists at all. Settling it early is safe even if you are merely slow: a launch that happens afterwards files a fresh row at the same attempt, and v_run_jobs takes the later one.
CAPACITY_ABANDONED is not CAPACITY_EXHAUSTED, and the difference is what you do next. Exhausted means the policy did its job and ran out of candidates — raise max_attempts, add a region, or accept that the region has no room. Abandoned means the driver went away mid-walk, so the run tells you nothing about whether capacity existed. Just re-run it. Note the missing in that row of the table: an AWAITING_CAPACITY row whose every expected series is already in BigQuery is refused rather than failed, because the work plainly happened whatever the row says.
Settle never deletes and never invents time. The landed data is untouched; the row gets a status, a failure_reason when there is a token for it, and an audit blob. It does not stamp ended_at or runtime_seconds — a row settled three days after the fact would report three days of runtime for a ten-minute job.
LIKELY_COMPLETED settles to COMPLETED, and that arm has to exist. A Ray persistent resource is collected when its job finishes, so "no record of the job, and all the work is in BigQuery" is the normal trace of a successful Ray run whose driver died before closing the row.
It leaves the run header alone. registry.ops close-runs owns that tier, and settling the job rows is what unblocks it (it refuses any run with a non-terminal row). The report prints the hint.
A row that reached a terminal status on its own wins. Every settle write is guarded against all four terminal statuses, because the process that ran the job outranks a repair verb. That skip is silent, so the rows are re-read afterwards and each outcome reports the status it actually ended up with — which is what the headline counts.

The audit trail lands under run_jobs.job_telemetry.$.settle: settled_by (resolved from ADC unless you pass actor), settled_at, your reason, plus the whole of the evidence — from_status, verdict, native_state, n_done / n_expected. A settled row is the one row in the registry whose terminal status was written by somebody who was not running the job, so it carries enough for a reader a week later to re-derive the decision and disagree with it.

Cancel safely¶

python -m scale_forecasting.main --config CONFIG --cancel                      # preview only
python -m scale_forecasting.main --config CONFIG --cancel --force \
    --reason "superseded by the re-tuned config"                               # execute

Forecaster(cfg).cancel()  # preview
Forecaster(cfg).cancel(confirm=True, reason="…")  # execute

Preview is the default. Without --force / confirm=True, cancel probes, prints the full blast radius — every family, its runtime state, how many series have landed — and touches no runtime and no registry. Read it before you commit; it is also the fastest way to see what a run has actually produced so far.

On execute, each cancellable family's runtime job is stopped, its row is finalized to CANCELLED with an audit blob, and the run header is rolled up. --job FAMILY narrows the whole operation to one family.

Three behaviours that surprise people:

A family already terminal is untouched — CANCELLED counts as terminal, so cancelling twice is a no-op rather than an error.
The ensemble node is suppressed when any base family is cancelled, and the preview says so. An ensemble over a truncated base set would be quietly wrong, so it is skipped rather than computed on partial input.
The header is only rolled up when every family has settled. All-CANCELLED ⇒ the run is CANCELLED; a mix of CANCELLED and other terminals ⇒ PARTIAL. If any stop failed, the header is deliberately left alone — a run whose jobs are not all settled is never finalized.

What a cancel keeps¶

Cancel never deletes. Every cell, metric row and artifact that had already landed stays exactly where it is. What changes is the label: the family's status becomes CANCELLED and its n_done / n_expected counts record how far it got, so the partial data is identifiable rather than masquerading as a complete run. The dedupe-on-read views mean a later full re-run of the same config supersedes it without double-counting.

The audit trail lands under run_jobs.job_telemetry.$.cancel and records both intent and observed reality: cancelled_by (resolved from ADC unless you pass actor), cancelled_at, your reason, plus native_state_at_cancel and n_done_at_cancel — what was actually stopped, not just what was asked for.

To remove the partial data as well, that is a separate, explicit step: python -m scale_forecasting.registry.ops drop-run RUN_ID (see operations.md). Keeping the two apart is the point — stopping a run and discarding its results are different decisions.

Common shapes¶

Symptom: a bar has been quiet for a long time. Probe. RUNNING_CONFIRMED ⇒ it's slow, wait. LOST ⇒ it died; --settle --force finalizes the row FAILED, then re-run. Families that write their cells at job end are legitimately quiet for their whole run, which is why an age alone is never a verdict.

Symptom: the run header says RUNNING but nothing is happening anywhere. Probe. Expect STALE_REGISTRY or LIKELY_COMPLETED — the work finished but the finalize write was lost. --settle --force writes the rows to what the runtime says: the landed results are intact and, on LIKELY_COMPLETED, complete, so the rows go COMPLETED. Do not cancel here — cancel would stamp CANCELLED over work that succeeded, and the label is the only thing left to get right. Once every job row is terminal, registry.ops close-runs finalizes the header to what those rows say. It skips any run whose job rows are still unsettled, which is exactly what settling them unblocks.

Symptom: a header is stuck RUNNING and the run has no job rows at all. The driver died in the submit path before it recorded a single family, so there is nothing to probe. registry.ops close-runs closes it as FAILED; nothing completed and nobody stopped it.

Symptom: every job row says COMPLETED, but a family you configured has no row at all. The driver died between families — after the last one finished, before the next was submitted — so the missing family left nothing behind to notice its absence by. close-runs catches this by comparing the rows against the families the run's own config planned, and closes the header to what the run would have written: FAILED when the gap is the ensemble (you asked for an ensemble and there isn't one), PARTIAL when a base family is missing and the others landed (their forecasts are still usable). Nothing is deleted either way, and the predictions that did land stay queryable.

Symptom: no handle recorded. The job predates the probe surface, or its telemetry blob is malformed. There is no way to address the runtime job, so the verdict is UNKNOWN and cancel reports the family as not cancellable. Stop it in the Cloud Console and use registry.ops to tidy the row.

Symptom: registry.ops drop-run refuses. It will not touch a run whose header is still RUNNING or PENDING. Probe first — a RUNNING row can also be a dead job — then settle the rows (--settle --force), cancel it properly if it really is live, close the header with close-runs, or pass --force. Reach for drop-run --force only when you actually want the run's data gone; if all you want is an accurate status, --settle plus close-runs gets there without deleting anything.

Capacity — the cloud has no room¶

A cluster create that fails for want of machines is not a bug in your config, and it is the one failure worth re-submitting unchanged. It gets its own state so it stops looking like a hang and stops looking like a crash.

A family sits at `AWAITING_CAPACITY`¶

Symptom: monitor shows a job that is neither RUNNING nor terminal, sometimes for many minutes. Cause: the launcher is walking candidate regions/zones looking for one that can provision the cluster. This is a live, non-terminal state — the work has not started and has not failed. Fix: usually nothing; wait. Two things are worth knowing:

The row's capacity ledger says exactly what has been tried. Read it from v_run_jobs:

SELECT family, status, failure_reason, capacity FROM `PROJECT.DATASET.v_run_jobs`
WHERE run_id = 'RUN_ID'

Each attempt carries its candidate, the verdict, and the cloud's verbatim message. A shared cluster is provisioned before any job row exists, so its ledger is on the run header instead — same key, in v_run_summary.capacity, one entry per service. - --cancel cannot interrupt a walk. The loop runs inside the submitting process, so there is no server-side thing to cancel: stop the launching process, or let the budget expire. The row itself is addressable — it carries a handle like any other — so cancel will try, and report that it stopped nothing. That is honest rather than a failure.

A job FAILED with `failure_reason = CAPACITY_EXHAUSTED`¶

Symptom: a family went terminal without ever running. Cause: the walk used up its attempt budget or its wall-clock budget without finding room. The finished ledger is still on the row. Fix: re-submit — often unchanged, since a stockout is transient. If the ledger shows HARD_CEILING on every candidate, waiting will not help: that is a quota this project is not allowed past, so either request more quota or ask for less (fewer GPUs, a different accelerator). Verdicts, and what each one does to the walk:

Verdict	Means	Effect
`TRANSIENT_CAPACITY`	No machines right now.	Hop to the next candidate, and come back to this one after the back-off.
`HARD_CEILING`	A quota you cannot exceed.	Hop, and drop this candidate from later passes — waiting cannot raise a quota.
`CONFIG_FAULT`	The request itself is wrong (bad machine type, retired image).	Stop immediately and re-raise. Another zone cannot fix it.

An unrecognised message is treated as transient, deliberately: the cost of one extra hop is smaller than the cost of abandoning a run over a phrasing nobody had seen yet.

The wait is too long, or too short¶

Fix: compute.capacity sets per-service patience — max_attempts, max_passes, max_wall_seconds, and the back-off. {"enabled": false} collapses every service to a single pass, which is what you want when you would rather fail fast than wait. None of it moves the run_id: how patient you are is an operational knob, not part of what you asked for. See the compute block in configuration_reference.md.

Ray on Vertex¶

Ray job submission hangs, or HTTP 524 on `/api/version`¶

Symptom: the driver connects but job submission never returns, or the client gets a 524 timeout hitting the cluster's /api/version. Cause: the cluster needs both a PSC-I network attachment and a sufficiently large head node (n1-standard-16+). Either one alone still 524s — it is a cluster-side capacity/networking issue, not the client's location. Fix: ensure the Terraform network attachment is applied and compute.ray_head_machine_type is n1-standard-16 or larger. See deploying_on_gcp.md.

Ray job fails at env setup — `No matching distribution for torch==…+cu126`¶

Symptom: the cluster comes up and the job is submitted, but it goes straight to FAILED with runtime_env setup failed and, in the job logs, ERROR: No matching distribution found for torch==2.13.0+cu126. No cells are written. Cause: the x86_64/linux torch pin is a CUDA local build (+cu126) that exists only on the PyTorch wheel index, not PyPI. The Ray runtime_env uv install needs the same --extra-index-url the container image build uses; without it — and without letting uv prefer that index — the pinned wheel can't resolve and the whole job dies before any work runs. Fix: handled — build_runtime_env() passes --extra-index-url https://download.pytorch.org/whl/cu126 and --index-strategy unsafe-best-match in the uv plugin's uv_pip_install_options (kept in lockstep with docker/Dockerfile, with PyPI still primary; unsafe-best-match is what lets uv pick the +cu126 build from the extra index even though a same-named wheel exists on PyPI). If you re-pin torch to a different CUDA minor, update the URL in both places.

Ray driver OOM — no traceback, killed at "uploading package"¶

Symptom: the run dies with no Python traceback, often right after logging the package upload. Cause: the driver/orchestrator RSS outgrew a small VM. An OOM-kill leaves no traceback. Fix: run the orchestrator on a VM with ≥64 GB (e.g. e2-highmem-8). See operations.md.

Long Ray run 401s after ~60 minutes¶

Symptom: a run polling for status starts returning 401 Unauthorized well into the run. Cause: the bearer token in the poll loop hit its TTL (~60 min). Neural Prophet on GPU is the long pole that pushes total runtime past the expiry. Fix: handled — the status poller does a single-retry re-auth on a 401. If you see a hard 401, confirm ADC is still valid on the orchestrator.

Regional GPU capacity stockout¶

Symptom: cluster create fails to obtain T4s in a region. Cause: transient regional accelerator stockout. Fix: compute.ray_regions is the candidate list the launcher walks, and the walk retries the whole list rather than giving up after one pass. See Capacity for the state it reports while it waits, the ledger it keeps, and how to tune the patience. Add more regions if one is chronically short.

Ray cluster won't shrink when idle / can't grow under load¶

Symptom: the expensive GPU pool stays allocated while idle, or the CPU pool can't grow to work through the queue. Cause: the run used fixed-size sizing (ray_autoscale=false). Fix: autoscaling is the default now (ray_autoscale=true), with independent per-pool [min, max]. Set a low ray_gpu_min_nodes to shrink idle T4s and a high ray_cpu_max_nodes to grow under load. See the compute block in configuration_reference.md.

Spark on Dataproc¶

100k Spark run OOM — exit 137 → FetchFailedException¶

Symptom: executors killed with exit 137, cascading into FetchFailedException at large scale. Cause: bucket sizing. Sizing buckets by max_parallelism silently fattened each pandas frame until executors OOM'd. Fix: buckets are sized by compute.bucket_target_cells (default 8) — buckets = ceil(cells / target) bounds per-task memory at every scale. Lower bucket_target_cells if a heavy model set still OOMs. See configuration_reference.md.

When a profile is available the target is applied per model rather than fleet-wide: a model measured slower than the run's typical cell gets a proportionally smaller target and therefore more buckets, so a bucket is closer to a fixed unit of work instead of a fixed number of cells. A model measured faster keeps the global target exactly — the weights are floored at 1.0, so this can only make frames smaller, never larger. The run's executed fan-out record (sizing_executed.<family>.fanout) carries cost_weights and allocation, which is where to look when one stage's tasks finish long after the rest.

100k batch cancelled at ~4 hours¶

Symptom: a large batch is cancelled around the 4-hour mark. Cause: the default Dataproc batch TTL. Fix: handled — the submitter sets an explicit 24h batch TTL.

Registry / writes¶

`RegistryError: Storage Write API … 500/503/429`, or `400 Cannot route on empty project id`¶

Symptom: intermittent write failures over long, high-volume runs. Cause: transient backend/routing blips on the Storage Write API over sustained streams. Fix: handled — _append_via_write_api retries with exponential backoff (all engines route writes through it). A genuine 400 (real bad request) still fails fast.

A submit is refused — the job id is already taken¶

Symptom: a launch fails immediately with ALREADY_EXISTS from Dataproc, or "job with submission_id … already exists" from Ray. The registry row for it says failure_reason = JOB_ID_TAKEN.

Cause: two different things decide the same name. The attempt number in a job id comes from the registry — MAX(attempt) over run_jobs for that run and family — while the uniqueness of the id is enforced by the platform. Those two agree only while every job that reached a platform also left a row behind, and one path breaks that: a command that stage, plan or the Airflow emitter printed is run by you, not by the launcher, so the job exists on Dataproc or Ray while the registry never hears about it. The next --force reads the same MAX(attempt), hands out the same id, and the platform refuses it.

Fix: nothing, usually — a forced launch now asks the platform whether the id it is about to stamp is free, and walks the attempt forward until it finds one that is. You reach this error only when that check could not get an answer (a degraded probe declines to walk rather than inflating the attempt on a flaky read) or when twenty consecutive ids were taken. Either way the move is the same: re-run with --force. To see what the platform actually holds before you do, probe the run:

python -m scale_forecasting.main --config CONFIG --probe

What you will see in the registry. Staging now files a row for every job id it hands out, with status EMITTED — "this id has been spent, whether or not anyone ran the command". It is a live status, so it holds a repair back the way a RUNNING row does, and it is why the counter no longer loses track of a pasted command. Two things clear it. If you run the command, the launch writes a real row at the same attempt and the serving view v_run_jobs takes the later one, so the EMITTED row disappears from every read with no cleanup. If you never run it, --settle reaps the row to FAILED with failure_reason = NEVER_LAUNCHED once the platform confirms the id was never created — spelled apart from RUNTIME_LOST because nothing ran, so there is nothing to go looking for.

A staged run has a header too, and its status is STAGED. Staging a config that has never run opens its run_registry header, because --probe, --settle and --retry read a run's expected work off that header and skip the run entirely when it is missing — without one they could not see the EMITTED rows at all. The status is STAGED rather than RUNNING because nothing is running: the artifacts are up and the job ids are spent, but no compute exists. It is non-terminal, so --settle and close-runs still treat the run as open work, and it is a live status, so drop-run will not quietly delete a run whose command you may still be holding. When you finally paste the command, the launch appends its own RUNNING header and the newest row wins.

Your next command on that config will say "already staged", not "new run" and not "already ran". An unforced re-stage reuses the attempt the staged command was told to use, which is what you want if nobody pasted it — the commands you are already holding stay valid. --force spends the next attempt, which is what you want if somebody might have.

An immediate re-run double-counts rows¶

Symptom: re-running the same config right away appears to duplicate rows. Cause: the Storage Write API streaming buffer (~90 min) blocks a same-key DELETE, so the design is append-only + dedupe-on-read, never delete-then-write. Fix: this is by design and the serving views dedupe on run_id. If you want a physically separate run, use a fresh/timestamped run_name (which changes the run_id). See output_schemas.md.

Install / extras¶

`MissingExtraError: ... needs the 'gcp' extra, which is not installed`¶

Symptom: main.run, Forecaster.run(), a python -m scale_forecasting.*_submit command, --probe, --dry-run --feasibility, registry.ops, or a plot call raises MissingExtraError (also an ImportError) naming an extra — gcp, notebook, ray, or models-automl. Cause: pip install scale-forecasting installs the pure offline layer only (config, every model and metric, the planner, the playground). Everything that talks to Google Cloud lives behind [gcp]; plotting behind [notebook]; the Ray client behind [ray]; the AutoML pipeline components behind [models-automl]. Each entry point checks once, up front, so the failure is a sentence with the install line rather than a ModuleNotFoundError from inside a client library. Fix: install the extra the message names — pip install "scale-forecasting[gcp]" (or, from a clone, uv sync --extra gcp) — and re-run. The full extra-by-extra table is in runtime_dependencies.md.

`ModuleNotFoundError: No module named 'google'` from a reading or review method¶

Symptom: a core-only install raises a raw ModuleNotFoundError (not MissingExtraError) from Forecaster.predictions_df(), Registry, or another method that reads BigQuery. Cause: the read path constructs a BigQuery client lazily at the call site; only the launch, submit, ops, and plotting entry points carry the guided check. Fix: same remedy — pip install "scale-forecasting[gcp]". A notebook or workstation that both launches and reviews wants [gcp,notebook]; the per-surface install table is in runtime_dependencies.md.

`list_models(available_only=True)` returns 15, not 37¶

Symptom: the catalogue lists all 37 models but only 15 report as available. Cause: availability is honest about what is importable: the 15 are the models whose library is in the core install (statsmodels, scikit-learn, scipy). The rest declare an optional_extra — models-stats, models-trees, models-prophet, models-dl, models-foundation, models-automl for the four Vertex AI AutoML models, or gcp for the two BigQuery-native models. Fix: install the family extras you intend to run ([models] is all six model extras, AutoML included) plus [gcp] for the BigQuery-native pair. The model→extra map is the "Granular installation extras" table in models_reference.md.

Versions / runtimes¶

`PYTHON_VERSION_MISMATCH`, or "why can't I move to 3.12?"¶

Symptom: a version-mismatch error between driver and workers, or a desire to bump to 3.12. Cause: the whole system is pinned to Python 3.11: Vertex Ray caps at 3.11, and Spark Connect uses runtime 2.3 (not 3.0). 3.11-everywhere is the only configuration that satisfies all runtimes. Fix: stay on 3.11. Full rationale and the per-surface matrix: version_matrix.md.

`libgomp1` missing, or neuralprophet + pandas-3 breakage¶

Symptom: an import/link error for libgomp1, or neuralprophet failing against pandas 3. Cause: neuralprophet needs the OpenMP runtime, and it is incompatible with pandas 3 (Series.view removal). Fix: handled — the image includes libgomp1 and pandas is pinned <3. neuralprophet is an optional extra; the registry registers-and-skips it when unavailable.

Deploy / Terraform¶

Cloud Shell `Bus error` running terraform¶

Symptom: terraform dies with a Bus error. Cause: the Cloud Shell disk filled up. Fix: disk-hygiene first (clear caches / large artifacts), then retry. See disk-hygiene in operations.md. For long runs, prefer a persistent VM over Cloud Shell.

`google_dataproc_batch` apply errors, but the batch SUCCEEDED¶

Symptom: terraform/apply reports an error after submitting a batch that actually ran fine. Cause: the provider's client-side wait raced the batch; the batch's state in GCP is the source of truth. Fix: treat the GCP batch state as authoritative; terraform import reconciles the resource.

Notebooks¶

`--ack-out-of-org` FAILED_PRECONDITION on notebook fan-out¶

Symptom: the notebook fan-out step fails a precondition for a project not in the corp org. Cause: the out-of-org acknowledgement flag applies only to corp-org projects. Fix: for a non-corp-org project, the flag is not applicable — see the headless/Colab specifics in notebook_runtimes.md.