Quota and scale¶

How much GCP quota do you need to forecast N series? This page answers that with numbers measured on real runs, not with a rule of thumb.

The short version: this product is quota-bound long before it is architecture-bound. A default GCP project gives you enough headroom to run tens of thousands of series comfortably. Past that, every additional unit of parallelism is a quota request, and nothing in a config file can substitute for one. The point of this page is to let you file that request before you discover you needed it, with a number you can defend.

1. The arithmetic¶

Three quantities, and everything follows from them.

flowchart LR
    Workload["Series × Models<br/>= Cells"] --> Fits["Cells × (Folds + 1)<br/>= Total Fits"]
    Fits --> Split{"Family Placement<br/>(compute.families)"}
    Split -->|"Spark, GCE Single-VM & GKE"| GCE["Compute Engine Quota<br/>CPUS · NVIDIA_*_GPUS"]
    Split -->|"Ray (Vertex), Vertex CustomJob & Vertex AutoML"| VTX["Vertex AI Quota<br/>custom_model_training_cpus<br/>custom_model_training_nvidia_*_gpus"]
    Split -->|"BigQuery-native"| BQS["BigQuery Slots<br/>(Zero VM / GPU Quota)"]

Cells. The unit of work is one cell — one model fitted to one series.

cells = series x models
fits  = cells x (folds + 1)

10,000 series with 6 models is 60,000 cells. The + 1 is the one people miss. Each backtest fold is a fit on truncated history, and then the model is fitted once more on all of it to produce the forecast you actually ship. So n_folds: 2 is three fits per cell, not two — and it costs about three times a no-backtest run, not twice.

Measured, on the same config with and without backtesting (all_families_10k vs all_families_10k_full, 2026-09-04/05): the deep-learning family went 6,582 s → 18,372 s, 2.79x. Watch that number; it is the one that sets your bill and your wall clock.

It came in under 3.0 for a reason worth knowing: a fold fits on truncated history, so the fold fits are cheaper than the final one. --feasibility prices that exactly for your own panel and reports it as full_fit_equivalents — 2.94 rather than 3.0 on four years of daily history at two folds. Use folds + 1 to size a fleet and the equivalents figure to forecast a bill. Folds never multiply the cell count; see the configuration reference.

Throughput. Cells are independent, so they run in parallel and throughput is linear in the size of the fleet. What matters is throughput per node, measured end to end:

wall clock = cells / (nodes x cells-per-minute-per-node)

Quota. Nodes come out of a regional CPU allowance, and GPU nodes additionally out of a per-accelerator allowance. Those two numbers are the ceiling on nodes, and therefore on how fast any of this can go.

vCPUs needed = (worker nodes x vCPUs per worker) + vCPUs for the head/driver node

2. What one node actually delivers¶

Measured, not estimated. Every row below is the same shape — 100,000 series x 4 models = 400,000 cells on Ray, 20 x n1-standard-8 workers — and the anchor is ray-100k-3fbc82fe3b6d, the run the validation ledger currently holds for that config. Its predecessor ray-100k-dcc77a9d1e9b is superseded, and is shown alongside it to illustrate the impact of per-core vs per-node slot scheduling (§5).

Run	Cells	Wall clock	Cells/min	Per node
2026-09-03, pre-fix (superseded run)	400,000	19,069 s (5 h 18 m)	1,259	63
2026-09-05, post-fix (superseded run)	400,000	6,290 s (1 h 45 m)	3,816	191
2026-09-10, current architecture (`ray-100k-3fbc82fe3b6d`)	400,000	7,405 s (2 h 3 m)	3,241	162

So for a mixed statistical + ML workload on 8-vCPU nodes:

160–190 cells per minute per 8-vCPU node, and plan at ~162 — about 20 per vCPU per minute.

Use the low end of that range, not the middle and not the top. The last two rows are the same config on the same hardware, and they are 18% apart. That spread is what an autoscaled pool does: the fleet starts at ray_cpu_min_nodes and climbs toward ray_cpu_max_nodes, so how much of a run is spent below full width differs from attempt to attempt. Neither number is wrong. There is simply no single number here, and a plan built on the fastest attempt anyone ever observed will come up short.

All three rows are total cells over total wall clock, which is the figure that plans a run. On 2026-09-05 the pool held 139–140 of its 140 cores busy for its hour and three quarters, with up to 38,000 tasks queued behind it.

Per-cell fit times behind that figure, from the 2026-09-05 run:

Model	Family	Avg fit	p95 fit
`holtwinters`	statistical	0.41 s	0.49 s
`theta`	statistical	0.42 s	0.87 s
`xgboost`	ml	0.75 s	0.88 s
`sarimax`	statistical	1.68 s	2.41 s

Your mileage will vary with the model mix, and sarimax is the reason. It is 4x the cost of theta on the same data. A theta-only run is much faster per cell than the table above; a run dominated by sarimax is slower. Series length matters too — these were ~4 years of daily observations.

Deep learning is a different regime entirely. neuralprophet measures at 21–65 s/fit against sub-second statistical models — 50x or more. What it does not need is a GPU. At the shipped n_lags=0 configuration the model is a few hundred parameters of trend and Fourier seasonality, so it allocates about 87 KB of device memory — 0.0005 % of a 16 GiB T4 — and spends 93–99.6 % of its wall clock saturating a single CPU core. Two head-to-head A/B runs measured what that costs, and the CPU arm was not merely as fast as the GPU arm; it was faster. The comparison is worked through in §3.

Both deep-learning anchors on this page are therefore per node, never per device:

`n1-standard-8` node (7 usable cores)	`neuralprophet` fits/min per node	Measured on
CPU	10.1	`neuralprophet-ab-cpu-f4bfff3b39e9` (2026-09-11)
GPU, one T4 attached	7.9	`neuralprophet-ab-gpu-e530eea3a755` (2026-09-10)

Both are 30,000 fits over the run's full wall clock across twelve nodes — the same end-to-end measure as the 162 cells/min figure above, so the two are comparable. The 7.6 and 8.2 figures elsewhere on this page are the same GPU-node measurement taken on all_families_10k and all_families_10k_full, and they agree with the 7.9 here. The arithmetic in §3 is built on all of them.

Every GPU figure on this page is conservative. With GPU worker calibration packing seven cells onto a card (all-families-10k-a0f6797d69c1), a mixed-family run of 10,000 fits measures 9.0 fits/min per node — faster than the 7.6–8.2 baseline in the table above. So no GPU projection here will leave you short; if anything you will ask for slightly more quota than you need. The 10.1 and 7.9 figures remain the controlled A/B anchors (30,000 fits whose two arms differed by a single line of config).

3. The table¶

Assumptions: 6 models, 1 fold (so cells = series x 6), the mixed statistical + ML profile measured above, n1-standard-8 workers (8 vCPU) and an n1-highmem-32 head node (32 vCPU). The head node is a fixed cost you pay once per run.

Series	Cells	Wall clock on a default 200-vCPU project (20 nodes)	Nodes for a ~1-hour run	vCPUs to request
10	60	seconds	1	40 — default is fine
100	600	seconds	1	40 — default is fine
1,000	6,000	~2 min	1	40 — default is fine
10,000	60,000	~16 min	6	80 — default is fine
100,000	600,000	~2.6 hours	53	~460
1,000,000	6,000,000	~26 hours	524	~4,225

Add the start-up floor to the top two rows

Provisioning a cluster takes 10–15 minutes regardless of how little work you then give it. At 10 or 100 series the compute is instant and the wall clock is essentially all start-up. That floor does not scale, so it disappears into the noise by 10,000 series — but it is why a tiny run does not feel fast, and why iterating on config is better done against a few hundred series than a few dozen.

Read it as two separate questions.

"Can I do this at all today?" — column 3. A stock project runs 10,000 series in about a quarter of an hour and 100,000 series over an afternoon. Both are useful; neither needs a support ticket. This is why the demonstration configs in configs/ top out at 10,000 series: it is the largest scale that a default project runs comfortably, which makes it the largest scale a reader can reproduce.

"How much quota to make it fast?" — columns 4 and 5. Wanting 100,000 series in an hour rather than two and a half means asking for roughly 460 vCPUs, a bit over 2x a default allowance — small enough to be a routine request. A million series in an hour is ~4,200 vCPUs and is a conversation with your account team, not a form.

The relationship is linear in both directions, so interpolate freely: halve the wall clock, double the quota.

Deep learning is on a separate, much lower ceiling¶

neuralprophet fits ~50x slower than a statistical model, so the deep-learning family is the hardest limit in this product wherever you put it. On CPU nodes it draws on the same regional vCPU allowance as everything else — the looser of the two ceilings, and the shipped default. Route it to GPU and it moves onto an accelerator allowance roughly an order of magnitude tighter, and which allowance depends on the runtime:

Deep-learning runtime	Quota metric	Typical default
Spark on a Dataproc cluster / Compute Engine (`gce`)	`NVIDIA_T4_GPUS` / `GPUS_ALL_REGIONS` (Compute Engine)	4
Ray on Vertex / Vertex AI `CustomJob` (`vertex`)	`custom_model_training_nvidia_t4_gpus` (Vertex AI)	12

These are separate pools in separate services. Raising one does nothing for the other, and the gcloud compute regions describe recipe below shows only the first — see §4 for how to read the Vertex one. The Vertex allowance is also far more regionally uneven than the Compute Engine one: on the project used for the validation ledger it is 12 in us-central1 but 2 in both us-east1 and us-west1, so a failover region may be a third of the capacity you sized for.

At 7.6 fits/min per GPU node and 10.1 fits/min per CPU node — both measured, see below — one deep-learning model over the same series counts, without backtesting (with n_folds: 2, multiply by three):

Series	On 4 GPU nodes (Dataproc T4 default)	On 12 GPU nodes (Ray T4 default)	On 20 CPU nodes (reference-config cap)	Nodes for a ~1-hour run, CPU / GPU
100	~3 min	~1 min	under a minute	1 / 1
1,000	~33 min	~11 min	~5 min	2 / 3
10,000	~5.5 hours	~1.8 hours	~50 min	17 / 22
100,000	~55 hours	~18 hours	~8.3 hours	165 / 219
1,000,000	~23 days	~7.6 days	~3.4 days	1,650 / 2,193

Read the CPU column first, because it is the shipped default and it wins twice. It is faster per node, and its nodes come out of a far looser pool: 20 CPU workers is where the reference configs cap themselves to stay inside a 200-vCPU Compute Engine allowance, and on Ray the same fleet bills to custom_model_training_cpus, whose us-central1 default is 2,200. Getting to 165 CPU nodes is a routine vCPU request. Getting to 219 T4s is not.

A default project runs out of deep-learning headroom somewhere around 10,000–20,000 series for a run you are willing to sit through, an order of magnitude before it runs out of headroom for the statistical and ML families. This is why all_families_10k.json is the largest config here that includes neuralprophet. If you do route the family to GPU, it pins ray_gpu_max_nodes: 12, the Vertex default for us-central1; it previously pinned 4, which was the Compute Engine number applied to a Ray run by mistake and left two-thirds of the allowance unused. If you deploy to a region with a smaller Vertex allowance, lower it — the pool starts at ray_gpu_min_nodes and a max above your quota is not an error, just a ceiling the autoscaler never reaches.

Where the two anchors come from, and why neither is "per T4". The GPU figure replaces an extrapolation from runs of 100 series or fewer, in which cluster start-up was a large fraction of the span. On 2026-09-04 all_families_10k.json fit 10,000 neuralprophet series across a twelve-node T4 fleet in 1 h 50 m — 91 fits/min for the fleet, 7.6 per node. The 10,000-series GPU row above is therefore not a projection; it is that run. all_families_10k_full.json re-measured it the next day at 3 fits per cell and landed at 8.2, which is the same number: the anchor is per fit, and backtesting buys more of them. Both were once written "per T4" because each node carried exactly one card — but the card is not what produces the rate, so the label was wrong and the arithmetic was right. The 2026-09-10 A/B GPU arm, on the same twelve-node shape, independently gives 7.9 fits/min per node.

The CPU figure is new, and the project had no measurement of it until the A/B ran. neuralprophet-ab-cpu-f4bfff3b39e9 landed 30,000 neuralprophet fits across twelve n1-standard-8 CPU nodes in 14,828 s on 2026-09-11 — 121 fits/min for the fleet, 10.1 per node — against neuralprophet-ab-gpu-e530eea3a755's 19,054 s for the same 30,000 fits on an identical fleet shape with a T4 attached to every node. Same config but for one line, 10,000 cells landed on both arms, WAPE identical to ten decimal places.

The same comparison was then run on a Dataproc cluster, and it agreed. neuralprophet-ab-cluster-cpu-a402414abf5e and neuralprophet-ab-cluster-gpu-273d32b553c8 (2026-09-14/15) put 3,000 cells through four workers each: 158.9 minutes of fitting on CPU against 188.0 on GPU, 27.6 s/fit against 31.7, mean WAPE 0.3485 on both. Different scheduler, different fleet size, different quota pool, same direction and the same decision. A result on one runtime is a result about that runtime; two runtimes disagreeing would have meant the finding was about the platform rather than about the model. All four arms are written up in the validation ledger under the NeuralProphet accelerator A/B.

Four things about these anchors are worth understanding before you plan with them.

They are throughput figures, not latency figures. The average individual fit took 43.5 s on the 2026-09-04 run, which is 1.4 fits/min if you watch a single series — roughly seven fits share each node, and they contend. Sizing from the per-fit time will over-provision you by about 5x.

Your GPU pool will run out of cores before it runs out of GPU. A deep-learning task asks for one vCPU and a fraction of a device, and on the default n1-standard-8 GPU worker the vCPUs run out first: 12 nodes give 84 usable cores, so 84 fits run concurrently and the T4s sit at 8.4 of 12 busy — 70 % — for the whole run. That is not a defect and not the memory bug from §5; it is the machine shape.

It stays true on a wider machine, and that took a second fix. The fraction a task requests has a floor, because a card split into slivers is a card nobody is really using. That floor used to be a flat 0.1 — ten cells per device, whatever the device was bolted to — and since NeuralProphet's measured footprint is about six millionths of a T4, every auto-calibrated run landed on it. On an eight-core worker nobody could tell, because seven cores are tighter than ten cells anyway. On a sixteen-core worker the constant would have been the limit: ten cells, five cores idle, and the run record naming device as the axis to go and fix, when the only thing that could move the number was the machine type. The floor is now derived from the node's own cores, so a wider GPU worker buys the density you paid for. Whether you should buy one is the question the next two paragraphs answer, and for neuralprophet the answer is no.

But "8.4 of 12 busy" is reservation accounting, not work. It is Ray reporting how much of the declared GPU resource is claimed by running tasks, and every one of those claims is a gpu_fraction reservation held by a model that has ~87 KB resident on the card. The remaining 30 % is not idle silicon you could put to work; the 70 % is not busy silicon either. Earlier versions of this page read that number as under-utilised hardware and told you to raise ray_gpu_machine_type so each card could be fed more tasks. That advice is withdrawn: feeding more tasks to a device that is doing ~nothing buys nothing. The measurement that settles it is the A/B above, where taking the card away cut 22 % off the wall clock. If your deep-learning family is neuralprophet at these hyperparameters, the lever is not a bigger GPU worker — it is hardware: "cpu".

And this is neuralprophet at this data's shape. A different model, a different device, or much longer series will move it. The rate is the right starting point precisely because it is now measured rather than guessed; it is still your own first run that tells you your number. Plan with these, then measure.

What `hardware: "gpu"` guarantees, and what it does not¶

Setting a family to GPU is a statement about placement, and the product enforces it. It is not a statement about utilisation, and no platform can enforce that one — utilisation is a property of your model, not of the machinery around it. Keeping the two apart is what lets this page say "the card was reached" and "the card was wasted" about the same run without contradicting itself.

What the contract guarantees. Three things, checked at three different moments:

Guarantee	When	Mechanism
Provisioning coherence — the hardware a job is provisioned onto is the hardware its cells are routed to	plan time, offline	both read one resolver, `resolve_family_compute`, so a fleet and its cells can never disagree. `use_gpu: true` with `families.deep_learning.hardware: "cpu"` therefore buys no accelerator at all, rather than buying one and not using it
Fail fast when the device is not there	the first cell, on the worker	`worker._require_device`. A cell told `hardware="gpu"` that finds no CUDA device raises immediately, with the service named and the fix spelled out, instead of quietly fitting on the CPU and handing you a correct forecast and a GPU bill
Placement is recorded, never inferred	every cell	the model writes `device_used` (`cuda` or `cpu`) and `peak_gpu_bytes` onto its `forecast_metadata` row. The verdict below is computed from what was recorded, not from what was requested

What it does not guarantee: that the device does any work. Once a family job's cells are written, device_audit aggregates those recorded facts into one verdict for the job:

Verdict	Meaning
`MISSING_DEVICE`	the family asked for a device and no cell reports having run on one
`ENGAGED_IDLE`	cells ran on the device and barely touched it
`ENGAGED_UTILISED`	cells ran on the device and used a real share of it — currently ≥ 1 % of the card's memory
`None`	the family never asked for a device, so there is nothing to judge

It warns; it never fails a job. An idle accelerator still produces a correct run. It is a cost finding, not a fault, and the product's job is to name it rather than to overrule you. The verdict lands on the family's run_jobs row and surfaces as device_verdict in review.py.

How the guarantee is proved, per service. Placement is proved in both directions on each of the three services that can carry a GPU:

Service	Positive arm — placement works	Negative arm — the guard fires
Dataproc Serverless	smoke 03 (L4)	smoke 17
Dataproc cluster	smoke 06 (T4)	smoke 18
Ray on Vertex	smoke 08 (T4)	smoke 19

The negative arms hide the card from the worker through the environment seam each service already exposes, and require the run to refuse. They exist because three green positive arms are indistinguishable from assert True without them. Smoke 20 covers the opposite mistake — use_gpu: true against a family overridden to cpu, which must complete on CPU having provisioned no accelerator. All four are in the validation ledger.

And the same runs measure the thing the contract does not guarantee. Every positive arm comes back ENGAGED_IDLE. That is the honest verdict for neuralprophet at n_lags=0 on all three services, and it is the observation the A/B above was run to price. The ledger records those GPUs as audited, not as used.

So the contract and the recommendation point in the same direction rather than against each other: hardware: "gpu" does exactly what it says, and for neuralprophet at these hyperparameters the right thing to say is hardware: "cpu". See the configuration reference for where that field lives.

4. Which quotas, and where¶

Quotas are per region, so a project with 200 vCPUs in us-central1 has a separate 200 in us-east1. Request them in the console under IAM & Admin → Quotas & System Limits, filtered to the region you deploy into, or with gcloud:

# What you have today
gcloud compute regions describe us-central1 \
  --project="${SF_PROJECT_ID}" \
  --format="table(quotas.metric, quotas.limit, quotas.usage)"

The metrics that bind this product:

Quota metric	What it limits	Typical default
`CPUS`	Every worker and head/driver vCPU in the region	200
`NVIDIA_T4_GPUS`	Deep-learning worker nodes on Dataproc	4
`PREEMPTIBLE_NVIDIA_T4_GPUS`	The same, for preemptible pools	4
`DISKS_TOTAL_GB`	Boot disks across the fleet	40,960
`IN_USE_ADDRESSES`	Rarely binding — clusters are private	69

There is no Dataproc capacity quota to look for. Every dataproc.googleapis.com quota is an API request rate, not an amount of hardware — including for Serverless, whose batch vCPUs bill to Compute Engine CPUS in the region. If a Serverless batch is capacity-starved, the meter to raise is the Compute Engine one above.

Ray on Vertex is not in that table, and gcloud compute cannot show it. Vertex draws GPUs from its own service quota, so a Ray run is bound by a metric the recipe above never prints:

# The quota that actually binds a Ray GPU run
gcloud alpha services quota list \
  --service=aiplatform.googleapis.com \
  --consumer="projects/${SF_PROJECT_ID}" \
  --filter="metric:custom_model_training_nvidia_t4_gpus" \
  --format=json

Read the effectiveLimit for your region out of quotaBuckets. Checking only the Compute Engine number is how you conclude you have 4 T4s on a project that has 12.

You do not have to run either command. --quota reads the same meters for the run you are about to launch, in every region it would consider, and prints what they allow:

python -m scale_forecasting.main --config configs/all_families_10k.json --quota

quota report for all-families-10k-eb01dcfecfab — candidate regions: us-central1, us-east1, us-west1
  native: on bigquery, no per-job capacity meter to read
  deep_learning on ray/gpu: gpu[1,12]
    us-central1: OK
      quota: gpu pool in us-central1 — aiplatform.googleapis.com/custom_model_training_nvidia_t4_gpus = 12, usable nodes = 12
        this run would saturate at 5000 nodes (10000 cells / 2 per node); the ceiling holds it to 12
        now   12 node(s): ~1h29m
        at    24 node(s): ~45m
    us-east1: UNREACHABLE — the PSC-I network attachment is in us-central1; ...

It reads only and launches nothing. The wall-clock projections appear when a measured profile exists for the family; without one you still get the ceiling and the throttle ratio.

A family on an ephemeral Dataproc cluster is reported the same way, against Compute Engine's meters rather than Vertex's, and over the zonal candidate list the Dataproc walk actually uses:

  statistical on spark/cluster/gpu: 8 worker(s) + 1 master
    us-central1: CLAMPED
      quota [CLAMPED] compute.googleapis.com/nvidia_t4_gpus allows 4 in us-central1, so the gpu
        pool tops out at 4 node(s) instead of 8
      the cluster would be built with 4 worker(s), not 8

Families on Serverless, on BigQuery, and on a cluster you told the run to reuse are named but not metered: a reused cluster already exists so nothing is being asked for, a Serverless batch has no fixed allocation to pre-read, and BigQuery has no capacity meter at all.

The launch path runs the same check by itself, before the first cluster create, on both the Ray and the Dataproc-cluster path. A region that cannot host the fleet is recorded as a hard ceiling and skipped without spending the ~12 minutes a failed Ray create costs; a region that can host a smaller fleet has this run's pool ceilings — or its physical worker count, on Dataproc — lowered to fit rather than failing. It never raises a floor: quota is evidence about what you are permitted, not about what the work needs. And it never changes run_id, so the same config is still the same run in a region that grants less. Set compute.capacity.preflight: false to skip it (the only reason to: a runner service account without serviceusage.services.get).

vCPUs are reported, never clamped. Compute Engine's regional vCPU pool is shared with every other VM in the project, so shrinking your cluster because something else is using the region is a decision the product leaves to you — you get the sentence, not a silently smaller fleet. The one exception is a region that cannot seat even a single worker, which is not a judgement call.

Two things that surprise people:

The head node counts. An n1-highmem-32 driver consumes 32 vCPUs before a single worker starts. On a 200-vCPU project that is 16% of everything you have, which is why the reference configs cap CPU workers at 20: 32 + (20 x 8) = 192, and 200 is the ceiling. This applies to the Dataproc path. Ray on Vertex bills its vCPUs to custom_model_training_cpus, whose us-central1 default is 2,200 — an order of magnitude looser, and the reason a Ray config can carry a 12-node GPU pool and a 20-node CPU pool at once (288 vCPU) without CPU ever binding.
Vertex's vCPU quota is per machine family. custom_model_training_cpus covers N1 and E2 only; n2, n4, c2, g2, a2, a3 and m1 machines each have their own metric (custom_model_training_g2_cpus, and so on). Reading the N1 meter for a G2 pool reports a number about machines your run is not using. The reference configs are all N1, so the default meter is the right one unless you change ray_machine_type.
GPU nodes consume both. Four T4 workers on n1-standard-8 cost 4 T4s and 32 vCPUs. Raising NVIDIA_T4_GPUS without also raising CPUS will not get you more GPU workers.

When you file the request, ask for the number in the table plus headroom for a concurrent run. Quota is granted per project and shared across everything in it — two runs at once need two runs' worth.

5. Cores are the unit, and it took a bug to find out¶

This section used to advise the opposite of what it advises now, and the history is worth two paragraphs because it is the reason to trust the current version.

On ray-100k-dcc77a9d1e9b, 37,500 tasks were queued against 20 eight-vCPU nodes and the resource plan expected 4–8 concurrent cells per node. Measured concurrency was 0.97 cells per node, flat across the entire steady state — roughly one core in eight doing arithmetic. With no explanation for that, the only safe advice was to buy nodes rather than cores.

The explanation arrived on 2026-09-04. A live read of a 10,000-series run showed every worker holding 1.0 of 7.0 CPUs and 17.6 of its 18.1 GiB of memory. Ray schedules on memory as hard as it schedules on cores, so a task asking for 97 % of a node's memory takes the whole node and the other six cores are unreachable. The request came from a driver-side sizing pre-pass charging each task the driver's memory footprint rather than a worker's. Fixed the same day, and the identical config re-run under the fix went from 0.93 to 5.5 concurrent cells per node — 5.9x the density and 3.8x the wall-clock at identical quota. ray-100k-dcc77a9d1e9b itself was re-run on 2026-09-05 and came back at 139–140 of 140 cores busy for 2.9x the wall clock, which is what re-anchored §2. Recorded as ray_slot_memory in the validation ledger.

So the advice inverts, and the numbers in this document are the post-fix ones:

Cores are the unit. A node's worth of throughput is its core count divided by the cores a cell asks for, and you should expect a fleet to reach it. Twenty n1-standard-8 workers and ten n1-standard-16 workers are now roughly equivalent for the same vCPU spend; prefer the larger node where per-task overhead matters, and the smaller one where a single cell's memory footprint is large enough that a big node would be under-filled anyway.
Do not size a quota request off node count alone. Ask for the vCPUs, then check the node shape divides into them sensibly.
Expect per-cell latency to get worse as density improves, and do not read that as a regression. The re-run above made each neuralprophet fit 44 % slower (30.25 s → 43.48 s) while the fleet did 4.1x the work. What got packed is the 8-vCPU node, not the card on it — these were GPU nodes, but the contention is seven fits sharing seven cores, which is exactly what a well-used node looks like. Read the per-fit number as evidence the fleet is full, not as evidence something broke.

If you see one busy core in N, suspect a memory request before you suspect the scheduler. The Ray dashboard's /api/cluster_status reports usageByNode, and a node pinned this way is obvious in it: cores idle, memory at the ceiling. The product now names the condition itself — RuntimeResourcePlan.binding_axis lands in the run's telemetry, and both the quota preflight and the Ray engine print a density note whenever memory and the scheduler disagree about how many cells fit on a node.

A footnote on how that note reads on Ray, because it is the opposite of what you might expect. The 2026-09-04 fix did not correct the memory request; it stopped making one. A Ray task now asks for cores and a GPU fraction and nothing else, which means Ray has no memory figure to schedule against and a memory bound cannot hold a Ray pool below its cores no matter how large the measured footprint is. The product still measures the footprint — it is the honest cost of a cell, and the thing to look at when a pool thrashes — so when the measurement implies a lower density than the one Ray will run, the note says so in as many words and tells you the figure is a footprint rather than a limit. Do not size a Ray fleet off it. On a Dataproc cluster or on Serverless it is a different story: an executor running N concurrent tasks really is bounded by its heap, memory binds there for real, and the same note reports it as the constraint it is.

6. Cheaper than more quota¶

Before filing for 460 vCPUs, four things cost nothing:

Run the deep-learning family on CPU. It is the shipped default and it is the single largest lever on this page, because the deep-learning family is ~96 % of a mixed run's wall clock when it is present. A GPU node costs roughly 1.8–1.9x a CPU node of the same shape, and both A/B runs found the CPU arm faster as well as cheaper: 2.34x the cost per thousand fits on Ray, 2.01x on a Dataproc cluster, at identical accuracy. It also moves the family off the tightest quota in your project and onto the loosest. Set compute.families.deep_learning.hardware to "cpu", or leave it alone — that is already the default. configs/per_family_runtimes_cpu_demo.json is a finished example to copy: it is the GPU demo config with that one word changed.

Cut cells, not corners. fits = series x models x (folds + 1). Dropping one expensive model from a 100,000-series run removes 100,000 cells. Backtesting with 2 folds costs ~2.8x, measured. Both are one-line config changes, and both are usually the right answer during development — run the full model set at 1,000 series, then the shortlist at 100,000.

Let BigQuery-native models carry their share. arima_plus and timesfm run inside BigQuery, on BigQuery's own slots, and consume none of your Compute Engine quota. A run that mixes them with Python models gets that family for free in vCPU terms, and it runs concurrently with the rest of the DAG rather than after it.

Use a second region — but deploy into it first. Quota is regional, so if us-central1 is capped and the work splits naturally (different business units, say), two runs in two regions have two independent allowances. This does not speed up a single run — ray_regions is failover, not simultaneous execution — but it does raise total throughput across a workload.

The catch, and it is a real one: a Ray cluster on the private path can only be created in the region its network attachment lives in, and Terraform builds one attachment. Listing three ray_regions against a single-region deployment does not give you three candidates; it gives you one, and two regions that 404 on the attachment before quota is ever consulted. --quota reports those as UNREACHABLE and the launch path skips them. To make a second region real, deploy a network attachment there too.