Compute Profiling¶
The driver-side pre-pass — decide whether to measure, then where the numbers come from.
The consumer of everything else in this package, and the two questions a run actually asks.
Is measuring worth it? — should_profile, a pure function of compute.profile.mode and
the fan-out size.
Where do the numbers come from? — the two answers are deliberately separate functions.
resolve_profile_source reuses prior evidence: a named run's harvest, the newest
auto-discovered harvest whose signature matches, or the shipped baseline, in that order,
each stamped with ProfileProvenance so a sizing decision can always say which of the three
it came from. resolve_profile produces fresh evidence by sampling the panel and
running real fits. Both return None for "no evidence", which is a decision rather than a
failure: sizing from declared config is the floor under every path here.
One case does not get the graceful floor: an operator who pinned profile.source to a
specific run id asserted that that evidence applies here, and check_pinned_source fails the
run at the entry point when the data has since moved out from under that assertion.
Every loader — and the measurement function itself — is injected, so the whole precedence chain and the whole pre-pass are exercised offline against deterministic stand-ins.
Raise errors.ConfigError when a pinned profile.source has silently rotted (I/O).
The asymmetry with auto is deliberate and is about who made the claim. auto warns and
degrades — it is a hint, it has a fallback, and hard-failing would make an unattended run
brittle for no safety gain. An explicit run id is a human assertion; sizing off evidence that
no longer describes this data is exactly the failure the provenance machinery exists to
prevent, so it fails loudly and names both sides. force=True overrides — the same verb that
already overrides the idempotency guard, so there is no second escape hatch to learn.
Called once at the entry points (main.run, launch_plan.plan_run / stage_run) rather than
at each of the sizing sites: the operator should hear about this before anything is submitted,
once, not six times from inside a launch.
Source code in src/scale_forecasting/profiling/source.py
The shipped, versioned reference profile — the cold-start floor under the chain above.
Kept as a seam in this module rather than resolving baseline.baseline_profile at the call
site, because "is there a baseline, and can it be loaded" is a precedence question and this
is where the precedence lives. It stays | None in the signature for the same reason: the
BaselineLoader protocol lets a caller inject a deployment that ships none, and
resolve_profile_source already treats that as "fall through to static config".
See profiling.baseline for where the numbers came from and what they deliberately omit.
Source code in src/scale_forecasting/profiling/source.py
Signature drift on an explicitly pinned profile.source, or () (pure + injected).
Only an explicit source: "<run_id>" is checked. auto, baseline and none are the
system choosing, and the system has a fallback; a pinned run id is a person asserting that
this specific evidence applies to this run. When the signature says the data has moved out from
under it, that assertion is false, and this is what check_pinned_source turns into an error.
Returns the same warning strings resolve_profile_source would have stamped on the provenance
— computed by calling it, so the two can never disagree about what "drifted" means. A pin that
cannot be loaded at all returns (): an unreachable registry is not evidence of drift, and
the sizing path already degrades to config for it.
Source code in src/scale_forecasting/profiling/source.py
The profile this run sizes from, with the real registry loaders bound (I/O; memoized).
The impure half of resolve_profile_source: it supplies the BigQuery readers and nothing else.
Split that way so the whole precedence chain stays testable with no cloud, and so the four
sizing call sites (submit.sizing_properties, dataproc_cluster.cluster_sizing) each stay a
pure function that is handed a profile rather than one that goes and finds one.
Returns None for "size from declared config" — including whenever the registry is
unreachable, which resolve_profile_source degrades to rather than raising.
Source code in src/scale_forecasting/profiling/source.py
Driver-side measurement pre-pass: sample the panel, fit, aggregate → ComputeProfile.
The structural twin of the fleetwide-HPO pre-pass (resolve_fleetwide_hpo) — same seam,
same place, same "resolve once on the driver before fanning out" shape — and it runs after
that one, so tuned hyperparameters can be measured rather than defaults.
Returns None when no measurement was taken: profiling is off, the fan-out is below
min_cells, nothing profilable is in the model list, or the panel yields no usable
statistics. None is the signal to size from static config, and every consumer already
treats it that way (resources.slot.resource_slot takes profile=None as its
identity case). A profile that was taken but measured nothing usable comes back as an empty
ComputeProfile rather than None — the distinction is "we did not look" versus "we
looked and found nothing", and only the second is worth an audit record.
The memory axis is dropped before aggregation — see _without_driver_rss. Everything the
pre-pass keeps is a cost of the fit; absolute RSS is a cost of the process, and the driver
is not the process a slot describes.
The sample loop is the outer one, deliberately. Cycling every model across every sampled
series is what gives each model the same spread of series lengths and shapes, so one model's
reading is never taken from the one short series while another's came from the long one; it
also charges the first fit's import cost to a single reading that the median aggregation
then steps over. (At samples=1 there is no spread at all — one sample is a smoke-test
setting, not a sizing one.)
Hyperparameters. params_by_model (the fleetwide pre-pass's output) is passed straight
through, so the measured fit is the fit that will run. Under per-series HPO the pre-pass
instead passes {}: leaving it None would make each probe call worker._resolve_params
and run a full Optuna study, turning an 8-sample pre-pass into hundreds of driver-side fits
with nothing to bound it (see measure_fit). Profiling an untuned fit under per-series
tuning is a known under-statement; running the tuner samples x models times to avoid it is
worse.
measure injects the measurement function for the offline gate — the default is
measure_fit, and the tests pass a deterministic stand-in so the whole pre-pass, gate
included, is testable with no fit, no accelerator, and no cloud.
Source code in src/scale_forecasting/profiling/source.py
375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 | |
The profile this run should size from, per compute.profile.source (pure + injected I/O).
Returns None for "no evidence" — the static-config case every consumer already handles
(resources.slot.resource_slot takes profile=None as its identity case). None is a
decision, not a failure: sizing from declared config is the behaviour this product shipped
with, and it stays the floor under every path here.
The precedence, resolved outside-in:
mode == "off"orsource == "none"→None. Nothing consulted, nothing loaded.source == "<run_id>"→ that run's harvest. An operator naming a run has made a decision; it is honoured even if the signature has drifted, and the drift comes back as warnings on the provenance rather than as a substitution they did not ask for.source == "auto"→ the newest run whose harvest matches this run's signature, ifdiscoverfinds one.- the shipped baseline, if one is loadable.
None.
Every loader is injected. That keeps this function pure enough to test the whole precedence
chain offline with no BigQuery and no baseline file — the same seam resolve_profile uses
for measure. A loader that raises is treated as a loader that found nothing: sizing
evidence is an optimisation, and a registry hiccup must not sink a run that would otherwise
size itself from config.
Source code in src/scale_forecasting/profiling/source.py
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 | |
Is a measurement pre-pass worth running for a fan-out of n_cells? (pure)
The three compute.profile.mode values, and the reasoning behind the middle one:
off— never. The escape hatch: static config, exactly as before the profiler existed.always— unconditionally. What the smokes use, so the path is exercised on runs small enough that exercising it is cheap.auto(default) — only when the fan-out is big enough to repay the pre-pass. The pre-pass costssamples x modelsreal fits on the driver; belowmin_cellsthat is a measurable fraction of the whole run, spent to size a fleet that barely needs sizing. Above it, the same fixed cost is rounding error against the work it is optimising.
The comparison is on cells, not series, because a 100-series run over 6 models is the same amount of work as a 600-series run over one, and it is the work — not the panel — that the fleet is sized for.
Source code in src/scale_forecasting/profiling/source.py
Measurements -> a per-family cost model. Pure arithmetic, no accelerator required.
build_profile folds a list of MeasuredFit records into a ComputeProfile: per-model
costs, rolled up per family, with safety margins applied. This is the pure half of the
package's pure/I-O seam — the arithmetic that decides how much hardware a run gets must be
testable with no accelerator, no cluster, and no cloud, exactly as calibrate_gpu_fraction
is.
Why both tails. build_profile keeps the max of the peaks and the median of the
times. Max governs safety — how many tasks may share a device or an executor without an
OOM. Median governs throughput — how much work the fleet has, i.e. how wide it needs to
be. Sizing a fleet off the worst case over-provisions every run; sizing memory off the
median OOM-kills it. Using one tail for both is the mistake this split exists to prevent.
Absence is a value. Every aggregated axis is | None, and None means "we have no
basis for this number" — never "zero"; see numeric for the helpers that enforce it.
This module reports bytes, seconds and cores. It never emits a runtime knob — no GPU
fraction, no executor cores, no node count, no autoscaling bound. Turning a ComputeProfile
into those is three different translations of the same numbers and lands later as
plan_resources, which consumes a ComputeProfile plus a cell count and nothing else.
The GPU-memory denominator stays in ray_io.device_memory_bytes; this module never
re-declares a device table.
dataclass
¶
ComputeProfile(
families,
models,
memory_margin,
time_margin,
n_measurements,
n_ok,
sample_ts_ids,
dropped_models=(),
first_error_by_model=None,
sample=(),
provenance=None,
)
The measured cost model for one run: per-model and per-family, both tails kept (pure).
The whole object is stamped into run telemetry, so it stays JSON-representable and carries the margins it was built with — a sizing decision that cannot be re-derived from its own record is not auditable. Both mappings are built from sorted keys so the serialized form is byte-stable and an audit diff is meaningful.
A family or model is absent when nothing usable was measured for it. Absence is the
signal to fall back to static config; a zero-valued entry would be consumed as a real
size. Absence is deliberately not silent: dropped_models and first_error_by_model
name what fell out and why, so "this family was sized off one of its two models, because
the other one OOM'd" is readable straight off the record instead of being inferred from a
gap in it.
property
¶
True when nothing usable was measured — the caller must fall back to static config.
This family's cost, or None when it was never measured (fall back, don't guess).
JSON-safe dict of the whole profile — raw aggregates and the sized values.
The sized values are serialized alongside the raw ones so telemetry answers both
"what did we measure" and "what did we ask the platform for" without the reader
re-deriving a margin. Plain ints/floats/strings/None only: json.dumps must
succeed with no custom encoder.
Bounded by construction, because this is what lands in a registry JSON column. Every
field above is sized by the model list except sample_ts_ids, which is sized by the
panel: a profile resolved from a harvest (resolve_profile_source) can carry tens of
thousands of ids, and a telemetry blob that grows with the data is one nobody can read and
one that eventually will not write. The count is kept whole (n_sample_series) and the
ids are truncated to _TELEMETRY_SAMPLE_IDS; a reader sees the truncation as
len(sample_ts_ids) < n_sample_series without needing a flag for it.
Source code in src/scale_forecasting/profiling/cost.py
dataclass
¶
FamilyCost(
family,
models,
n_fits,
n_ok,
max_peak_rss_bytes,
max_peak_gpu_bytes,
max_effective_cores,
median_wall_s,
total_wall_s_per_series,
memory_margin,
time_margin,
max_process_rss_bytes=None,
)
One family's slot shape and workload, rolled up from its models (pure).
Roll-up rule, and the reason it differs per axis: peaks take the max across the family's models, because one slot must hold whichever model lands in it; times take the median for the per-cell question and the sum for the per-series question, because a family's models all run — they do not compete for one cell.
The margins are carried on the record rather than applied by the caller so the sized numbers and the raw measurements travel together: an audit reads both "measured 3.1 GiB" and "sized 4.1 GiB" off one object, and a consumer cannot accidentally apply the margin twice.
Invariant: every derived property is None if and only if its raw basis is
None. There is no fabricated fallback here — a consumer that gets None has
been told, unambiguously, that it must fall back to its own static default (for
slot_cores that default is 1, today's hardcoded value).
property
¶
Expected seconds to run every model of this family for one series, margin applied.
property
¶
Expected seconds for the median cell of this family, time margin applied.
property
¶
Whole cores one cell needs: ceil(max_effective_cores), at least 1.
No margin — a core count is already discrete and already the max over the family — but
a _CORE_SNAP_TOLERANCE snap before rounding up, because the input is a ratio of two
clocks. A single-threaded fit lands on either side of 1.0 at random, and since this is
the max over the family, the chance that at least one member lands a hair above
grows with family size: five single-threaded models trip it ~97% of the time. Without
the snap that reads as slot_cores=2 and halves fleet density, with an audit record
showing 1.0000005 that looks correct to two decimal places.
property
¶
Device memory one cell needs, margin applied; None when the axis is unmeasured.
property
¶
Host memory one cell of this family needs, margin applied (bytes, rounded up).
Built on max_process_rss_bytes — the absolute footprint — not on the marginal
max_peak_rss_bytes. A slot holds an interpreter with the model stack imported, not
just the incremental allocation of one fit, and the marginal number is in any case
unstable: it swings 17x on the order the sample was measured in and reads 0.00 MB for a
model whose fit is served entirely from already-resident pages. See MeasuredFit.
None when nothing measured the absolute footprint (no resource module, or every
fit failed), which tells the consumer to fall back to its own static default rather
than to size against a number that was never taken.
dataclass
¶
ModelCost(
model_type,
family,
n_fits,
n_ok,
max_n_obs,
max_peak_rss_bytes,
max_peak_gpu_bytes,
median_wall_s,
median_cpu_s,
max_effective_cores,
max_process_rss_bytes=None,
)
Aggregated cost of one model across the profiled sample — RAW, no margin (pure).
Every axis is | None and independently derived: a measurement can be good evidence
for wall time and no evidence at all for GPU memory, so usability is decided per axis,
not per record. None means no usable measurement contributed to that axis.
No margins are applied here. They ride on FamilyCost and ComputeProfile, which is
what a consumer reads, so a margin can never be applied twice on the way through.
n_fits here counts fits that already happened and were measured — it is a property
of the profile sample, not of a run. config.Workload.n_fits is the forward-looking
quantity (how many fits a planned run will perform). Same word, opposite direction; the
name stays because it is a key in the persisted sizing telemetry JSON.
dataclass
¶
ProfileProvenance(
basis,
source,
run_id=None,
baseline_version=None,
measured_at=None,
signature=None,
warnings=(),
)
Where a profile's numbers came from, carried with the numbers (pure).
A sizing decision an operator cannot attribute is one they cannot argue with. The load-bearing
field is basis:
measured— taken on this run's own data. The strongest claim.reference— measured, genuinely, but not on your data: another run's harvest, or the baseline shipped with the product. Real evidence with a caveat, and without a name for that state an operator reading a resolved fleet shape cannot tell whose evidence produced it.assumed— no measurement; the static arithmetic. Recorded so "we had nothing" is a stated outcome rather than an absent field.
JSON-safe dict, for the telemetry stamp.
Source code in src/scale_forecasting/profiling/cost.py
build_profile(
measurements,
*,
sample=(),
memory_margin=_DEFAULT_MEMORY_MARGIN,
time_margin=_DEFAULT_TIME_MARGIN,
)
Aggregate measurements into a per-family cost model: max for peaks, median for times (pure).
The whole point of the split: max(peak) x memory_margin governs safety — how many
tasks may share a device or an executor without an OOM — while median(time) x
time_margin governs throughput — how many slots the load needs. Using the max for
both systematically over-provisions every run; using the median for both OOM-kills it. Both
tails are therefore kept and both are exposed.
Usability is decided per axis, not per record: a measurement contributes to an axis iff
it succeeded and that axis's value is finite and positive. One fit can be perfectly good
evidence for wall time and no evidence at all for GPU memory, so filtering whole records
would throw the first away and trusting whole records would fabricate the second. An axis
with no usable value is None, and a model or family with no usable axis is absent
from the profile — absence is what tells a consumer to fall back to its static default.
Margins are validated (below 1.0 asks for less headroom than the measurement, which is
never valid), recorded on every record, and applied only in the slot_* /
planning_* properties, so they can never be applied twice. Input is never mutated,
grouping keys are iterated sorted, and the result is a pure function of the measurement
set — a shuffled input yields an equal profile.
sample is optional and carried through verbatim for the audit record: the profile then
answers "what did this cost" and "which series was that measured on, and why those" from
one object. It is not read by any arithmetic here.
Source code in src/scale_forecasting/profiling/cost.py
515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 | |
Aggregate persisted forecast_metadata rows into a ComputeProfile (pure).
The second producer, and the one that scales. build_profile aggregates a pre-pass that
deliberately fits a small sample; this aggregates the fits a completed run already performed.
Both feed the identical aggregation, so a harvested profile and a measured one are the same
object and every translator consumes them the same way. The difference is only in how the
evidence was obtained — and this way it is obtained for free, from every cell rather than
from eight, on the real hardware rather than on a driver.
That is what makes "size this run like run X" a query instead of an artifact store: a
completed run_id is a profile, with its config, data signature and lineage already
recorded next to it in the registry. Nothing new to version, expire, or garbage-collect.
rows are mappings with forecast_metadata column names — from a BigQuery read, a test
fixture, or a committed baseline file; the function never touches BigQuery itself. Missing
keys read as NULL, so rows written before the measurement columns existed degrade to
wall-time-only evidence rather than raising.
Two filters, both structural. Backtest fold rows (fold_id set) are skipped because
fit_seconds on the full-fit row already brackets the whole cell, folds included — counting
folds too would double-count the same work. Ensemble rows (ensemble_id set) are skipped
because an ensemble is arithmetic over predictions, not a fit whose cost sizes a slot.
How success is inferred, since forecast_metadata carries no status column. A cell that
errored returns before its wall clock is recorded and lands with fit_seconds of zero, so a
usable wall time is exactly the signal that a fit happened. Those rows still count toward
n_measurements and n_failed, so "we sized off 940 of 1000 cells" stays visible — the
same honesty build_profile gives a pre-pass, at run scale.
Source code in src/scale_forecasting/profiling/cost.py
Rebuild a ComputeProfile from a ComputeProfile.to_dict payload (pure).
The reader half of the shipped baseline (profiling.baseline), and the reason a baseline can
be a committed artifact at all: without it, shipping numbers would mean shipping a second,
hand-written cost model beside this one, free to drift from it.
Only the raw fields are read. slot_*, planning_*, n_failed and
n_sample_series are properties derived from those raw fields, so they are recomputed rather
than trusted — a payload whose derived values were hand-edited out of step cannot mislead the
arithmetic, only its human reader, and test_profiling closes that second half by asserting the
shipped payload's derived values still equal what the properties compute.
Three fields do not survive a round trip, all documented losses rather than oversights.
sample_ts_ids is truncated by to_dict to _TELEMETRY_SAMPLE_IDS, so what comes back is
the truncation — the whole count survives on n_measurements. sample is not reconstructed
at all: sampling.SampleSpec.to_dict flattens its SeriesStats, and a harvested profile (the
only kind ever serialized for reuse) carries no sample anyway. And first_error_by_model
comes back as {} where it went in as None, because to_dict does not distinguish them;
build_profile always sets a dict, so the distinction never arises on a real profile.
Source code in src/scale_forecasting/profiling/cost.py
340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 | |
One instrumented fit — what a single cell of a model family actually consumed.
Static per-model resource guesses fail in both directions: too generous and every run
over-provisions, too tight and the task OOMs. measure_fit replaces the guess by
bracketing exactly one run_cell and reporting five axes — wall time, CPU time, absolute
process footprint, marginal host RSS, peak device bytes — from which
effective_cores = cpu_s / wall_s answers "does this model use more than one core" by
observation instead of by assertion.
Two of those axes are traps, and both traps were found by measuring rather than by reasoning. They are stated up front because in each case the obvious implementation produces a plausible number that is wrong by more than an order of magnitude — the failure mode a sizing pre-pass can least afford, since nothing downstream can tell a bad measurement from a good one.
The core count measures the machine unless the fit is pinned twice. An unpinned probe on
an idle driver measures the driver: OpenBLAS/OpenMP inside statsmodels take every free
core, so on a 32-core box theta reported 19.7 effective cores and sarimax 11.5 — the
same models report ~3 on a 4-core box, and a naive mean can score highest of all. That CPU
bought no wall-clock win. Pinning takes both threadpool_limits and the
_INTRAOP_ENV_VARS, because the two reach pools with different lifetimes and each alone
leaves half the threads running (threadpoolctl alone still gave theta 4.8 and holtwinters
7.3). Under both, every model measures 1.00. See _pinned_intraop_threads for why.
The memory number must be the absolute footprint, not the per-fit delta. The intuitive
"RSS after minus RSS before" is unusable: it swings 17x on the order the sample ran in,
because the first fit is charged for lazily importing the shared model stack while later
fits are served from an already-warm heap and report 0.00 MB. The absolute high-water lands
within 0.6% regardless of order, and it is also the number a slot actually needs — a slot
holds the interpreter and the libraries too. See MeasuredFit for the measurements.
A real fit runs here, so this module is the I/O side of the package's pure/I-O seam: it is
exercised live, while cost — the arithmetic that turns these records into a sizing
decision — stays testable with no accelerator, no cluster and no cloud.
dataclass
¶
MeasuredFit(
ts_id,
model_type,
family,
n_obs,
wall_s,
cpu_s,
peak_rss_bytes,
peak_gpu_bytes,
ok,
error,
intraop_threads=None,
host_cpu_count=None,
rss_peak_reset=False,
process_rss_bytes=None,
)
Four measured axes from exactly one run_cell, plus what it was measuring.
family is carried on the measurement rather than re-derived in build_profile, so
the aggregation is pure arithmetic over injected records: an offline test constructs
MeasuredFit values directly and never touches the model registry, a GPU, or a
cluster. Same injection seam as calibrate_gpu_fraction(measured_peaks_bytes=...).
Two memory numbers, because the obvious one does not survive contact with an allocator.
process_rss_bytes is what sizes a slot; peak_rss_bytes is a diagnostic. The reason
is measured, not assumed — the same four models, fitted in three different orders:
========== ===================================== =========================
model peak_rss_bytes (delta), by order process_rss_bytes
========== ===================================== =========================
theta 79.0 MB / 4.6 MB / 13.1 MB 646 / 672 / 649 MB
sarimax 27.8 MB / 97.7 MB / 27.6 MB 676 / 666 / 676 MB
========== ===================================== =========================
The delta swings 17x on the order the models ran in; the absolute high-water lands
within 0.6% of 676 MB every time. Two effects cause the swing, and they push opposite ways:
whichever model fits first is charged for lazily importing the shared model stack, while
every model after it allocates inside a heap that is already warm, so the allocator serves
the fit from resident pages and ru_maxrss never rises — a warmed-up run reports 0.00 MB
for theta and holtwinters, which is not a claim that they are free.
So attribution is the wrong question. A worker slot must hold the interpreter, the
libraries, and the fit; the shared residency is not overhead to be factored out, it is
part of what the slot must fit. process_rss_bytes measures exactly that and is stable,
which is why FamilyCost.slot_rss_bytes is built from it. It over-states for a light family
profiled in the same process as a heavy one — the deliberate direction, per the asymmetry
_DEFAULT_MEMORY_MARGIN is chosen on: over-estimating memory costs money, under-estimating
it kills the task. None means NOT MEASURED (no resource module), never zero.
peak_rss_bytes is kept because the marginal cost is worth seeing even when it is not
worth sizing on. Read it as a lower bound where 0 means "no evidence", never as proof a
fit was free; build_profile discards non-positive values rather than folding them into a
max. It assumes fits are sequential in this process — concurrent measure_fit calls
make each other's delta meaningless.
rss_peak_reset records whether the high-water mark could be zeroed before the fit.
Without the reset the mark is monotonic for the life of the interpreter, so the delta
degrades from "order-dependent" to "zero for everything after the first heavy model".
measure_fit resets it on Linux; False means the delta is worth even less than usual.
The reset also rebases process_rss_bytes, and helpfully so: the absolute reading becomes
"this process's live footprint plus what this fit added" rather than "the largest transient
any earlier fit ever reached and has since freed". The former is what a slot running this
model needs; the latter is another model's spike wearing this model's name.
peak_gpu_bytes is None means NOT MEASURED — never "measured zero". The profile
runs on the driver at submit time, where there is usually no accelerator. If a missing
GPU reading arrived as 0, a consumer would compute a minimum GPU fraction and pack
ten tasks onto a device that fits two. None forces the fall-back to a nominal
fraction and a refinement on-cluster, which is the existing two-phase behaviour.
ok=False carries a failure the way CellResult does, rather than raising: a flaky
probe widens sizing to nominal, it does not sink the run. Failed records are excluded
from every aggregate but counted, so "we sized off 6 of 8 fits" stays visible.
property
¶
cpu_s / wall_s, floored at 1.0 — measured thread-parallelism of this fit.
The direct answer to "can this model use more than one CPU", with no per-model
declaration to maintain and no per-runtime variation. A wall_s below
_MIN_WALL_S yields _MIN_EFFECTIVE_CORES rather than a division blow-up.
Only meaningful relative to intraop_threads. The ratio counts threads that
actually ran, so an unpinned fit on an idle many-core driver reports that driver's core
count for almost any model. Read as "parallelism this fit found under a cap of
intraop_threads"; intraop_threads=None means the cap could not be applied and
the ratio is an upper bound contaminated by host_cpu_count.
Run exactly one run_cell and report what it consumed (I/O; never raises).
Mirrors run_cell(series, model_name, cfg, params) argument-for-argument, so the thing
being measured is unmistakably the thing that runs in production. params is carried
because HPO-tuned hyperparameters demonstrably change fit cost (n_estimators, epochs),
and measuring an untuned fit for a tuned run would size the fleet for different work.
Fixed measurement order, and it is part of the contract: reset the RSS high-water mark,
reset the CUDA high-water mark, read ru_maxrss, cap the native thread pools, start both
clocks, run the cell, stop both clocks, release the cap, re-read ru_maxrss, read the
CUDA peak. There is no measure_gpu flag — the read is attempted and the absence of a
device is expressed as peak_gpu_bytes=None, so there is one path to that outcome.
The thread cap is part of the measurement, not an optimization. Both resets and the cap
exist because the two host axes are otherwise measurements of the machine rather than of
the model: ru_maxrss is monotonic, so without a reset a fit's memory reads as whatever
the fits before it left behind, and an uncapped fit takes every idle core, so its
cpu_s / wall_s reads as the driver's nproc. What each attempt achieved is recorded
on the record (rss_peak_reset, intraop_threads, host_cpu_count) rather than
assumed, because both are platform-dependent.
Writes nothing — no registry, no GCS, no log line. The CellResult is deliberately
discarded: this is a probe, not a run, and its forecasts must never reach the registry under
the run's run_id.
No timeout, deliberately. signal.alarm is unsafe off the main thread (a Spark driver,
an Airflow worker) and a thread-based kill cannot interrupt a C-extension fit, so any guard
written here would be theatre. The real bound is the budget: samples x models sequential
fits on the driver. A hung fit stalls submit visibly; it cannot corrupt sizing.
That budget is not the whole story under per-series HPO. params=None defers to
worker._resolve_params, which runs a full Optuna study when hpo.enabled and
hpo.granularity == "per_series" — so one call becomes n_trials fits (20 by default)
times the backtest folds, and an 8-sample x 5-model pre-pass becomes hundreds of driver-side
fits with, by the paragraph above, nothing to stop it. A caller profiling such a run should
pass already-resolved params or shrink the sample.
Sequential use only. peak_rss_bytes is a delta on a process-wide high-water mark, so
two concurrent measure_fit calls make each other's RSS number meaningless.
Best-effort throughout, exactly like the existing GPU calibration probe: a status="error"
cell keeps its measured numbers and records ok=False (never discard a fact at the
measurement layer — build_profile decides usability), and anything that raises returns a
zeroed ok=False record. There is no raising path.
Source code in src/scale_forecasting/profiling/measure.py
316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 | |