BaseMetric

base_metric

The metric interface every metric file implements.

This is the linchpin of the one-metric-one-file rule and the metric factory, and it is a deliberate mirror of models/base_model.py: every metric is a subclass of BaseMetric living in its own file that ends with register(...), and the factory (metrics/__init__.py) builds {name: class} at import. Adding a metric is a new file plus one import line — plus, unlike a model, one ADD COLUMN IF NOT EXISTS migration, because a metric's value lands in a typed column of forecast_metadata and a model's predictions do not.

A metric is computed by the framework, never supplied by a model. That is the rule the whole leaderboard rests on. engines/bigquery_engine._score_fold goes to the trouble of pulling each BigQuery-native fold back to the driver and scoring it here, with this fold's training window as the scale denominator, precisely so a native model's wape and a Ray model's wape are the same quantity. A model handing back its own number would break that silently, in a column that still looks uniform. Numbers only a fitting library can produce — AIC, a training loss, an early -stopping iteration — are fit diagnostics, not metrics: they exist for some models and not others and are not comparable across them, so they belong in their own JSON channel beside best_params, never in the panel.

Metrics never read global config: everything a metric may see arrives through MetricContext, and a metric that cannot be computed returns NaN rather than raising, so one undefined cell never sinks a batch.

Public surface: BaseMetric, MetricContext, MetricDirection, register (plus the _REGISTRY the factory reads).

BaseMetric

Bases: ABC

One metric: a name, a direction, what it needs, and how to compute it.

Subclasses set the class-level attributes and implement compute. The needs_* flags are declarative, not enforcement — every metric is already NaN-safe when its inputs are absent and that behaviour must not change. Their job is to let config validation say at plan time that a chosen decision_metric will be NaN for every cell of this run, instead of leaving an operator to work it out from an empty leaderboard.

compute abstractmethod

compute(ctx)

The metric's value for ctx, or NaN where it is undefined. Must never raise.

Source code in src/scale_forecasting/metrics/base_metric.py

@abstractmethod
def compute(self, ctx: MetricContext) -> float:
    """The metric's value for ``ctx``, or NaN where it is undefined. Must never raise."""

MetricContext dataclass

MetricContext(
    y_true,
    yhat,
    err,
    abs_err,
    y_train=None,
    lower=None,
    upper=None,
    seasonal_period=None,
    _memo=dict(),
)

One scored window, and everything any metric is allowed to see.

Built once per call to compute_metrics and shared by the whole panel, so the arrays every metric needs — the residual and its absolute value — are computed once rather than fifteen times. Construct it with from_arrays, which is where the shape validation lives.

value(name) is the other half of the sharing, and the reason a per-metric contract does not cost anything in arithmetic. The panel is not fifteen independent numbers: mase divides mae, rmsse divides rmse, and mase_seasonal divides mae again. Rather than have those three recompute their numerator, each asks the context for it by name and the context memoises the answer. The dependency stays visible in the file that has it, the arithmetic still happens once, and the math for mae stays in mae.py where a reader looking for it will go.

has_intervals property

has_intervals

True when both bounds are present — what the interval metrics check before scoring.

bounds

bounds()

(lower, upper) when both bounds are present, else None.

The same check as has_intervals, shaped so that the one guard an interval metric makes also hands it the arrays — a boolean property cannot tell a type checker the bounds exist.

Source code in src/scale_forecasting/metrics/base_metric.py

def bounds(self) -> tuple[np.ndarray, np.ndarray] | None:
    """``(lower, upper)`` when both bounds are present, else ``None``.

    The same check as `has_intervals`, shaped so that the one guard an interval metric makes
    also hands it the arrays — a boolean property cannot tell a type checker the bounds exist.
    """
    if self.lower is None or self.upper is None:
        return None
    return self.lower, self.upper

from_arrays classmethod

from_arrays(
    y_true,
    yhat,
    y_train=None,
    lower=None,
    upper=None,
    seasonal_period=None,
)

Normalise a caller's arrays into a context, or raise on a window that cannot be scored.

The two raises are the only ones in the metric layer, and they are here rather than in any metric because they are not about any one metric: a window whose forecast and actuals have different lengths, or no actuals at all, is a caller bug that should stop, not fifteen NaNs that read like fifteen undefined metrics.

Source code in src/scale_forecasting/metrics/base_metric.py

@classmethod
def from_arrays(
    cls,
    y_true: Sequence[float] | np.ndarray,
    yhat: Sequence[float] | np.ndarray,
    y_train: Sequence[float] | np.ndarray | None = None,
    lower: Sequence[float] | np.ndarray | None = None,
    upper: Sequence[float] | np.ndarray | None = None,
    seasonal_period: int | None = None,
) -> MetricContext:
    """Normalise a caller's arrays into a context, or raise on a window that cannot be scored.

    The two raises are the only ones in the metric layer, and they are here rather than in any
    metric because they are not about any one metric: a window whose forecast and actuals have
    different lengths, or no actuals at all, is a caller bug that should stop, not fifteen
    NaNs that read like fifteen undefined metrics.
    """
    yt = np.asarray(y_true, dtype=float)
    yh = np.asarray(yhat, dtype=float)
    if yt.shape != yh.shape:
        raise ValueError(f"y_true and yhat shape mismatch: {yt.shape} vs {yh.shape}")
    if yt.size == 0:
        raise ValueError("y_true is empty")
    err = yh - yt
    return cls(
        y_true=yt,
        yhat=yh,
        err=err,
        abs_err=np.abs(err),
        y_train=None if y_train is None else np.asarray(y_train, dtype=float),
        lower=None if lower is None else np.asarray(lower, dtype=float),
        upper=None if upper is None else np.asarray(upper, dtype=float),
        seasonal_period=seasonal_period,
    )

value

value(name)

This window's value for the registered metric name, computed at most once.

Used by a metric that builds on another (mase on mae). A metric that names one that is not registered gets NaN rather than an exception, on the same principle as everything else here: an undefined input makes an undefined metric, not a failed run.

Source code in src/scale_forecasting/metrics/base_metric.py

def value(self, name: str) -> float:
    """This window's value for the registered metric ``name``, computed at most once.

    Used by a metric that builds on another (``mase`` on ``mae``). A metric that names one
    that is not registered gets NaN rather than an exception, on the same principle as
    everything else here: an undefined input makes an undefined metric, not a failed run.
    """
    if name in self._memo:
        return self._memo[name]
    metric_cls = _REGISTRY.get(name)
    if metric_cls is None:
        return float("nan")
    # Seeded before computing so a metric that (wrongly) depends on itself gets NaN and
    # returns, instead of recursing until the interpreter gives up.
    self._memo[name] = float("nan")
    result = float(metric_cls().compute(self))
    self._memo[name] = result
    return result

register

register(metric_cls)

Register a metric class under its name. Returns the class so it doubles as a decorator. Raises on a missing, duplicate or unusable name.

Source code in src/scale_forecasting/metrics/base_metric.py

def register(metric_cls: type[BaseMetric]) -> type[BaseMetric]:
    """Register a metric class under its ``name``. Returns the class so it
    doubles as a decorator. Raises on a missing, duplicate or unusable name.
    """
    name = getattr(metric_cls, "name", None)
    if not name:
        raise ConfigError(f"{metric_cls.__name__} must set a class-level 'name' before register()")
    # The name becomes a FLOAT64 column on `forecast_metadata` and a generated SQL identifier in
    # the leaderboard projection, so a name that is not a bare identifier would produce SQL that
    # fails at query time, far from the file that caused it.
    if not name.isidentifier():
        raise ConfigError(f"metric name '{name}' is not a valid column identifier")
    existing = _REGISTRY.get(name)
    if existing is not None and existing is not metric_cls:
        raise ConfigError(
            f"duplicate metric name '{name}': {existing.__name__} vs {metric_cls.__name__}"
        )
    _REGISTRY[name] = metric_cls
    return metric_cls