Metrics

metrics

Forecast metric panel — pure, and the factory that assembles it.

One entry point, compute_metrics, returns the full panel every run so users never re-run to get a different metric; the decision metric is then a pure config choice. All values are floats, with NaN where a metric is undefined (MAPE with zeros, MASE without training history, coverage without intervals) rather than raising — a metric that can't be computed for one cell must not sink the batch.

Importing this package registers every metric file by name (each metric module ends with register(...)); get_metric(name) returns the class and list_metrics() lists the registered names. Adding a metric is one new file, one line in the import block, one line in METRIC_NAMES, and the ADD COLUMN IF NOT EXISTS migration registry.ddl.render_migrations already emits — see docs/metric_template.py.

Public surface: compute_metrics, METRIC_NAMES, METRIC_DIRECTION, loss_of, get_metric, list_metrics, BaseMetric, MetricContext, MetricDirection, register.

Definitions (n = horizon, e = yhat - y_true):

• mae = mean(|e|)
• rmse = sqrt(mean(e²)); mse = mean(e²)
• mape = mean(|e| / |y_true|) (NaN if any y_true == 0)
• smape = mean(2|e| / (|y_true| + |yhat|))
• wape = sum(|e|) / sum(|y_true|) (NaN if sum(|y_true|) == 0)
• mase = mae / mae_naive, naive = one-step (m=1) on y_train
• rmsse = rmse / rmse_naive, naive = one-step (m=1) on y_train
• bias = mean(e) (mean error / ME)
• coverage = fraction of y_true within [lower, upper] (needs intervals)
• pinball = mean quantile loss across the interval bounds (needs intervals)
• mase_seasonal = mae / mae_naive, naive = seasonal (m=seasonal_period) on y_train
• maape = mean(arctan(|e| / |y_true|)) — defined at y_true == 0, unlike MAPE
• interval_score = mean Winkler score of [lower, upper] at α = 0.2 (needs intervals)
• interval_width = mean(upper - lower) (needs intervals)
• ope = |sum(yhat) - sum(y_true)| / |sum(y_true)| (NaN if sum(y_true) == 0)
• rmsle = sqrt(mean((log1p(y_true) - log1p(yhat))²)) (NaN if any y_true < 0 or yhat < 0)
• msse = mse / mean(diff(y_train)²) (needs y_train)
• msis = interval_score / mae_naive_seasonal (needs intervals, y_train, seasonal_period)
• r2 = 1 - sum(e²) / sum((y_true - mean(y_true))²) (higher is better; 1.0 is ideal)
• cv = rmse / |mean(y_true)| (NaN if mean(y_true) == 0)

BaseMetric

Bases: ABC

One metric: a name, a direction, what it needs, and how to compute it.

Subclasses set the class-level attributes and implement compute. The needs_* flags are declarative, not enforcement — every metric is already NaN-safe when its inputs are absent and that behaviour must not change. Their job is to let config validation say at plan time that a chosen decision_metric will be NaN for every cell of this run, instead of leaving an operator to work it out from an empty leaderboard.

compute abstractmethod

compute(ctx)

The metric's value for ctx, or NaN where it is undefined. Must never raise.

Source code in src/scale_forecasting/metrics/base_metric.py

@abstractmethod
def compute(self, ctx: MetricContext) -> float:
    """The metric's value for ``ctx``, or NaN where it is undefined. Must never raise."""

MetricContext dataclass

MetricContext(
    y_true,
    yhat,
    err,
    abs_err,
    y_train=None,
    lower=None,
    upper=None,
    seasonal_period=None,
    _memo=dict(),
)

One scored window, and everything any metric is allowed to see.

Built once per call to compute_metrics and shared by the whole panel, so the arrays every metric needs — the residual and its absolute value — are computed once rather than fifteen times. Construct it with from_arrays, which is where the shape validation lives.

value(name) is the other half of the sharing, and the reason a per-metric contract does not cost anything in arithmetic. The panel is not fifteen independent numbers: mase divides mae, rmsse divides rmse, and mase_seasonal divides mae again. Rather than have those three recompute their numerator, each asks the context for it by name and the context memoises the answer. The dependency stays visible in the file that has it, the arithmetic still happens once, and the math for mae stays in mae.py where a reader looking for it will go.

has_intervals property

has_intervals

True when both bounds are present — what the interval metrics check before scoring.

bounds

bounds()

(lower, upper) when both bounds are present, else None.

The same check as has_intervals, shaped so that the one guard an interval metric makes also hands it the arrays — a boolean property cannot tell a type checker the bounds exist.

Source code in src/scale_forecasting/metrics/base_metric.py

def bounds(self) -> tuple[np.ndarray, np.ndarray] | None:
    """``(lower, upper)`` when both bounds are present, else ``None``.

    The same check as `has_intervals`, shaped so that the one guard an interval metric makes
    also hands it the arrays — a boolean property cannot tell a type checker the bounds exist.
    """
    if self.lower is None or self.upper is None:
        return None
    return self.lower, self.upper

from_arrays classmethod

from_arrays(
    y_true,
    yhat,
    y_train=None,
    lower=None,
    upper=None,
    seasonal_period=None,
)

Normalise a caller's arrays into a context, or raise on a window that cannot be scored.

The two raises are the only ones in the metric layer, and they are here rather than in any metric because they are not about any one metric: a window whose forecast and actuals have different lengths, or no actuals at all, is a caller bug that should stop, not fifteen NaNs that read like fifteen undefined metrics.

Source code in src/scale_forecasting/metrics/base_metric.py

@classmethod
def from_arrays(
    cls,
    y_true: Sequence[float] | np.ndarray,
    yhat: Sequence[float] | np.ndarray,
    y_train: Sequence[float] | np.ndarray | None = None,
    lower: Sequence[float] | np.ndarray | None = None,
    upper: Sequence[float] | np.ndarray | None = None,
    seasonal_period: int | None = None,
) -> MetricContext:
    """Normalise a caller's arrays into a context, or raise on a window that cannot be scored.

    The two raises are the only ones in the metric layer, and they are here rather than in any
    metric because they are not about any one metric: a window whose forecast and actuals have
    different lengths, or no actuals at all, is a caller bug that should stop, not fifteen
    NaNs that read like fifteen undefined metrics.
    """
    yt = np.asarray(y_true, dtype=float)
    yh = np.asarray(yhat, dtype=float)
    if yt.shape != yh.shape:
        raise ValueError(f"y_true and yhat shape mismatch: {yt.shape} vs {yh.shape}")
    if yt.size == 0:
        raise ValueError("y_true is empty")
    err = yh - yt
    return cls(
        y_true=yt,
        yhat=yh,
        err=err,
        abs_err=np.abs(err),
        y_train=None if y_train is None else np.asarray(y_train, dtype=float),
        lower=None if lower is None else np.asarray(lower, dtype=float),
        upper=None if upper is None else np.asarray(upper, dtype=float),
        seasonal_period=seasonal_period,
    )

value

value(name)

This window's value for the registered metric name, computed at most once.

Used by a metric that builds on another (mase on mae). A metric that names one that is not registered gets NaN rather than an exception, on the same principle as everything else here: an undefined input makes an undefined metric, not a failed run.

Source code in src/scale_forecasting/metrics/base_metric.py

def value(self, name: str) -> float:
    """This window's value for the registered metric ``name``, computed at most once.

    Used by a metric that builds on another (``mase`` on ``mae``). A metric that names one
    that is not registered gets NaN rather than an exception, on the same principle as
    everything else here: an undefined input makes an undefined metric, not a failed run.
    """
    if name in self._memo:
        return self._memo[name]
    metric_cls = _REGISTRY.get(name)
    if metric_cls is None:
        return float("nan")
    # Seeded before computing so a metric that (wrongly) depends on itself gets NaN and
    # returns, instead of recursing until the interpreter gives up.
    self._memo[name] = float("nan")
    result = float(metric_cls().compute(self))
    self._memo[name] = result
    return result

compute_metrics

compute_metrics(
    y_true,
    yhat,
    y_train=None,
    lower=None,
    upper=None,
    seasonal_period=None,
)

Compute the full metric panel for one forecast window.

Builds one MetricContext and asks it for every name in METRIC_NAMES. Because the context memoises, a metric that another metric already needed — mae under mase — is computed once no matter how many times it is asked for or what order the panel is in.

Parameters:

Name	Type	Description	Default
y_true	Sequence[float] | ndarray	actuals over the evaluation window.	required
yhat	Sequence[float] | ndarray	point forecasts, aligned to y_true.	required
y_train	Sequence[float] | ndarray | None	training-history actuals; required for scale-free MASE/RMSSE (else NaN). This fold's training window, not the whole series. MASE and RMSSE divide by the mean step of whatever is handed in, so passing history that overlaps y_true scales the score by data the model was judged on, and two engines that disagree about it publish two incomparable numbers into the same leaderboard column. Every caller derives it the same way — backtest.training_window, or the fold's own training slice in backtest.backtest_cell.	None
lower	Sequence[float] | ndarray | None	lower prediction bound; with upper enables coverage/pinball (else NaN).	None
upper	Sequence[float] | ndarray | None	upper prediction bound.	None
seasonal_period	int | None	steps in one seasonal cycle, from seasonality.seasonal_period; required for mase_seasonal (else NaN). Callers pass the run frequency's period rather than a default, because guessing it here would silently score an hourly run against a weekly naive.	None

Returns:

Type	Description
dict[str, float]	{name: float} for every name in METRIC_NAMES, in that order. Undefined metrics
dict[str, float]	are NaN.

Raises:

Type	Description
ValueError	if y_true and yhat have different lengths or are empty.

Source code in src/scale_forecasting/metrics/__init__.py

def compute_metrics(
    y_true: Sequence[float] | np.ndarray,
    yhat: Sequence[float] | np.ndarray,
    y_train: Sequence[float] | np.ndarray | None = None,
    lower: Sequence[float] | np.ndarray | None = None,
    upper: Sequence[float] | np.ndarray | None = None,
    seasonal_period: int | None = None,
) -> dict[str, float]:
    """Compute the full metric panel for one forecast window.

    Builds one `MetricContext` and asks it for every name in `METRIC_NAMES`. Because the context
    memoises, a metric that another metric already needed — ``mae`` under ``mase`` — is computed
    once no matter how many times it is asked for or what order the panel is in.

    Args:
        y_true: actuals over the evaluation window.
        yhat: point forecasts, aligned to ``y_true``.
        y_train: training-history actuals; required for scale-free MASE/RMSSE (else NaN).
            **This fold's training window, not the whole series.** MASE and RMSSE divide by the
            mean step of whatever is handed in, so passing history that overlaps ``y_true``
            scales the score by data the model was judged on, and two engines that disagree
            about it publish two incomparable numbers into the same leaderboard column. Every
            caller derives it the same way — `backtest.training_window`, or the fold's own
            training slice in `backtest.backtest_cell`.
        lower: lower prediction bound; with ``upper`` enables coverage/pinball (else NaN).
        upper: upper prediction bound.
        seasonal_period: steps in one seasonal cycle, from `seasonality.seasonal_period`;
            required for ``mase_seasonal`` (else NaN). Callers pass the run frequency's
            period rather than a default, because guessing it here would silently score
            an hourly run against a weekly naive.

    Returns:
        ``{name: float}`` for every name in `METRIC_NAMES`, in that order. Undefined metrics
        are NaN.

    Raises:
        ValueError: if ``y_true`` and ``yhat`` have different lengths or are empty.
    """
    ctx = MetricContext.from_arrays(
        y_true,
        yhat,
        y_train=y_train,
        lower=lower,
        upper=upper,
        seasonal_period=seasonal_period,
    )
    return {name: ctx.value(name) for name in METRIC_NAMES}

get_metric

get_metric(name)

Return the registered metric class for name.

Raises ConfigError with the available names when name is unknown.

Source code in src/scale_forecasting/metrics/__init__.py

def get_metric(name: str) -> type[BaseMetric]:
    """Return the registered metric class for ``name``.

    Raises ``ConfigError`` with the available names when ``name`` is unknown.
    """
    try:
        return _REGISTRY[name]
    except KeyError:
        known = ", ".join(sorted(_REGISTRY)) or "(none registered)"
        raise ConfigError(f"unknown metric '{name}'; registered metrics: {known}") from None

list_metrics

list_metrics()

All registered metric names, sorted.

Source code in src/scale_forecasting/metrics/__init__.py

def list_metrics() -> list[str]:
    """All registered metric names, sorted."""
    return sorted(_REGISTRY)

loss_of

loss_of(metric, value)

value restated as a loss: non-negative, zero is perfect, smaller is better (pure).

One direction map, one conversion, for every caller that has to rank models — the optimiser, the inverse-error weighting and the pruner. Each of those wants the same thing, and each used to hard-code its own idea of what "worse" means.

The conversions:

• lower → the value itself.
• higher → 1 - value. Both coverage and r2 are bounded above by 1.0, so the shortfall from perfection is a non-negative loss. Returning -value would rank identically but be negative, and a negative loss cannot be inverted into a weight.
• zero → abs(value). A bias of −0.1 is better than one of +0.4, and both are worse than 0.
• A metric this table does not know, or a NaN, → inf: unrankable, so it can never win.

Two caveats worth stating rather than burying. coverage is treated as higher-is-better although it is really best at nominal — 98% coverage from a wildly over-wide interval is not better than 80% from a calibrated one. Judging it against nominal needs the interval's α, which is not in the panel, and interval_score is the metric that already penalises width honestly. And interval_width is scored lower-is-better, which in isolation rewards an interval of zero width; it is a diagnostic to read beside coverage, not a decision_metric to optimise alone.

Source code in src/scale_forecasting/metrics/__init__.py

def loss_of(metric: str, value: float) -> float:
    """``value`` restated as a **loss**: non-negative, zero is perfect, smaller is better (pure).

    One direction map, one conversion, for every caller that has to rank models — the optimiser,
    the inverse-error weighting and the pruner. Each of those wants the same thing, and each used
    to hard-code its own idea of what "worse" means.

    The conversions:

    * ``lower`` → the value itself.
    * ``higher`` → ``1 - value``. Both ``coverage`` and ``r2`` are bounded above by ``1.0``, so the
      shortfall from perfection is a non-negative loss. Returning ``-value`` would rank
      identically but be *negative*, and a negative loss cannot be inverted into a weight.
    * ``zero`` → ``abs(value)``. A bias of −0.1 is better than one of +0.4, and both are worse
      than 0.
    * A metric this table does not know, or a NaN, → ``inf``: unrankable, so it can never win.

    Two caveats worth stating rather than burying. ``coverage`` is treated as higher-is-better
    although it is really best *at nominal* — 98% coverage from a wildly over-wide interval is not
    better than 80% from a calibrated one. Judging it against nominal needs the interval's α,
    which is not in the panel, and `interval_score` is the metric that already penalises width
    honestly. And ``interval_width`` is scored lower-is-better, which in isolation rewards an
    interval of zero width; it is a diagnostic to read beside `coverage`, not a `decision_metric`
    to optimise alone.
    """
    if value != value:  # NaN — a metric that could not be computed ranks last, never first
        return float("inf")
    # Read through the registry rather than the frozen `METRIC_DIRECTION` snapshot, so a metric
    # registered after this module was imported still ranks correctly.
    metric_cls = _REGISTRY.get(metric)
    direction = None if metric_cls is None else metric_cls.direction
    if direction == "higher":
        return 1.0 - value
    if direction == "zero":
        return abs(value)
    if direction == "lower":
        return value
    return float("inf")  # unknown metric: no opinion is safer than a wrong one

register

register(metric_cls)

Register a metric class under its name. Returns the class so it doubles as a decorator. Raises on a missing, duplicate or unusable name.

Source code in src/scale_forecasting/metrics/base_metric.py

def register(metric_cls: type[BaseMetric]) -> type[BaseMetric]:
    """Register a metric class under its ``name``. Returns the class so it
    doubles as a decorator. Raises on a missing, duplicate or unusable name.
    """
    name = getattr(metric_cls, "name", None)
    if not name:
        raise ConfigError(f"{metric_cls.__name__} must set a class-level 'name' before register()")
    # The name becomes a FLOAT64 column on `forecast_metadata` and a generated SQL identifier in
    # the leaderboard projection, so a name that is not a bare identifier would produce SQL that
    # fails at query time, far from the file that caused it.
    if not name.isidentifier():
        raise ConfigError(f"metric name '{name}' is not a valid column identifier")
    existing = _REGISTRY.get(name)
    if existing is not None and existing is not metric_cls:
        raise ConfigError(
            f"duplicate metric name '{name}': {existing.__name__} vs {metric_cls.__name__}"
        )
    _REGISTRY[name] = metric_cls
    return metric_cls