Synthetic Data Generator

generator

Pure example-data generator — no I/O.

Deterministic panel math: each series is a sum of readable components — trend + short-cycle & yearly seasonality + holiday bumps + AR(1) noise — then shaped by an archetype (smooth-seasonal, intermittent, trending, promo-spiky, noisy) that applies intermittency, level shifts, and promo spikes. Seasonality is measured in steps (via the shared seasonality maps), so the panel is coherent at any freq — daily, weekly, monthly, hourly — and daily output is byte-for-byte the same as day-count math. Every series draws its parameters from an rng seeded by its own index, so:

• series i is byte-for-byte identical no matter which partition produced it — hence generate_panel(n) equals the union of any partitioning of range(n) (the property the Spark seed job relies on), and
• one master seed reproduces the whole 100k dataset on every deployment.

The five archetypes deliberately favor different models — that's what makes the ensemble and straggler contrasts visible downstream. The golden test fixture is a tiny call into this same code, so tests and shipped data share one path.

Public surface: GenConfig, generate_partition, generate_panel.

Archetype dataclass

Archetype(
    name,
    base,
    trend_frac,
    weekly_amp,
    yearly_amp,
    noise_frac,
    ar1_rho,
    holiday_frac,
    zero_inflation,
    spike_rate,
    spike_mult,
    level_shift_prob,
)

One series "personality". Each field is a (low, high) range the per-series rng samples uniformly, or a scalar probability. Distinct profiles make distinct models win.

GenConfig dataclass

GenConfig(
    history=1460,
    freq="D",
    start="2021-01-01",
    holidays=("US",),
    with_exog=False,
    with_hierarchy=False,
)

How to shape each series — not how many. The count is the caller's id_range / n (a partition of series ids), which keeps the partition-union invariant orthogonal to the per-series knobs. Pure parameters — no source/sink here (that's seed_spark.py).

generate_panel

generate_panel(n, cfg, seed)

Generate the full panel of the first n series — generate_partition(range(n)).

Because each series is seeded by its own index, this is exactly the union of any partitioning of range(n) (the invariant the distributed seed job depends on), and subsetting to the first k series (data.series_limit) is just a prefix of this.

Source code in src/scale_forecasting/data_gen/generator.py

def generate_panel(n: int, cfg: GenConfig, seed: int) -> pd.DataFrame:
    """Generate the full panel of the first ``n`` series — ``generate_partition(range(n))``.

    Because each series is seeded by its own index, this is exactly the union of any
    partitioning of ``range(n)`` (the invariant the distributed seed job depends on), and
    subsetting to the first ``k`` series (``data.series_limit``) is just a prefix of this.
    """
    if n < 0:
        raise ValueError(f"n must be non-negative, got {n}")
    return generate_partition(range(n), cfg, seed)

generate_partition

generate_partition(id_range, cfg, seed)

Generate the long-format panel for a set of series ids.

id_range is any iterable of integer series indices (a partition of the full range in the Spark seed job). Returns a frame with columns ts_id, archetype, ds, y (plus region, category when cfg.with_hierarchy or cfg.with_exog, and promo_flag, price_index, temperature when cfg.with_exog), sorted by ts_id then ds. Pure and deterministic: identical output for identical (id, cfg, seed).

Source code in src/scale_forecasting/data_gen/generator.py

def generate_partition(id_range: Iterable[int], cfg: GenConfig, seed: int) -> pd.DataFrame:
    """Generate the long-format panel for a set of series ids.

    ``id_range`` is any iterable of integer series indices (a partition of the full range in
    the Spark seed job). Returns a frame with columns ``ts_id, archetype, ds, y`` (plus
    ``region, category`` when ``cfg.with_hierarchy`` or ``cfg.with_exog``, and
    ``promo_flag, price_index, temperature`` when ``cfg.with_exog``), sorted by ``ts_id``
    then ``ds``. Pure and deterministic: identical output for identical ``(id, cfg, seed)``.
    """
    index, pos = _time_axis(cfg)
    holiday_mask = _holiday_mask(index, cfg.holidays)
    ds = index.to_numpy()

    frames: list[pd.DataFrame] = []
    for i in id_range:
        comp = _one_series(i, cfg, seed, pos, holiday_mask)
        cols: dict[str, object] = {
            "ts_id": f"s_{i:06d}",
            "archetype": comp["archetype"],
        }
        if "region" in comp:
            cols["region"] = comp["region"]
            cols["category"] = comp["category"]
        cols["ds"] = ds
        cols["y"] = comp["y"]
        for cov in ("promo_flag", "price_index", "temperature"):
            if cov in comp:
                cols[cov] = comp[cov]
        frames.append(pd.DataFrame(cols))

    if not frames:
        base_cols = ["ts_id", "archetype"]
        if cfg.with_hierarchy or cfg.with_exog:
            base_cols.extend(["region", "category"])
        base_cols.extend(["ds", "y"])
        if cfg.with_exog:
            base_cols.extend(["promo_flag", "price_index", "temperature"])
        return pd.DataFrame({c: pd.Series(dtype="object") for c in base_cols})
    return pd.concat(frames, ignore_index=True)

is_holiday_flags

is_holiday_flags(ds, holidays)

Boolean holiday flags for a sequence of dates — the public view of the panel's calendar.

The generator applies holidays as a numeric bump on y (it emits no holiday column), but the source_series table carries an is_holiday column the models read. This derives that flag from the same holidays calendar the bump uses, so the shipped is_holiday agrees with the effect baked into y (parity). Pure — used by the Spark seed transform (data_gen.seed_spark).

ds is any date-like sequence (a column of generate_partition output, a list of date/Timestamp); holidays is the country codes (e.g. ("US",)). Returns a bool array aligned to ds.

Source code in src/scale_forecasting/data_gen/generator.py

def is_holiday_flags(ds: Sequence[object] | pd.Series, holidays: Sequence[str]) -> np.ndarray:
    """Boolean holiday flags for a sequence of dates — the public view of the panel's calendar.

    The generator applies holidays as a numeric *bump* on ``y`` (it emits no holiday column),
    but the ``source_series`` table carries an ``is_holiday`` column the models read. This
    derives that flag from the **same** ``holidays`` calendar the bump uses, so the shipped
    ``is_holiday`` agrees with the effect baked into ``y`` (parity). Pure — used by
    the Spark seed transform (`data_gen.seed_spark`).

    ``ds`` is any date-like sequence (a column of ``generate_partition`` output, a list of
    ``date``/``Timestamp``); ``holidays`` is the country codes (e.g. ``("US",)``). Returns a
    ``bool`` array aligned to ``ds``.
    """
    index = pd.DatetimeIndex(pd.to_datetime(list(ds))).as_unit("ns")
    return _holiday_mask(index, holidays)