Running the Benchmark

This guide takes your own DP algorithm from a bare class to a per-dataset results table on the real benchmark corpus — then packages the run as a portable, hash-verified archive you can share. It assumes the library is already installed (see Getting Started).

The corpus and the protocol below are for linear_regression, the task the benchmark currently covers. The same five steps apply to any task once its corpus lands; swap in that task and its datasets.

The path is: wrap your algorithm → point at the corpus → run at the protocol grid → read the results → export a run archive.

1. Wrap your algorithm

Your algorithm is a BaseAlgorithm subclass registered on the task. Set TASK, declare HYPERPARAM_INFO, and implement run() — the single method the engine calls. Data arrives already in the task's format (LinearRegression.Input), so run() does no adaptation of its own; it returns a LinearRegression.Output.

Put it in an importable module, not your run script (see the warning below):

# my_lr_algorithm.py
import numpy as np

from dp_testing_platform import LinearRegression, BaseAlgorithm
from dp_testing_platform.core import accounting
from dp_testing_platform.core.dp_budget import DPBudget


@LinearRegression.register_algorithm
class NoisyOLS(BaseAlgorithm):
    """Ordinary least squares with Gaussian noise added to the fit.

    A stand-in for your own DP algorithm — it shows the wrapping and
    benchmarking loop, not a rigorous mechanism.

    Hyperparameters:
        noise_multiplier: Scales the Gaussian noise added to each
            coefficient (noise std is ``noise_multiplier / eps``). Default 1.0.
    """

    TASK = LinearRegression
    HYPERPARAM_INFO = {
        "noise_multiplier": {
            "type": float,
            "description": "Scales the Gaussian noise added to the OLS fit.",
            "default": 1.0,
        },
    }

    def __init__(self, noise_multiplier=None):
        super().__init__()
        # Fall back to the declared default only when the arg is omitted.
        self.noise_multiplier = (
            noise_multiplier
            if noise_multiplier is not None
            else self.HYPERPARAM_INFO["noise_multiplier"]["default"]
        )

    def run(self, metadata, input, budget: DPBudget, seed: int):
        eps = accounting.spend_as_pure(budget).eps
        beta = np.linalg.lstsq(input.X, input.y, rcond=None)[0]
        rng = np.random.default_rng(seed)
        noise = rng.normal(0.0, self.noise_multiplier / eps, size=beta.shape)
        return LinearRegression.Output(beta=beta + noise)

That is the whole contract. For the deeper authoring rules — how to design problem-agnostic hyperparameters, read domain bounds from metadata, and seed correctly — see Adding Algorithms.

Define algorithms in an importable module, not your entry script

On macOS (and Windows) the engine runs each cell in a spawn subprocess, which re-imports your module. A class defined in your top-level script (__main__) is sent to the worker by value, duplicating the LinearRegression task object and breaking the framework's task-identity checks (you will see Cannot lift: ... errors and inf results). Defining your algorithm in a separate module — my_lr_algorithm.py — and importing it sends it by reference and works correctly.

2. Point at the benchmark corpus

The corpus is a directory of real regression datasets, each with a converter that produces a linear_regression instance whose error metric is the L2 distance to the non-private OLS fit (l2_error_to_ols). Load them with get_real_datasets_for_task, pointing at the corpus directory:

from dp_testing_platform import get_real_datasets_for_task, LinearRegression

CORPUS = "path/to/corpus/regressions"

datasets = get_real_datasets_for_task(LinearRegression, CORPUS)
print(f"{len(datasets)} linear_regression datasets")
for name in list(datasets)[:5]:
    print(name)

datasets maps each instance name to a RealInstanceGenerator you pass straight to run_experiment. The names are what the result store records, so they are also how you read a dataset back later. For example:

23 linear_regression datasets
icpsr_113559_reg_11:full
icpsr_131981_reg_15__1:full
icpsr_197964_reg_2__20:full
icpsr_169121_reg_11:full
icpsr_197964_reg_2__27:full

To browse what is available before running — sizes and which tasks each dataset supports — use list_real_datasets, which reads each dataset's datasheet without touching the data arrays:

from dp_testing_platform import list_real_datasets

for row in list_real_datasets(CORPUS)[:3]:
    print(row["id"], "n=", row["n_samples"], "d=", row["n_features"])

icpsr_113559_reg_11 n= 1080 d= 110
icpsr_113559_reg_12 n= 1081 d= 110
icpsr_113559_reg_13 n= 1076 d= 109

Getting the corpus onto your machine

Today you point at a local corpus directory. A hosted fetch verb that downloads a pinned corpus snapshot for you — so you can benchmark against the exact same datasets as everyone else — is planned (issue #188). Until then, obtain the corpus/regressions directory and pass its path as above.

3. Run the benchmark

run_experiment sweeps every (algorithm, dataset, budget, seed) cell and persists results to a workdir. The benchmark protocol fixes the grid and the resource caps so runs are comparable across algorithms and machines:

Axis	Protocol value	Notes
budget	ApproxDPBudget(eps, 1e-6) for eps 0.1, 0.3, 1.0, 3.0, 10.0	log-spaced approximate-DP allowances
algorithm seeds	up to ~20 per cell	distributional outputs need enough seeds for stable medians; a deterministic algorithm needs only 1
instance seed	0 (a single seed)	real datasets are fixed, converters deterministic
time_limit	60 seconds per cell	a cap, recorded as protocol
memory_limit_gb	4 GB per cell	a cap, recorded as protocol

Start with a small slice to check everything runs, then scale up the datasets, budgets and seeds to the full protocol. This script compares your algorithm against a built-in DP method and the OLS baseline (the metric's reference — it scores 0):

# run_benchmark.py
from dp_testing_platform import (
    ApproxDPBudget,
    LinearRegression,
    run_experiment,
    load_results,
    get_real_datasets_for_task,
)
from my_lr_algorithm import NoisyOLS

CORPUS = "path/to/corpus/regressions"

# The full protocol epsilon grid — see the "quick first run" note below.
EPSILONS = [0.1, 0.3, 1.0, 3.0, 10.0]
DELTA = 1e-6
BUDGETS = [ApproxDPBudget(eps, DELTA) for eps in EPSILONS]


def main():
    datasets = get_real_datasets_for_task(LinearRegression, CORPUS)
    # A two-dataset first run; use list(datasets.values()) for the whole corpus.
    subset = list(datasets.values())[:2]

    # Built-in algorithms are fetched by name; instantiate the ones you want.
    builtins = LinearRegression.get_algorithms()
    algs = [NoisyOLS(), builtins["DPGradientDescentLR"](), builtins["OLS"]()]

    run_experiment(
        task=LinearRegression,
        algs=algs,
        instance_generators=subset,
        budgets=[ApproxDPBudget(1.0, DELTA), ApproxDPBudget(10.0, DELTA)],
        # Use BUDGETS above for the full protocol.
        alg_seeds=2,  # scale to ~20 for the protocol
        instance_generator_seeds=1,  # real datasets are fixed, so one seed
        workdir="results/lr_benchmark",
        time_limit=60.0,  # protocol cap: 60 s per cell
        memory_limit_gb=4.0,  # protocol cap: 4 GB per cell
        n_jobs=4,  # run cells in parallel worker processes
    )

    results = load_results("results/lr_benchmark")
    print(results.groupby(["alg_name", "budget"])["l2_error_to_ols"].median())


if __name__ == "__main__":
    main()

python run_benchmark.py

The printed index contains the canonical ApproxDPBudget(...) strings, so rows from another privacy family cannot collapse onto the same epsilon value.

The engine owns persistence — you never construct a registry. run_experiment defaults to mode="resume", so re-running the same call recomputes only the missing cells: a full-corpus, full-grid, 20-seed sweep is an overnight job you can interrupt and restart freely. n_jobs=-1 uses all cores. The Evaluation guide covers the workdir, resume, and parallelism in depth.

Statuses are results, not gaps

Every cell produces a row. When a cell does not succeed, the engine records why rather than dropping it — the honesty rules that keep the benchmark comparable:

• A cell that exceeds time_limit is recorded with status timeout; one that exceeds memory_limit_gb with memory_limit_exceeded. The sweep continues — one failing cell never loses the others.
• Non-finite outputs are failures. An output or metric containing inf is recorded as output_infinity (arithmetic that diverged or overflowed); one containing NaN as output_invalid (an undefined operation). An algorithm that raises reports no_release or error.
• Finite-but-huge error stays a success. The value is the measurement — there are no magnitude thresholds. A method that returns a wildly wrong but finite estimate is graded, not hidden.

Non-success rows carry np.inf in alg_performance, a human-readable failure_reason, and count against an algorithm's coverage; they are excluded from aggregates like the medians above. An algorithm's feasible envelope — which cells it can complete under the caps — is itself a benchmark result.

4. Read the results

A completed run consolidates into a single workdir:

results/lr_benchmark/
  results.csv                        # one row per completed cell
  algorithm_registry.json            # algorithm configs (UUID -> config)
  instance_generator_registry.json   # dataset/instance configs
  outputs.npz                        # each cell's released output (for post-hoc metrics)

(A shards/ directory appears only while a run is in flight or after an interruption; it is gone once the run consolidates.)

load_results(workdir) reloads the rows from disk alone — in a fresh process, without re-running — already joined to human-readable alg_name and instance_name columns. Each success row carries the task's key-metric columns alongside the primary alg_performance scalar; for linear_regression those are l2_error_to_ols (the benchmark metric), normalized_mse (a cost — the fraction of variance unexplained, lower is better), and mse:

from dp_testing_platform import load_results

results = load_results("results/lr_benchmark")
cols = ["alg_name", "instance_name", "budget", "status", "l2_error_to_ols", "normalized_mse"]
selected = results[
    (results["budget_family"] == "approx_dp")
    & (results["eps"] == 10.0)
    & (results["delta"] == 1e-6)
]
print(selected[cols].to_string(index=False))

To go further — performance profiles, per-algorithm CDFs, or recomputing a new metric from the stored outputs without re-running any algorithm — see Running Evaluations.

5. Produce a run archive

Once a run is complete, package it as a portable, hash-verified record with export_run_archive:

from dp_testing_platform import export_run_archive

export_run_archive("results/lr_benchmark", dest="lr_benchmark_archive")

The archive is a self-describing directory: the results split into one results/<Algorithm>.csv per algorithm (readable, replaceable slices), copies of both registries, and a MANIFEST.json describing the run — a content_hash over the results and registries, the algorithm roster, the concrete budget grid and per-cell seed counts, the datasets present, and the execution environment (hostname, Python version, pip freeze). Attach your own context with the optional metadata= argument, recorded verbatim:

export_run_archive(
    "results/lr_benchmark",
    dest="lr_benchmark_archive",
    metadata={"algorithm": "NoisyOLS", "author": "you@example.org"},
)

The manifest makes the run verifiable and reproducible-to-inspect on another machine — the same run produces the same content_hash anywhere. Export refuses a workdir with in-flight shards (consolidate a completed run first) and a destination whose results/ already holds foreign CSVs.

Sharing and submission

The run archive is a record of your run, not a submission format. A hosted leaderboard and a submission flow are planned but not yet defined; when they land, the archive is the intended unit to submit.

Next steps

• Running Evaluations — the workdir, resume/parallelism, and post-hoc analysis in depth
• Adding Algorithms — the full algorithm-authoring rules
• Adding Datasets — benchmark on your own data, not just the corpus
• Linear Regression — the task's input/output types, built-in algorithms, and metrics