Running Evaluations

This guide covers how to run comprehensive benchmarks comparing DP algorithms.

Overview

The evaluation workflow:

Select a task (e.g., mean estimation, linear regression)
Configure algorithms to compare (or use defaults)
Choose concrete privacy budgets (pure DP, approximate DP, or zCDP)
Choose instance generators (synthetic) or datasets (real)
Run the evaluation and analyze results

Quick Start

Run an evaluation from the command line:

dp-testing-platform evaluate \
  --task-name logistic_regression \
  --epsilons 0.1 1.0 10.0 \
  --deltas 1e-5 \
  --alg-seeds 0 1 2 \
  --instance-seeds 0

With no algorithms or instance generators specified, the run sweeps the task's full default rosters (every registered algorithm across its default hyperparameter grid, on every default generator). The resolved grid prints before any cell runs — check the cell count there before launching a big task this way, and pin specific algorithms/generators in a config file for a scoped run.

CLI Options

Option	Description
`--task-name`	Task ID (e.g., `mean_estimation`, `linear_regression`)
`--epsilons`	Convenience values for constructing pure/approximate-DP budgets
`--deltas`	Optional delta values for that CLI convenience syntax
`--alg-seeds`	Random seeds for algorithm randomness
`--instance-seeds`	Random seeds for data generation
`--config`	Path to JSON configuration file
`--no-save`	Don't persist results to disk
`--verbose`	Print detailed progress
`--reset-results`	Clear previous results for this task

Privacy budgets

The engine's privacy axis is a sequence of concrete PureDPBudget, ApproxDPBudget, or ZCDPBudget values. See Privacy budgets for the spending rules and Python API. The CLI's --epsilons/--deltas flags are only construction sugar: an omitted or zero delta creates pure-DP budgets; a positive delta creates approximate-DP budgets. They are not separate engine axes and do not construct zCDP budgets.

Epsilon-bearing budgets

Epsilon controls the privacy-utility tradeoff:

Lower epsilon = stronger privacy, higher error
Higher epsilon = weaker privacy, lower error

Common testing ranges: 0.01, 0.1, 1.0, 10.0, 100.0

Approximate DP and zCDP

ApproxDPBudget(eps, delta) carries both parameters as one allowance; pure-DP mechanisms may soundly spend only its epsilon, while native approximate-DP and Gaussian mechanisms use the full family-specific accounting. ZCDPBudget(rho) is a distinct family and never shares identity with an epsilon-bearing budget.

Available Tasks

Task ID	Description
`mean_estimation`	Estimate mean of d-dimensional data
`median_estimation`	Estimate median of 1D data
`linear_regression`	Linear model fitting
`single_parameter_estimation`	One coefficient of interest from a linear regression
`logistic_regression`	Binary classification
`m_estimation`	General M-estimation problems

Listing Algorithms for a Task

from dp_testing_platform.tasks import ALL_TASKS

task = ALL_TASKS["mean_estimation"]
for name, alg_class in task.get_algorithms().items():
    print(f"  {name}")

Output:

  SampleMean
  GaussianMechanismMeanEstimation
  CoinPressMeanEstimation

Using Configuration Files

For complex experiments, use JSON config files:

{
  "task_name": "linear_regression",
  "budgets": [
    {"eps": 0.1, "delta": 1e-5},
    {"eps": 1.0, "delta": 1e-5},
    {"rho": 0.5}
  ],
  "alg_seeds": [0, 1, 2, 3, 4],
  "instance_seeds": [0, 1, 2],
  "instance_generators": [
    {
      "path": "dp_testing_platform.tasks.linear_regression.instance_generators:GaussianCovariatesAndNoise",
      "params": {
        "dim": 10,
        "n_samples": 1000,
        "response_noise": 0.1
      }
    }
  ],
  "verbose": true
}

Run with:

dp-testing-platform evaluate --config my_config.json

Config Path

--config is optional — flags alone suffice. Any flag given alongside a config file overrides the matching config key.

Example Configs

The package ships ready-to-adapt config files under dp_testing_platform/scripts/example_configs/ — copy one and edit it:

File	Command	Purpose
`evaluate_synthetic_config.json`	`evaluate`	Sweep on synthetic instance generators
`evaluate_real_config.json`	`evaluate`	Sweep on real datasets
`evaluate_dphyper.json`	`evaluate`	Evaluate a DP-tuned (Liu–Talwar) algorithm
`register_algorithms_config.json`	`register-algorithms`	Register algorithms into a task registry
`register_instances_config.json`	`register-instances`	Register instance generators into a task registry

Instance Generators

Instance generators create synthetic data for evaluation. Each task has registered generators.

Example: Mean Estimation Generators

from dp_testing_platform.tasks import ALL_TASKS

task = ALL_TASKS["mean_estimation"]
for gen_class in task.get_instance_generators():
    print(f"  {gen_class.__name__}")

Output:

  SimpleGaussian
  ClippedNormal
  ClippedLaplacian

Specifying Generators in Config

{
  "instance_generators": [
    {
      "path": "dp_testing_platform.tasks.mean_estimation.instance_generators:SimpleGaussian",
      "params": {
        "dimension": 10,
        "n_samples": 1000,
        "mean_radius": 5.0
      }
    }
  ]
}

Real Data Evaluation

To evaluate on real datasets, use --dataset-specs:

dp-testing-platform evaluate \
  --task-name linear_regression \
  --dataset-specs openml_123:full \
  --epsilons 1.0 10.0 \
  --deltas 1e-5

The format is dataset_id:converter_id. See Adding Datasets for details.

Understanding Results

All persistence for a sweep lives in one workdir (run_experiment(..., workdir="results/my_sweep"); the CLI uses <results_root>/<task_name>/). The engine creates, loads, and saves everything in it — you never construct or save a registry yourself:

results/my_sweep/                    # the workdir
  algorithm_registry.json            # Algorithm configurations (UUID -> config)
  instance_generator_registry.json   # Instance generator configurations
  results.csv                        # One row per completed cell
  shards/                            # Only present while a run is in flight
    shard-<run_id>-<pid>.csv         #   (or after an interrupted run)

Every completed cell — one (instance generator, algorithm, budget, instance seed, algorithm seed) combination — is persisted to a shard file the moment it finishes, so an interrupted run loses at most the cells that were mid-execution. Re-running the same call resumes: mode="resume" (the default) recomputes only the missing cells and returns the full stored+new union. A run that completes consolidates everything into a single results.csv (openable in Excel or any editor) and removes shards/.

The row's budget value is the canonical dataclass representation and participates in resume/output-sidecar identity. budget_family plus sparse eps, delta, and rho columns are non-identity conveniences for filtering; do not reconstruct a cell key from them.

Parallelism

run_experiment(..., n_jobs=4) runs cells in parallel worker processes (via joblib/loky); n_jobs=-1 uses all cores. The per-run time_limit still applies to each cell.

Known limitation — stopping a parallel run. Prefer interrupting a running sweep with Ctrl-C (KeyboardInterrupt): the parent shuts the loky workers down cleanly and completed cells stay saved under <workdir>/shards/ (re-run in the default resume mode to continue). Hard-killing the parent with kill -9 instead leaves the loky worker processes and their resource tracker orphaned — durability is unaffected (shards keep being written), but the orphans idle (~150 MB RSS each) until cleared manually. If that happens, clear them with pkill -f popen_loky.

Memory model and the per-cell memory cap

Nothing run by the platform should be able to crash the machine — the engine assumes nothing about how much memory a cell (instance generation + algorithm + metric evaluation) might try to allocate, since users run their own algorithm code through it. Two mechanisms enforce this:

Per-cell memory cap, on by default. Each cell runs in a supervised child process. A watchdog in the supervising process polls the resident memory (RSS) of the cell's whole process tree every ~100 ms; if it exceeds memory_limit_gb (run_experiment(memory_limit_gb=4.0) by default), the cell is killed and recorded with status "memory_limit_exceeded" (with the observed RSS in failure_reason), and the sweep continues. The same mechanism is used on every OS — no rlimits or platform branches. Pass memory_limit_gb=None to disable the cap (only in environments where a runaway allocation is acceptable); with both memory_limit_gb=None and time_limit=None, cells run in-process.
Instances are materialized only inside cell children. The parent plans cells as (instance generator, instance seed) references — the instance arrays are generated (or loaded, for real datasets) inside the supervised child, and only the scalar metric value is returned. At most n_jobs instances are alive at any moment, each freed when its cell's child exits, regardless of how large the sweep grid is.

Caveats: the cap measures the cell child's tree RSS, which on POSIX includes pages inherited from the worker process at fork — caps well below ~0.5 GB are mostly consumed by interpreter baseline. A cell killed by the cap (or by time_limit) reports np.inf as its performance, like other failed cells.

Analyzing Results

Use the analyze-results command, pointing it at the sweep's workdir (the --workdir). Flags alone suffice — the JSON --config is optional:

dp-testing-platform analyze-results \
  --workdir results/linear_regression \
  --output-dir results/linear_regression/analysis

It writes performance_ratios.csv, one cdf_<alg_uuid>.csv per algorithm, and a cdf_legend.csv mapping each UUID to its human-readable algorithm name. Optional convenience filters: --epsilons, --deltas, --alg-uuids, --instance-gen-uuids.

Or load results programmatically — load_results(workdir) works in a fresh process, from disk alone, and returns rows already joined to human-readable alg_name and instance_name columns (read from the persisted registries):

from dp_testing_platform import load_results

df = load_results("results/my_sweep")
print(df[["alg_name", "instance_name", "budget", "alg_performance", "status"]])

Loading applies an explicit dtype schema (floats are written with enough precision to round-trip exactly) and includes any rows still sitting in shards from an interrupted run.

Every row also carries two provenance stamps: library_version (the installed package version) and library_commit (the git short hash of the library source at runtime or build time; "unknown" for a source tarball with no Git or embedded build metadata). The stamps identify which code computed a row — algorithm UUIDs hash configuration, not code — but are not part of the row's identity key, so upgrading the library never forces recomputation; filter on the stamps in analysis when mixing versions matters. CSVs written before stamping existed load with both columns defaulted to "unknown".

Note on library_commit for pip installs. Wheels and source distributions built from a Git checkout embed its short commit hash, so editable and non-editable installs retain the same provenance. A source tarball created without Git metadata or an embedded build stamp records "unknown" honestly.

Every row also records runtime_s: the cell's whole wall time (instance generation + algorithm run + metric evaluation — everything the time limit governs). Non-success rows record their real elapsed time too (timeout rows roughly the cap, memory-killed rows the time-to-kill). Like the stamps, runtime_s is not part of the row's identity key; CSVs written before the column existed load it as NaN.

Experiment Modes

The --mode option controls behavior when results already exist:

Mode	Behavior
`resume`	Compute only the cells missing from stored results; return the full union (default)
`overwrite`	Re-run and replace existing results

Tips

Start small: Use --no-save and few seeds for initial testing
Use verbose mode: Add --verbose to see progress
Parallel seeds: More algorithm seeds reduce variance in results
Instance seeds: More instance seeds test robustness to data variation

Next Steps

Adding Algorithms - Benchmark your own algorithm
Adding Datasets - Use real-world data
CLI Reference - Full command documentation