Open benchmark · reproducible runs

Measure the utility cost of privacy.

Bring your own differentially private algorithm, run it over real datasets at a fixed privacy grid, and compare it with off-the-shelf baselines.

Run the benchmark Browse datasets

17studies

58benchmark instances

7tasks

55algorithms

Start Install and run Go from a clean environment to your first finite result. Explore Inspect the corpus Trace benchmark instances back to studies, claims, and source regressions. Compare Analyze performance Build health summaries, privacy–utility curves, and performance profiles.

Quickstart

Install the library (from source until the first PyPI release):

git clone https://github.com/saminulh/dp-testing-platform.git
pip install -e /path/to/dp-testing-platform

Point at the benchmark corpus and load its real regression datasets:

from dp_testing_platform import get_real_datasets_for_task, LinearRegression

CORPUS = "path/to/corpus/regressions"

datasets = get_real_datasets_for_task(LinearRegression, CORPUS)
print(f"{len(datasets)} linear_regression datasets")

Each dataset is a generator you pass straight to run_experiment. The full loop — wrap your algorithm, run it at the protocol grid, read the per-dataset results, and export a shareable run archive — is the Running the benchmark guide. New to the library? Start with Getting started.

What is in the corpus

The Datasets catalogue documents every source study — its claims, quotes, and the backing regression datasets you benchmark against — with full provenance from each paper's own regression down to the graduated instance.

The framework consumes processed data (X, y, metric); the studies catalogue documents where it came from. This site is the reader-facing join of the two: the framework guide, the corpus catalogue, and the full API reference, each assembled at build time from its real source (never hand-duplicated).