CLI Reference

Complete reference for all dp-testing-platform commands.

Overview

dp-testing-platform <command> [options]

Available commands:

Command	Description
init	Initialize a new project directory
generate-task	Generate a new task from template
evaluate	Run algorithm evaluations
register-algorithms	Register algorithms in the registry
register-instances	Register instance generators in the registry
analyze-results	Analyze evaluation results

Every command accepts --help for its full option list. Ready-to-adapt config files for the config-driven commands ship under dp_testing_platform/scripts/example_configs/ (see Running Evaluations).

---

init

Initialize a new dp-testing-platform project directory.

dp-testing-platform init [options]

Options

Option	Required	Default	Description
--path	No	.	Project directory to initialize
--datasets	No	-	Path to existing datasets/raw directory
--no-notebook	No	false	Skip copying the intro notebook

Example

# Initialize in current directory
dp-testing-platform init

# Initialize in a specific directory
dp-testing-platform init --path ./my-project

# Initialize with existing datasets
dp-testing-platform init --datasets /path/to/datasets/raw

Output

Creates:

• config.py - Configuration for dataset paths
• pyrightconfig.json - IDE type checking configuration (for VS Code/Pylance)
• intro_notebook.ipynb - Example notebook (unless --no-notebook)
• datasets/raw/ - Local datasets directory (if --datasets not provided)

---

generate-task

Generate a new task scaffold from the template.

dp-testing-platform generate-task --task-name <name> [--task-dir <dir>]

Options

Option	Required	Default	Description
--task-name	Yes	-	Name of the new task (snake_case recommended)
--task-dir	No	Current directory	Directory where the task folder will be created

Example

# Create a new task in the current directory
dp-testing-platform generate-task --task-name quantile_estimation

# Create in a specific directory
dp-testing-platform generate-task --task-name my_task --task-dir ./tasks/

Output

Creates a directory structure:

<task_name>/
├── __init__.py
├── task.py
├── types.py
├── metrics.py
├── algorithms/
│   ├── __init__.py
│   ├── template_algorithm.py
│   └── lifted.py
└── instance_generators/
    ├── __init__.py
    └── template_instance.py

---

evaluate

Run algorithm evaluations on synthetic or real data.

dp-testing-platform evaluate [options]

Options

Option	Default	Description
--config	-	Optional path to a JSON config; omit to drive entirely from flags
--task-name	-	Task ID (e.g., linear_regression)
--epsilons	-	Sugar for constructing pure/approximate-DP budgets
--deltas	-	Optional delta values used only by that sugar
--alg-seeds	-	Random seeds for algorithm randomness
--instance-seeds	-	Random seeds for synthetic data generation (ignored with --dataset-specs: real data is deterministic)
--dataset-specs	-	Real data specs (dataset_id:converter_id)
--raw-directory	datasets/raw	Path to raw datasets
--mode	resume	How to handle existing results (resume or overwrite)
--time-limit	-	Max seconds per algorithm run
--no-save	false	Don't persist results to disk
--reset-results	false	Clear previous results
--hard-reset	false	Clear all registries and results
--verbose	false	Print detailed progress
--results-root	-	Parent dir for the run; the workdir is <results-root>/<task_id>
--progress-file	-	File to write progress updates
--output-file	-	File to write output UUIDs

When no algorithms and no instance generators are given (whether by flag or in the config file), evaluate runs the task's full default rosters — every registered algorithm across its default hyperparameter grid, and every default instance generator. The resolved grid (N algorithms × M instance generators × …) is printed before the run, and an empty roster is a hard error rather than a silent zero-cell "success".

The engine has one budget axis containing concrete typed values. The CLI flags --epsilons and --deltas are construction sugar, not independent engine axes: zero or omitted delta creates PureDPBudget; positive delta creates ApproxDPBudget. Use a config's budgets list to include ZCDPBudget values.

Examples

# Quick evaluation with minimal options. Flags alone sweep the task's full
# default rosters, so pick a small task/grid for smoke runs
# (logistic_regression's default roster is 20 algorithms x 1 generator).
dp-testing-platform evaluate \
  --task-name logistic_regression \
  --epsilons 1.0 10.0 \
  --deltas 1e-5 \
  --alg-seeds 0 \
  --instance-seeds 0 \
  --no-save

# Evaluation with configuration file (pin specific algorithms and instance
# generators there for a scoped run)
dp-testing-platform evaluate --config my_config.json

# Evaluation on real data
dp-testing-platform evaluate \
  --task-name linear_regression \
  --dataset-specs my_dataset:full \
  --epsilons 1.0 \
  --deltas 1e-5

# Verbose evaluation with custom results directory
dp-testing-platform evaluate \
  --task-name logistic_regression \
  --epsilons 0.1 1.0 10.0 \
  --deltas 1e-5 \
  --alg-seeds 0 1 2 \
  --instance-seeds 0 \
  --results-root ./my_results \
  --verbose

Configuration File Format

{
  "task_name": "linear_regression",
  "budgets": [
    {"eps": 0.1, "delta": 1e-5},
    {"eps": 1.0, "delta": 1e-5},
    {"rho": 0.5}
  ],
  "alg_seeds": [0, 1, 2, 3, 4],
  "instance_seeds": [0, 1, 2],
  "instance_generators": [
    {
      "path": "dp_testing_platform.tasks.linear_regression.instance_generators:GaussianCovariatesAndNoise",
      "params": {
        "dim": 10,
        "n_samples": 1000
      }
    }
  ],
  "algorithms": [
    {
      "path": "dp_testing_platform.tasks.linear_regression.algorithms:OLS",
      "hyperparams": {}
    }
  ],
  "verbose": true
}

---

register-algorithms

Register algorithms in the task's algorithm registry.

dp-testing-platform register-algorithms [options]

Options

Option	Default	Description
--config	configs/register_algorithms_config.json	Path to JSON config
--task	-	Task module path (package.module:ClassName)
--hard-reset	false	Clear existing registry
--results-root	-	Custom results directory
--verbose	false	Enable verbose output
--progress-file	-	File for progress updates
--output-file	-	File to write algorithm UUIDs

Example

dp-testing-platform register-algorithms \
  --task dp_testing_platform.tasks.linear_regression:LinearRegression \
  --verbose

Configuration File Format

{
  "task": "dp_testing_platform.tasks.linear_regression:LinearRegression",
  "algorithms": [
    {
      "path": "dp_testing_platform.tasks.linear_regression.algorithms:OLS",
      "hyperparams": {}
    },
    {
      "path": "my_algorithms:MyDPAlgorithm",
      "hyperparams": {"clip_norm": 1.0}
    }
  ]
}

---

register-instances

Register instance generators in the task's instance registry.

dp-testing-platform register-instances [options]

Options

Option	Default	Description
--config	configs/register_instances_config.json	Path to JSON config
--task	-	Task module path (package.module:ClassName)
--hard-reset	false	Clear existing registry
--results-root	-	Custom results directory
--verbose	false	Enable verbose output
--progress-file	-	File for progress updates
--output-file	-	File to write generator UUIDs

Example

dp-testing-platform register-instances \
  --task dp_testing_platform.tasks.mean_estimation:MeanEstimationTask \
  --verbose

Configuration File Format

{
  "task": "dp_testing_platform.tasks.mean_estimation:MeanEstimationTask",
  "instances": [
    {
      "path": "dp_testing_platform.tasks.mean_estimation.instance_generators:SimpleGaussian",
      "params": {"dimension": 10, "n_samples": 1000}
    }
  ]
}

---

analyze-results

Analyze saved evaluation results and compute performance metrics.

dp-testing-platform analyze-results [options]

Options

Option	Default	Description
--config	-	Optional path to a JSON config; omit to drive entirely from flags
--workdir	-	Sweep workdir (<results-root>/<task_id>) containing results.csv and the registries
--alg-uuids	-	Filter to specific algorithm UUIDs
--instance-gen-uuids	-	Filter to specific generator UUIDs
--epsilons	-	Convenience filter for epsilon-bearing budget rows
--deltas	-	Convenience filter for approximate-DP rows
--output-dir	<workdir>/analysis	Output directory
--progress-file	-	File for progress updates

Example

# Analyze all results for a task
dp-testing-platform analyze-results \
  --workdir results/linear_regression

# Analyze specific epsilon range
dp-testing-platform analyze-results \
  --workdir results/mean_estimation \
  --epsilons 1.0 10.0

Output

Creates analysis files in <output_dir>/:

• performance_ratios.csv - Performance ratio data
• cdf_<uuid>.csv - CDF data for each algorithm
• cdf_legend.csv - Maps each algorithm UUID to its human-readable name

Configuration File Format

The config file is optional — the flags above suffice for a basic run. Use it to pin filters/registry paths for reproducible protocol runs. Any flag given overrides the matching config key.

{
  "workdir": "results/linear_regression",
  "alg_registry": "results/linear_regression/algorithm_registry.json",
  "instance_registry": "results/linear_regression/instance_generator_registry.json",
  "epsilons": [1.0, 10.0],
  "deltas": [1e-5],
  "output_dir": "results/linear_regression/analysis"
}

---

Common Patterns

Quick Testing

For quick tests without persisting results (a flags-only run sweeps the task's full default rosters, so prefer a small task like logistic_regression — the roster line printed at startup shows the cell count before any work runs):

dp-testing-platform evaluate \
  --task-name logistic_regression \
  --epsilons 1.0 \
  --deltas 1e-5 \
  --alg-seeds 0 \
  --instance-seeds 0 \
  --no-save \
  --verbose

Full Benchmark

For comprehensive benchmarks (the full default rosters across a large grid — expect a long run):

dp-testing-platform evaluate \
  --task-name linear_regression \
  --epsilons 0.01 0.1 1.0 10.0 100.0 \
  --deltas 1e-5 1e-6 1e-7 \
  --alg-seeds 0 1 2 3 4 5 6 7 8 9 \
  --instance-seeds 0 1 2 3 4 \
  --verbose

Using Config Files

Create a config file for reproducible experiments:

# Create config (no algorithms/instance_generators listed -> the task's full
# default rosters run)
cat > my_experiment.json << 'EOF'
{
  "task_name": "logistic_regression",
  "budgets": [
    {"eps": 0.1, "delta": 1e-5},
    {"eps": 1.0, "delta": 1e-5},
    {"eps": 10.0, "delta": 1e-5}
  ],
  "alg_seeds": [0, 1, 2, 3, 4],
  "instance_seeds": [0, 1, 2],
  "verbose": true
}
EOF

# Run experiment
dp-testing-platform evaluate --config my_experiment.json

---

Exit Codes

Code	Meaning
0	Success
1	Error (invalid command, missing files, etc.)

Environment

The CLI automatically adds the current working directory to sys.path, allowing import of local task modules.