Adding Datasets

This guide shows how to add real-world datasets for evaluation.

Current Status

The raw data system is implemented but there's no automated download mechanism yet. You'll need to manually prepare datasets.

Overview

Each dataset lives in its own directory with a converters.py file. Converters are Python functions that load data (in any format) and transform it into task-specific formats.

Directory Structure

datasets/
  raw/
    my_dataset/
      converters.py     # Required: converter functions
      data.csv          # Your data in any format
      # or data.npz, data.json, multiple files, etc.
      info.json         # Source metadata datasheet (optional)

Step-by-Step Guide

1. Create the Dataset Directory

mkdir -p datasets/raw/my_dataset

2. Add Your Data

Store your data in whatever format is convenient:

# Option 1: CSV
import pandas as pd
df = pd.DataFrame({"feature1": [...], "feature2": [...], "target": [...]})
df.to_csv("datasets/raw/my_dataset/data.csv", index=False)

# Option 2: NumPy
import numpy as np
np.savez("datasets/raw/my_dataset/data.npz", features=X, target=y)

# Option 3: Multiple files, JSON, Parquet, etc.

3. Create a Converter

Create datasets/raw/my_dataset/converters.py:

from pathlib import Path
import numpy as np
import pandas as pd

from dp_testing_platform.core.raw_data import converter
from dp_testing_platform.core.metrics import Metric
from dp_testing_platform.tasks.linear_regression import LinearRegression


@converter(task="linear_regression", id="full")
def linear_regression_full(dataset_path: Path):
    """Convert dataset to linear regression format."""
    # Load data in any format you want
    df = pd.read_csv(dataset_path / "data.csv")
    X = df.drop("target", axis=1).values
    y = df["target"].values

    n, d = X.shape
    metadata = LinearRegression.Metadata(dim=d, n_samples=n)
    input_data = LinearRegression.Input(X=X, y=y)

    # Create error metric
    true_beta = np.linalg.lstsq(X, y, rcond=None)[0]
    metric = Metric(
        task=LinearRegression,
        name="l2_error_to_ols",
        fn=lambda output: float(np.linalg.norm(output.beta - true_beta)),
        args={},
    )

    return metadata, input_data, metric


@converter(task="linear_regression", id="subset_1000")
def linear_regression_subset(dataset_path: Path):
    """Use first 1000 samples only."""
    df = pd.read_csv(dataset_path / "data.csv").head(1000)
    X = df.drop("target", axis=1).values
    y = df["target"].values

    n, d = X.shape
    metadata = LinearRegression.Metadata(dim=d, n_samples=n)
    input_data = LinearRegression.Input(X=X, y=y)

    true_beta = np.linalg.lstsq(X, y, rcond=None)[0]
    metric = Metric(
        task=LinearRegression,
        name="l2_error_to_ols",
        fn=lambda output: float(np.linalg.norm(output.beta - true_beta)),
        args={},
    )

    return metadata, input_data, metric

4. Run Evaluation

Use the dataset with the CLI:

dp-testing-platform evaluate \
  --task-name linear_regression \
  --dataset-specs my_dataset:full \
  --epsilons 1.0 10.0 \
  --deltas 1e-5 \
  --alg-seeds 0 1 2

The --dataset-specs format is dataset_id:converter_id. Here --epsilons/--deltas are CLI sugar that constructs concrete ApproxDPBudget values; the engine itself sweeps one typed budget axis.

Converter Function Signature

Converters must:

1. Be decorated with @converter(task=..., id=...)
2. Accept a Path to the dataset directory
3. Return a tuple of (Metadata, Input, Metric)

from pathlib import Path
from dp_testing_platform.core.raw_data import converter

@converter(task="task_id", id="converter_id")
def my_converter(dataset_path: Path) -> tuple[Metadata, Input, Metric]:
    # Load data from dataset_path in any format
    ...

Example: Mean Estimation Dataset

# datasets/raw/temperature_readings/converters.py
from pathlib import Path
import numpy as np

from dp_testing_platform.core.raw_data import converter
from dp_testing_platform.core.metrics import Metric
from dp_testing_platform.tasks.mean_estimation import MeanEstimationTask


@converter(task="mean_estimation", id="daily_averages")
def mean_estimation_daily(dataset_path: Path):
    """Convert temperature readings to mean estimation task."""
    # Load from NPZ file
    data = np.load(dataset_path / "readings.npz")
    readings = data["temperatures"]  # Shape: (n_days, n_sensors)

    n, d = readings.shape
    metadata = MeanEstimationTask.Metadata(dim=d, n_samples=n)
    input_data = MeanEstimationTask.Input(X=readings)

    # Ground truth is the actual mean
    true_mean = np.mean(readings, axis=0)
    metric = Metric(
        task=MeanEstimationTask,
        name="l2_error_to_true_mean",
        fn=lambda output: float(np.linalg.norm(output.mean_est - true_mean)),
        args={},
    )

    return metadata, input_data, metric

Supported Data Formats

Converters can load any format:

Format	Example Loading Code
CSV	pd.read_csv(dataset_path / "data.csv")
NumPy	np.load(dataset_path / "data.npz")
JSON	json.load(open(dataset_path / "data.json"))
Parquet	pd.read_parquet(dataset_path / "data.parquet")
Multiple files	Load each file separately

Using Multiple Converters

Define multiple converters for different scenarios:

@converter(task="mean_estimation", id="all_data")
def full_dataset(dataset_path: Path):
    ...

@converter(task="mean_estimation", id="normalized")
def normalized_dataset(dataset_path: Path):
    df = pd.read_csv(dataset_path / "data.csv")
    X = df.values
    X_normalized = (X - X.mean(axis=0)) / X.std(axis=0)
    ...

@converter(task="linear_regression", id="full")
def for_linear_regression(dataset_path: Path):
    # Same data files, different task
    ...

Run multiple converters:

dp-testing-platform evaluate \
  --task-name mean_estimation \
  --dataset-specs my_dataset:all_data my_dataset:normalized \
  --epsilons 1.0 10.0 \
  --deltas 1e-5

Adding Domain Bounds

For algorithms that require bounded domains:

from pathlib import Path
import numpy as np
from dp_testing_platform.core import LpBallDomain

@converter(task="mean_estimation", id="bounded")
def bounded_dataset(dataset_path: Path):
    data = np.load(dataset_path / "data.npz")
    X = data["features"]
    n, d = X.shape

    # Clip data to unit ball
    norms = np.linalg.norm(X, axis=1, keepdims=True)
    X_clipped = X / np.maximum(norms, 1.0)

    # Create domain for algorithms
    domain = LpBallDomain(p=2, radius=1.0, dim=d)

    metadata = MeanEstimationTask.Metadata(
        dim=d,
        n_samples=n,
        domain=domain,
    )
    input_data = MeanEstimationTask.Input(X=X_clipped)

    true_mean = np.mean(X_clipped, axis=0)
    metric = Metric(
        task=MeanEstimationTask,
        name="l2_error",
        fn=lambda output: float(np.linalg.norm(output.mean_est - true_mean)),
        args={},
    )

    return metadata, input_data, metric

Discovering Available Datasets

List converters for a task programmatically:

from dp_testing_platform.core.raw_data import get_real_instance_generators_for_task

generators = get_real_instance_generators_for_task("linear_regression")
for gen in generators:
    print(f"{gen.dataset_id}:{gen.converter_id}")

Or load a specific dataset/converter directly:

from dp_testing_platform.core.raw_data import RealInstanceGenerator

gen = RealInstanceGenerator(
    dataset_id="my_dataset",
    converter_id="full",
    task="linear_regression",
)
metadata, input_data, metric = gen.generate()

Key Classes

• @converter(task, id): Decorator to mark a function as a converter
• RawDataRegistry: Discovers converters from datasets/raw/*/converters.py
• RealInstanceGenerator: Loads raw data and runs converters
• get_real_instance_generators_for_task(task_id): Returns all generators for a task

Discovery Is Cached Per Session

Converter discovery runs each converters.py once per raw directory and memoizes the result, so repeated lookups don't re-execute converter modules. If you add a dataset, or fix or edit a converters.py, in a running session (a notebook, a REPL), the change is not picked up until you rescan:

from dp_testing_platform import refresh_raw_data_registries

refresh_raw_data_registries()  # drop the cache; the next lookup rescans disk

Restarting the interpreter has the same effect. If a converter you just authored doesn't appear (or an edited one still shows its old behavior), this cache is the reason.

Common Data Sources

Source	Description
OpenML	ML benchmark datasets
UCI ML Repository	Classic ML datasets
Kaggle	Competition datasets

Example: Downloading from OpenML

This download script uses scikit-learn (not a dependency of this library — install it separately to run the example):

from pathlib import Path
from sklearn.datasets import fetch_openml
import pandas as pd

# Fetch dataset
data = fetch_openml(data_id=361072, as_frame=True)
df = data.frame

# Save as CSV
dataset_path = Path("datasets/raw/openml_361072")
dataset_path.mkdir(parents=True, exist_ok=True)
df.to_csv(dataset_path / "data.csv", index=False)

Troubleshooting

"Dataset directory not found"

Make sure datasets/raw/<dataset_id>/ exists:

ls datasets/raw/my_dataset/

"Converter not found"

Ensure your converter file is named converters.py and the decorator is correct:

# Must be in datasets/raw/my_dataset/converters.py
@converter(task="task_name", id="converter_id")  # Check task and id match
def my_converter(dataset_path: Path):
    ...

"Metric is None"

Converters must return a Metric object, not None:

# Wrong
return metadata, input_data, None

# Correct
metric = Metric(task=MyTask, name="error", fn=lambda output: ..., args={})
return metadata, input_data, metric

Next Steps

• Creating Tasks - Define new problem types
• Running Evaluations - Benchmark on your datasets
• CLI Reference - Full command documentation