Core API Reference

The public core abstractions, experiment runner, and results store. These are re-exported at the package top level (from dp_testing_platform import ...); the canonical definitions live in dp_testing_platform.core.

Tasks and algorithms

dp_testing_platform.core.task_spec

Task

register_instance

register_instance(instance_cls: type[TInstanceGenerator]) -> type[TInstanceGenerator]

Register an instance generator class for this task (use as decorator).

register_algorithm

register_algorithm(algorithm_cls: type[TAlgorithm]) -> type[TAlgorithm]

Register an algorithm class for this task (use as decorator).

get_algorithms

get_algorithms() -> dict[str, type[BaseAlgorithm]]

Algorithms registered directly to this task.

get_instance_generators

get_instance_generators() -> list[type[InstanceGenerator]]

Instance generators registered directly to this task.

get_all_default_instances

get_all_default_instances() -> Iterator['InstanceGenerator']

Yield default instances for this task (all param grid combinations).

get_all_default_algorithms

get_all_default_algorithms() -> Iterator['BaseAlgorithm']

Yield algorithm instances for all default hyperparameter combinations.

dp_testing_platform.core.algorithm

The algorithm base class every DP algorithm inherits from.

:class:BaseAlgorithm fixes the contract a task's algorithms share: a single :meth:BaseAlgorithm.run over instances in the algorithm's own task format, plus the HYPERPARAM_INFO introspection surface from which the DP hyperparameter tuners derive search grids and default values. Cross-task reuse is not handled here — a parent-task algorithm is presented to a child task as a lifted class (see core.lifted), never adapted inside run.

BaseAlgorithm

Bases: ABC

Abstract base class for an algorithm native to a specific :class:Task.

Subclasses implement the single abstract method :meth:run, which consumes an instance in self.TASK format and returns that task's output. All hyperparameters are declared once in HYPERPARAM_INFO; the base derives the search grid and default values from it, so the DP hyperparameter tuners introspect an algorithm without any per-algorithm wiring.

Attributes:

Name	Type	Description
TASK	Task	The task this algorithm is native to (set when the class is registered on a task). run receives instances in this format.
HYPERPARAM_INFO	dict[str, dict[str, Any]]	Per-hyperparameter metadata keyed by parameter name. Recognized fields per entry: type: the parameter's Python type (float or int). default: the value used when the argument is omitted. description: human-readable summary. default_range: [min, max] endpoints the tuner searches between; :meth:default_search_grid discretizes this into a grid. default_grid: an explicit list of candidate values; takes precedence over default_range in :meth:default_search_grid. scale: "log" makes :meth:default_search_grid space the grid geometrically (for clip multipliers and other magnitude-spanning / learning-rate-like parameters); "linear" (the default when absent) spaces it evenly. tunable: False excludes the parameter from :meth:default_search_grid — e.g. parameters whose off-default values are privacy-unsound. Defaults to True when absent.

hyperparams property

hyperparams: dict[str, Any]

Get the current hyperparameter values for this algorithm instance.

__init__

__init__(name: str | None = None, description: str | None = None, metadata: dict[str, Any] | None = None)

Initialize the algorithm.

Parameters:

Name	Type	Description	Default
name	str | None	Display name for the algorithm.	None
description	str | None	Brief description of the algorithm.	None
metadata	dict[str, Any] | None	Arbitrary metadata dict (e.g., tuning config).	None

get_hyperparam_keys classmethod

get_hyperparam_keys() -> list[str]

Get the hyperparameter keys for this algorithm.

get_default_hyperparam_grid classmethod

get_default_hyperparam_grid() -> dict[str, list[Any]]

Get the default hyperparameter grid for this algorithm.

default_grid (an explicit candidate list, required for categorical parameters) takes precedence over default_range, mirroring default_search_grid.

default_search_grid classmethod

default_search_grid(n_points: int = 7) -> dict[str, list[Any]]

Build a discrete search grid for the DP private-selection tuners.

Reads HYPERPARAM_INFO only (no data or problem metadata), so the grid is data-independent and safe to hand to a DP selector as alg_hyperparam_grid. For each tunable parameter:

• "default_grid" (an explicit candidate list), when present, is used verbatim;
• otherwise a numeric "default_range" [min, max] is discretized into n_points values -- geometrically spaced when "scale" == "log", evenly spaced otherwise. Integer-typed parameters are rounded to ints and de-duplicated (so a coarse integer range yields fewer points);
• a "default_range" holding non-numeric (categorical) values cannot be interpolated and is enumerated verbatim.

Parameters with neither default_grid nor default_range, and any marked "tunable": False, are omitted -- the latter guards against a tuner selecting an unsound configuration (e.g. ObjectivePerturbation's output_budget_frac).

Parameters:

Name	Type	Description	Default
n_points	int	Number of grid points per continuous (range-based) parameter.	7

Returns:

Type	Description
dict[str, list[Any]]	Maps each included parameter name to its list of candidate values.

Raises:

Type	Description
ValueError	If a "scale": "log" parameter has a non-positive default_range endpoint (geometric spacing is undefined).

get_default_hyperparams classmethod

get_default_hyperparams() -> dict[str, Any]

Get the default hyperparameters for this algorithm.

run abstractmethod

run(metadata: TMetadata, input: TInput, budget: DPBudget, seed: int) -> TOutput

Run the algorithm on an instance of its own task.

Data is guaranteed to be in self.TASK format; a cross-task algorithm is exposed as a native lifted class of the target task (see core.lifted) rather than adapted here.

Parameters:

Name	Type	Description	Default
metadata	TMetadata	Problem specification in self.TASK format.	required
input	TInput	The private input data in self.TASK format.	required
budget	DPBudget	Privacy allowance for this run.	required
seed	int	Seed for reproducibility.	required

Returns:

Type	Description
TOutput	The algorithm's output in self.TASK format.

from_grid classmethod

from_grid(**param_grid: list[Any]) -> Iterator[BaseAlgorithm]

Yield algorithm objects for all combinations of parameters.

Parameters:

Name	Type	Description	Default
**param_grid	list[Any]	Maps parameter names to lists of values.	{}

Yields:

Type	Description
BaseAlgorithm	One algorithm per parameter combination, with auto-generated name.

serialize_for_json

serialize_for_json() -> dict[str, Any]

Serialize the algorithm for JSON output.

dp_testing_platform.core.lifted

Factory for lifted-algorithm classes.

A lifted algorithm presents a parent-task algorithm as a native algorithm of a child task. The factory forwards the inner algorithm's introspection surface (HYPERPARAM_INFO and the derived search grids) verbatim, so the DP hyperparameter tuners and the result registry treat a lifted class exactly like a hand-written algorithm of the child task: it instantiates with cls(**hyperparams), exposes flat per-parameter grids, and serializes to an importable module:ClassName path.

The child->parent adaptation is baked in at import time: the lift functions and the inner class are captured in a module-level class, hyperparameters bind at construction, and only data moves at run time (lift input -> inner.run -> push output, per call). Depth-2 chains compose by wrapping an already-lifted class (its run performs its own inner lift).

Privacy invariant (checked by the privacy review): lift_input MUST be per-row -- record i of the child input maps to record i of the parent input, with no cross-row statistics. Under that condition add/remove-one neighbors map to add/remove-one neighbors, the inner privacy guarantee transfers verbatim to the child data, and push_output is post-processing (privacy-free).

Authoring note (import discipline): a task's lifted.py imports the donor algorithm classes it wraps from their leaf modules (...m_estimation.algorithms.dp_gradient_descent), never through a package __init__ re-export. Leaf modules depend only on core and their own siblings, so wrapping across tasks stays free of package-initialization import cycles.

make_lifted_algorithm

make_lifted_algorithm(inner_cls: type[BaseAlgorithm], *, task: Task, lift_metadata: Callable[[ChildMetadata], ParentMetadata], lift_input: Callable[[ChildInput], ParentInput], push_output: Callable[[ParentOutput, ChildMetadata], ChildOutput], name: str) -> type[BaseAlgorithm]

Build a class running inner_cls as a native algorithm of task.

The returned class carries TASK = task and forwards inner_cls's HYPERPARAM_INFO (so grids match the inner exactly); it records the donor task on LIFTED_FROM and the donor class on INNER. Its __init__ accepts the inner algorithm's hyperparameters (plus name/description/metadata), builds an inner instance, and mirrors the resolved hyperparameters onto self so the hyperparams property and registry identity match. run lifts the child metadata/input to the parent format, runs the inner algorithm, and pushes the parent output back to the child format.

__module__/__qualname__/__name__ are stamped to the calling module and name so get_function_path yields an importable path -- the class must be bound to a module-level global of the same name for the path to re-import.

Parameters:

Name	Type	Description	Default
inner_cls	type[BaseAlgorithm]	The parent-task algorithm class to wrap (may itself be a lifted class, giving a composed depth-2 lift).	required
task	Task	The child task the lifted class runs as a native algorithm of.	required
lift_metadata	Callable[[ChildMetadata], ParentMetadata]	Map child metadata to the inner algorithm's task metadata.	required
lift_input	Callable[[ChildInput], ParentInput]	Map child input to the inner algorithm's task input; MUST be per-row (privacy invariant).	required
push_output	Callable[[ParentOutput, ChildMetadata], ChildOutput]	Map the inner algorithm's output back to the child output, given the child instance metadata (enables metadata-dependent projections, e.g. selecting a coefficient of interest).	required
name	str	The class name (e.g. "DPGradientDescentLR"); also the display name and the module global the class must be bound to.	required

Returns:

Type	Description
type[BaseAlgorithm]	A BaseAlgorithm subclass registered-ready for task.

dp_testing_platform.core.instance_generator

InstanceGenerator

Bases: ABC

params property

params: dict[str, Any]

Get the current parameter values for this generator instance.

generate abstractmethod

generate(seed: int) -> tuple[TMetadata, TInput, Metric]

Generate an instance for the task.

Parameters:

Name	Type	Description	Default
seed	int	Seed for random number generation (may be ignored by some generators).	required

Returns:

Type	Description
tuple[TMetadata, TInput, Metric]	A (metadata, input_data, metric) tuple for the task.

get_param_keys classmethod

get_param_keys() -> list[str]

Get the parameter keys for this instance generator class.

get_default_param_grid classmethod

get_default_param_grid() -> dict[str, list[Any]]

Get the default parameter grid for this instance generator class.

get_default_params classmethod

get_default_params() -> dict[str, Any]

Get the default parameters for this instance generator class.

serialize_for_json

serialize_for_json() -> dict[str, Any]

Serialize the generator for JSON output.

MixtureInstanceGenerator

Bases: InstanceGenerator

Instance generator that samples from a weighted mixture of generator classes.

Each generator spec provides a class (or importable path), a parameter grid, and an optional weight. At generation time, a generator class is sampled according to the weights, then parameters are sampled uniformly from the grid. This avoids combinatorial explosion over all generator/parameter combinations.

__init__

__init__(generator_specs: Sequence[dict[str, Any]])

Initialize from a list of generator specifications.

Parameters:

Name	Type	Description	Default
generator_specs	Sequence[dict[str, Any]]	Each dict must have 'generator_class' (or 'path') and optionally 'params' (grid dict) and 'weight' (float).	required

generate

generate(seed: int) -> tuple[TMetadata, TInput, Metric]

Generate an instance by sampling a generator class and parameters from the mixture.

Parameters:

Name	Type	Description	Default
seed	int	Seed for reproducible generator/parameter selection and data generation.	required

Returns:

Type	Description
tuple[TMetadata, TInput, Metric]	A (metadata, input_data, metric) tuple from the sampled generator.

dp_testing_platform.core.metrics

Metric dataclass

Concrete metric with callable and bound arguments.

__call__

__call__(output: TOutput, **kwargs: Any) -> float

Evaluate the metric on the given output.

Parameters:

Name	Type	Description	Default
output	TOutput	The algorithm output to evaluate.	required
**kwargs	Any	Additional arguments merged with bound args.	{}

Returns:

Type	Description
float	The computed metric value.

serialize_for_json

serialize_for_json() -> dict[str, Any]

Serialize the metric for JSON storage.

MetricSpec dataclass

Specification for a metric, optionally allowing custom functions.

A DP quality metric (registered in a task's dp_quality_metrics and used for private hyperparameter selection) is a cost: lower is better, because the DP selectors take the argmin. Expose a utility-like quantity as its cost complement (e.g. error rate, not accuracy).

create_metric

create_metric(fn: TMetricFn | None = None, **override_args: Any) -> Metric

Create a concrete Metric instance from this spec.

Parameters:

Name	Type	Description	Default
fn	TMetricFn | None	Custom metric function. Uses the spec's default if not provided.	None
**override_args	Any	Override default_args for this instance.	{}

Returns:

Type	Description
Metric	A bound Metric ready to evaluate outputs.

serialize_for_json

serialize_for_json() -> dict[str, Any]

Serialize the metric specification for JSON storage.

__call__

__call__(fn: TMetricFn | None = None, **override_args: Any) -> Metric

Shortcut for create_metric.

dp_testing_platform.core.domain

Domain

Bases: ABC

Abstract base class for domain constraints.

Domains specify constraints on data points and provide methods for computing diameters (used in sensitivity calculations) and projecting points onto the domain.

l2_diameter abstractmethod

l2_diameter() -> float

Compute the L2 diameter of the domain.

Returns:

Type	Description
float	The L2 diameter (max L2 distance between any two points).

l1_diameter abstractmethod

l1_diameter() -> float

Compute the L1 diameter of the domain.

Returns:

Type	Description
float	The L1 diameter (max L1 distance between any two points).

project abstractmethod

project(x: ndarray) -> np.ndarray

Project point(s) onto the domain.

Parameters:

Name	Type	Description	Default
x	ndarray	Point(s) to project. Can be a single point (1D) or multiple points (2D with points as rows).	required

Returns:

Type	Description
ndarray	The projected point(s) with the same shape as input.

l2_norm_upper_bound

l2_norm_upper_bound() -> float | None

Sound upper bound on the L2 norm of any point in the domain.

Returns an upper bound on max_{x in domain} ||x||_2, or None when no sound L2 bound can be derived for this domain type. The base implementation returns None (the safe default): callers that tighten a norm-clipping threshold with this bound keep their own computed clip when None is returned, which is the conservative choice (more noise, never less).

This is privacy-load-bearing: it must be a true UPPER bound. Clipping rows to min(computed_clip, l2_norm_upper_bound()) and calibrating the mechanism noise to that same value is privacy-safe precisely because the clip is what enforces the sensitivity; an over-estimate only forgoes a tightening, while an under-estimate is never produced here. Subclasses override only when they can prove the bound sound.

Returns:

Type	Description
float | None	A sound L2 upper bound, or None if none is available.

BoxDomain dataclass

Bases: Domain

Axis-aligned bounding box domain.

A box domain is defined by per-dimension lower and upper bounds. The bounds array has shape (d, 2) where d is the dimension, bounds[:, 0] are lower bounds, and bounds[:, 1] are upper bounds.

Attributes:

Name	Type	Description
bounds	Array2D	Array of shape (d, 2) with lower and upper bounds.

dim property

dim: int

Dimension of the domain.

l2_diameter

l2_diameter() -> float

L2 diameter: sqrt(sum of squared side lengths).

l1_diameter

l1_diameter() -> float

L1 diameter: sum of side lengths.

l2_norm_upper_bound

l2_norm_upper_bound() -> float

Max L2 norm over the box: sqrt(sum_i max(lo_i^2, hi_i^2)).

Each coordinate independently attains its largest magnitude at whichever endpoint is farther from 0, so max_i x_i^2 = max(lo_i^2, hi_i^2) and the worst-case squared L2 norm is their sum. This is a sound (in fact tight) upper bound on the L2 norm of any point in the box.

project

project(x: ndarray) -> np.ndarray

Project by clipping to bounds.

LpBallDomain dataclass

Bases: Domain

Lp ball domain: {x : ||x - center||_p <= radius}.

An Lp ball is defined by a norm type p, a radius, and optionally a center point. If center is None, the ball is centered at the origin.

Attributes:

Name	Type	Description
p	LpNorm	The Lp norm (1, 2, or np.inf).
radius	float	Radius of the ball.
dim	int	Dimension of the space.
center	Array1D | None	Center of the ball. None means origin.

l2_diameter

l2_diameter() -> float

L2 diameter depends on the ball's norm type.

l1_diameter

l1_diameter() -> float

L1 diameter depends on the ball's norm type.

l2_norm_upper_bound

l2_norm_upper_bound() -> float | None

Max L2 norm over the ball -- sound only for the L2 ball (p == 2).

For p == 2 every point satisfies ||x - center||_2 <= radius, so by the triangle inequality ||x||_2 <= radius + ||center||_2 (just radius when the ball is centred at the origin). A sound upper bound.

For p != 2 (L1 / Linf balls) we return None rather than convert a different-norm radius into an L2 bound here, so callers conservatively keep their own clip. (A sound L2 bound does exist -- e.g. an L1 ball gives ||x||_2 <= ||x||_1 <= radius -- but is intentionally out of scope for this audited helper.)

project

project(x: ndarray) -> np.ndarray

Project point(s) onto the ball.

tighten_clip_to_domain

tighten_clip_to_domain(computed_clip: float, domain: Domain | None) -> float

Tighten an L2 norm-clipping threshold using a domain bound when one exists.

Returns min(computed_clip, domain.l2_norm_upper_bound()) when domain supplies a sound L2 upper bound (see :meth:Domain.l2_norm_upper_bound), otherwise computed_clip unchanged. There is no point clipping rows looser than the domain already guarantees, and a tighter clip means a smaller calibrated sensitivity (less noise).

Privacy note: the returned value is the clip that MUST also be passed to the mechanism's noise calibration. Clipping rows to it enforces exactly that L2 sensitivity, so the tightening is always sound; because the domain bound is a sound UPPER bound, min never relaxes the clip an algorithm would otherwise have used. Behaviour-preserving (returns computed_clip) when domain is None or yields no bound -- the common case.

Parameters:

Name	Type	Description	Default
computed_clip	float	The clip threshold an algorithm computed from its hyperparameters (e.g. C_x_clip * sqrt(d) for the X-row L2 norm).	required
domain	Domain | None	The public metadata domain bounding the clipped quantity, or None.	required

Returns:

Type	Description
float	The clip threshold to both clip to and calibrate noise against.

dp_testing_platform.core.exceptions

Custom exceptions for the DP Testing Platform.

NoRelease

Bases: Exception

Raised when a DP algorithm outputs "no result" (the failure outcome ⊥).

Contract: raising this is a RELEASE, not a crash. The failure event is an outcome of the mechanism itself, and the algorithm's charged (eps, delta) budget must cover it. Consumers may therefore observe and post-process the failure freely — the engine records an honest "no_release" status row, and the DP selection tuners score a failed candidate +inf without extra budget.

Two compliant kinds of raiser:

• Data-independent infeasibility: the failure is a deterministic function of public quantities (metadata, hyperparameters, eps/delta), so the failure bit carries no information about the private data. Example: a smoothing parameter admitting no valid noise dilation at the requested eps.
• Accounted data-dependent failure: the failure probability depends on the private data, and the raiser's privacy analysis covers the joint release over outputs-plus-⊥. Example: a PTR (Propose-Test-Release) check failure. Such a raiser must state that argument at the raise site.

A failure that is NOT part of the accounted release — a solver that could not certify its tolerance, non-convergence, numerical breakdown — must use an ordinary exception (e.g. RuntimeError) instead: those are execution errors, recorded as "error" rows, and never fed back into DP selection.

Distinct from ValueError (invalid inputs/configuration).

Experiments and results

dp_testing_platform.core.experiment

Sweep orchestration: evaluate DP algorithms across a privacy-parameter grid.

:func:run_experiment is the engine entry point. It runs every (algorithm, instance, budget, seed) cell of a sweep, each in a supervised child process, so a cell that times out or breaches its memory cap is recorded as a failure while the rest of the sweep continues (failure isolation). All persistence lives under one workdir: the results store and both registries are written there, each cell is committed as it completes, and the default resume mode recomputes only the cells missing from a prior run.

load_existing_result_keys

load_existing_result_keys(results_dir: Path) -> set[KeyType]

Return canonical keys for every stored result in results_dir.

Reads the union of results.csv and any in-flight shard files, so cells persisted by an interrupted run are seen as done.

Raises:

Type	Description
RuntimeError	If a results file exists but cannot be parsed (corrupt file or unexpected schema). Failing loudly prevents silently re-running or overwriting results that deduplication could not see.

run_experiment

run_experiment(task: Task, algs: Sequence[BaseAlgorithm], instance_generators: Sequence[InstanceGenerator], budgets: Sequence[DPBudget] = _UNSET, *, alg_seeds: int | Sequence[int] = _UNSET, instance_generator_seeds: int | Sequence[int] = _UNSET, workdir: Path | str | None = _UNSET, mode: str = _UNSET, time_limit: float | None = _UNSET, memory_limit_gb: float | None = _UNSET, n_jobs: int = _UNSET, verbose: bool = _UNSET, config: ExperimentConfig | None = None) -> pd.DataFrame

Run an experiment across concrete budgets, algorithms, and instances.

All persistence lives under workdir: the engine creates, loads, and saves both registry files (algorithm_registry.json / instance_generator_registry.json) and the results store (results.csv + in-flight shards/) directly in that directory — user code never constructs or saves a registry. Reload a complete, labelled result set later with load_results(workdir).

Each completed cell is persisted immediately (commit per cell): an interrupted run loses at most the cells that were in flight, and the default mode="resume" means a re-run computes only the missing cells — finished work is never silently recomputed. A completed run consolidates everything into a single results.csv.

The sweep is a parameter grid over budget and instance_seed. Concrete typed budgets= are the only privacy-axis input. Settings can equivalently be supplied as a validated :class:ExperimentConfig via config= (mutually exclusive with the individual settings keywords).

Every cell runs in a supervised child process (unless both time_limit and memory_limit_gb are None): the instance is generated, the algorithm run, and the metric evaluated inside the child, so instance data never accumulates in the parent and a cell that exceeds its memory cap is killed and recorded as a "memory_limit_exceeded" failure while the sweep continues.

Parameters:

Name	Type	Description	Default
task	Task	The task to evaluate.	required
algs	Sequence[BaseAlgorithm]	Algorithm instances to evaluate on every cell. Per-instance tuning is expressed by including a DPHyperparameterTuned algorithm, which tunes internally per run.	required
instance_generators	Sequence[InstanceGenerator]	Instance generators to produce test data.	required
budgets	Sequence[DPBudget]	Concrete privacy allowances (grid axis "budget").	_UNSET
alg_seeds	int | Sequence[int]	Algorithm seeds. An int N means the deterministic seeds range(N); under mode="resume" any of those cells already stored are skipped, not recomputed. A sequence is explicit.	_UNSET
instance_generator_seeds	int | Sequence[int]	Seeds for instance generation (grid axis "instance_seed").	_UNSET
workdir	Path | str | None	Directory owning all persistence for the sweep. None disables saving.	_UNSET
mode	str	"resume" (default) computes only cells missing from the stored results; "overwrite" recomputes and replaces every requested cell.	_UNSET
time_limit	float | None	Max seconds per cell. None disables the limit.	_UNSET
memory_limit_gb	float | None	Hard cap (in GB) on the resident memory of one cell's process tree, enforced by a watchdog that kills the cell on breach. None disables the cap — only do this in an environment where a runaway allocation is acceptable.	_UNSET
n_jobs	int	Number of parallel worker processes (joblib/loky) over cells. 1 runs serially in-process; -1 uses all cores.	_UNSET
verbose	bool	Render a per-completed-cell progress bar (tqdm; notebook or terminal automatically) and detailed logging.	_UNSET
config	ExperimentConfig | None	A pre-built :class:ExperimentConfig carrying all of the above settings. When given, no individual settings keyword may be passed.	None

Returns:

Type	Description
DataFrame	Results for every cell of the requested grid — the union of
DataFrame	previously stored rows and newly computed ones (so mode="resume"
DataFrame	re-runs return the full result set, not just the new rows). When
DataFrame	workdir is None, only the newly computed rows are returned.

dp_testing_platform.core.experiment_config

Validated configuration for experiment sweeps over concrete DP budgets.

ExperimentConfig

Bases: BaseModel

Hold the validated settings for one experiment sweep.

budgets is the single privacy axis. Identical budgets are deduplicated in caller order. JSON budget objects are converted to typed budgets at this validation boundary; the engine never sees epsilon/delta configuration sugar or a generic parameter grid.

sweep_grid

sweep_grid() -> dict[str, list[Any]]

Compile the concrete budget and instance-seed axes.

grid_points

grid_points() -> list[dict[str, Any]]

Enumerate scalar budget and instance-seed coordinates.

resolved_alg_seeds

resolved_alg_seeds() -> int | list[int]

Return explicit algorithm seeds, a seed count, or the default seed.

dp_testing_platform.core.result_manager

Durable, typed CSV storage for experiment results.

Results live in a results directory as one human-openable results.csv plus, while a run is in flight, per-process shard files under shards/. Each completed experiment cell is appended to a shard immediately (commit per cell), so an interrupted run loses at most the in-flight cells. Loaders always read the union of results.csv and any shards; a completing run consolidates everything back into a single results.csv.

All reads apply an explicit dtype schema. Result identity is carried by the canonical typed-budget string; analysis floats retain round-trip formatting but never participate in identity.

OutputWriter

Durably persist each cell's full released output to the sidecar.

Stores the task's Output as a field dict ({field_name: value}): None fields are omitted (the dataclass default refills them on reconstruct) and scalar fields are coerced to 0-d arrays, so the store stays task-agnostic (raw values, no Output type imported). Each field becomes one array in the cell's NPZ shard under output_shards/ (array name f"{key}::{field}"), the file named by a hash of the cell key (so re-running a cell overwrites its own file). Every write is flushed and fsynced, so a committed output survives a SIGKILL — mirroring :class:ShardWriter's durability. Instances are cheap and picklable (they hold only the results directory), so one writer ships to all parallel workers.

write_output

write_output(key: str, fields: Mapping[str, Any]) -> None

Append one cell's output field dict to its own sidecar shard.

Parameters:

Name	Type	Description	Default
key	str	The cell's content-addressed output key (see :func:output_key).	required
fields	Mapping[str, Any]	The output dataclass as {field_name: value}. None values are skipped (the field's default refills them on reconstruct); scalars are stored as 0-d arrays, arrays as-is.	required

ShardWriter

Durably append result rows to a per-process shard file.

Each writing process gets its own shard (named by run id + pid), so no file ever sees concurrent appends — safe under joblib worker processes. Every row is flushed and fsynced on write: once :meth:write_row returns, the cell survives a SIGKILL of the whole process tree.

Instances are cheap, stateless, and picklable (they hold only the results directory and run id), so one writer can be shipped to all parallel workers.

write_row

write_row(row: Mapping[str, Any]) -> None

Append one result row to this process's shard, creating it if needed.

Rows that do not already carry the library_version / library_commit provenance stamps get them filled in from the writing process's runtime environment; rows without a runtime_s measurement get NaN; rows without a privacy_sound verdict default to True (no diagnostic modification recorded); rows without a peak_rss_mb measurement get NaN.

Any keys beyond the fixed columns are task-derived key-metric columns: they are written (as %.17g floats) after the fixed columns. All rows from one process share a task, so they carry the same key-metric columns — the header (written once per shard) stays consistent.

results_frame

results_frame(rows: list[dict[str, Any]] | None = None) -> pd.DataFrame

Build a results DataFrame with the fixed columns, dtypes, and any key metrics.

Fixed columns always appear in their canonical order; any extra keys carried by the rows are task-derived key-metric columns, appended after the fixed columns (in first-seen order) and typed float64.

has_shards

has_shards(results_dir: Path) -> bool

Return whether any in-flight/interrupted-run shard files exist.

load_result_rows

load_result_rows(results_dir: Path) -> pd.DataFrame

Load the typed union of results.csv and any in-flight shards.

Shard rows (from interrupted or in-flight runs) are appended after the consolidated rows, so on duplicate keys the shard row is the later one. This is the raw row loader (no name labelling) used by the engine; user code usually wants :func:load_results.

Returns:

Type	Description
DataFrame	A DataFrame with the canonical result columns and dtypes; empty
DataFrame	(but correctly typed) when no results exist.

load_results

load_results(workdir: Path | str) -> pd.DataFrame

Load labelled results from an experiment workdir, fresh-process safe.

Reads the typed union of results.csv and any in-flight shards under workdir and joins human-readable names from the persisted registry files: an alg_name column (from algorithm_registry.json) and an instance_name column (from instance_generator_registry.json, dataset_id:converter_id for real datasets). Requires no in-memory state — a fresh kernel can reload a complete, labelled result set from disk alone. UUIDs without a registry entry keep the UUID as the name.

Parameters:

Name	Type	Description	Default
workdir	Path | str	The directory passed as run_experiment(workdir=...).	required

Returns:

Type	Description
DataFrame	The results DataFrame with the canonical columns plus alg_name
DataFrame	and instance_name.

Raises:

Type	Description
FileNotFoundError	If the workdir holds no results at all. Run run_experiment(..., workdir=...) pointing at this directory first.

unregistered_result_uuids

unregistered_result_uuids(workdir: Path) -> dict[str, set[str]]

Return result UUIDs absent from the persisted registry files.

The registries must cover every UUID appearing in the results: :func:load_results labels rows by joining on them, so a missing entry silently degrades a row to its raw content-addressed UUID. Checked at consolidation time so a multi-call sweep that failed to union its registries is caught immediately rather than surfacing later as unresolvable names.

Returns:

Type	Description
dict[str, set[str]]	A dict with keys "alg" and "instance", each mapping to the set
dict[str, set[str]]	of result UUIDs (from alg_uuid / inst_gen_uuid) that have no
dict[str, set[str]]	entry in the corresponding registry file. Empty sets mean the
dict[str, set[str]]	invariant (registry ⊇ every result UUID) holds.

consolidate_results

consolidate_results(results_dir: Path) -> pd.DataFrame

Merge results.csv and all shards into one results.csv.

Deduplicates on the canonical budget-bearing key keeping the last occurrence (shard rows win over previously consolidated rows, so re-running with mode="overwrite" replaces old rows). The merged file is written atomically (temp file + os.replace) and the shards/ directory is removed — shards exist on disk only for in-flight/interrupted runs.

Returns:

Type	Description
DataFrame	The consolidated, typed DataFrame.

result_key

result_key(inst_gen_uuid: str, alg_uuid: str, budget: BudgetString, alg_seed: int, instance_seed: int) -> ResultKey

Build the one canonical identity tuple for a result cell.

output_key

output_key(inst_gen_uuid: str, alg_uuid: str, budget: BudgetString, alg_seed: int, instance_seed: int) -> str

Build the output-sidecar key string for one result cell.

Uses the same canonical string identity as :func:result_key; callers convert typed budgets once at their boundary.

load_outputs

load_outputs(results_dir: Path | str) -> dict[str, dict[str, Any]]

Load the output sidecar as key -> {field name -> value}, fresh-process safe.

Reads the union of outputs.npz and any in-flight output_shards/ NPZ files (the latter present only during or after an interrupted run); shard entries win on duplicate keys. Each cell's value is the stored field dict of its task's Output (raw field values — task-agnostic, no Output type imported); None fields were omitted on write and scalar fields are 0-d arrays. Returns an empty dict when no sidecar exists (for example, when every output exceeded the size threshold), so callers degrade gracefully rather than crashing.

has_output_shards

has_output_shards(results_dir: Path) -> bool

Return whether any in-flight output-sidecar shard files exist.

consolidate_outputs

consolidate_outputs(results_dir: Path) -> dict[str, dict[str, Any]]

Merge outputs.npz and all sidecar shards into one outputs.npz.

Unions the previously consolidated sidecar with every in-flight shard (shard entries win on duplicate composite keys, so mode="overwrite" replaces old outputs), writes the result atomically, and removes output_shards/. A no-op (no file written) when nothing was stored.

Returns:

Type	Description
dict[str, dict[str, Any]]	The consolidated key -> {field name -> value} dict (empty if no
dict[str, dict[str, Any]]	outputs exist).

Registries

The engine owns these — it creates, loads, and saves both registries inside the experiment workdir, so user code never constructs one. They map algorithms and instance generators to the UUIDs that key the results store.

dp_testing_platform.core.algorithm_registry

AlgorithmRegistry

Registry for algorithm objects, with UUID management.

Two in-memory stores back the registry. _registry (uuid -> live algorithm object) holds the objects that could be reconstructed, used for identity matching and get_algorithm. _records (uuid -> serialized JSON record) is the source of truth for what is written: it accumulates the union of every entry ever loaded or registered, so a multi-call sweep grows the on-disk registry and never overwrites entries from earlier calls (even ones whose live object could not be rebuilt). On disk the file is a JSON list of those records.

save

save() -> None

Save the algorithm registry to disk as the union of all known entries.

Writes _records (every entry loaded from disk plus everything registered this session), refreshing the record of any entry whose live object is present (so metadata merges from get_uuid are captured). Entries from earlier run_experiment calls are therefore never dropped.

list_algorithms

list_algorithms()

Return a list of (uuid, alg) tuples.

get_uuid

get_uuid(algorithm: BaseAlgorithm) -> str

Get or register a UUID for the given algorithm.

Uses class path and hyperparams for identity. If a duplicate is found, merges metadata (new keys/values overwrite old).

get_algorithm

get_algorithm(uuid: str) -> BaseAlgorithm

Retrieve an algorithm object by its UUID.

Raises:

Type	Description
ValueError	If no algorithm with the given UUID is found.

__getitem__

__getitem__(uuid: str) -> BaseAlgorithm

Get an algorithm by UUID using dictionary-like access.

__contains__

__contains__(uuid: str) -> bool

Check if a UUID exists in the registry.

find_by_name

find_by_name(name: str) -> tuple[str, BaseAlgorithm] | None

Find the first algorithm matching the given name.

find_by_class

find_by_class(class_name: str) -> list[tuple[str, BaseAlgorithm]]

Find all algorithms matching the given class name.

dp_testing_platform.core.instance_generator_registry

InstanceGeneratorRegistry

Registry for instance generators and their parameters, with UUID management.

Two in-memory stores back the registry. _registry (uuid -> live generator object) is best-effort: it holds the objects that could be reconstructed, used for identity matching and get_generator. _records (uuid -> serialized JSON record) is the source of truth for what is written: it accumulates the union of every entry ever loaded or registered, so a multi-call sweep grows the on-disk registry and never overwrites entries from earlier calls. On disk the file is a JSON list of those records.

save

save() -> None

Save the instance registry to disk as the union of all known entries.

Writes _records (every entry loaded from disk plus everything registered this session), refreshing the record of any entry whose live object is present. Entries from earlier run_experiment calls are therefore never dropped.

list_generators

list_generators()

Return a list of (uuid, generator) tuples for reconstructed generators.

get_uuid

get_uuid(generator: InstanceGenerator) -> str

Get or register a UUID for the given generator (class + params identity, no seed).

get_generator

get_generator(uuid: str) -> InstanceGenerator

Retrieve a generator object by its UUID.

Raises:

Type	Description
ValueError	If no generator with the given UUID exists.

__getitem__

__getitem__(uuid: str) -> InstanceGenerator

Get a generator by UUID using dictionary-like access.

__contains__

__contains__(uuid: str) -> bool

Check if a UUID exists in the registry.

find_by_class

find_by_class(class_name: str) -> list[tuple[str, InstanceGenerator]]

Find all generators matching the given class name.

Real-dataset discovery

dp_testing_platform.core.real_dataset_utils

get_real_datasets_for_task

get_real_datasets_for_task(task: Union[Task, str], raw_directory: Path | str = Path('datasets/raw'), modifications: Sequence[str] | None = None) -> dict[str, RealInstanceGenerator]

Get all available real datasets for a task.

Parameters:

Name	Type	Description	Default
task	Union[Task, str]	Task object or its id	required
raw_directory	Path | str	Directory containing raw datasets with converters. A str is accepted and coerced to Path at this boundary.	Path('datasets/raw')
modifications	Sequence[str] | None	Optional ordered names of task-declared dataset modifications applied to every dataset (looked up in the task's Task.dataset_modifications registry). None/empty leaves the converted instances untouched.	None

Returns:

Type	Description
dict[str, RealInstanceGenerator]	Dict mapping the instance name to its RealInstanceGenerator. The key is
dict[str, RealInstanceGenerator]	"{dataset_id}:{converter_id}" plus a self-marking suffix for any
dict[str, RealInstanceGenerator]	applied modification (e.g. "…:full+<markerA>+<markerB>"), matching
dict[str, RealInstanceGenerator]	the instance_name the result store records.

get_real_dataset_info_for_task

get_real_dataset_info_for_task(task: Union[Task, str], raw_directory: Path | str = Path('datasets/raw'), include_instance_metadata: bool = False) -> dict[str, dict[str, Any]]

Get metadata about all available real datasets for a task.

Parameters:

Name	Type	Description	Default
task	Union[Task, str]	Task object or its id	required
raw_directory	Path | str	Directory containing raw datasets. A str is accepted and coerced to Path at this boundary.	Path('datasets/raw')
include_instance_metadata	bool	If True, run converters to include instance metadata (of type task.Metadata) with key 'instance_metadata'	False

Returns:

Type	Description
dict[str, dict[str, Any]]	Dict mapping "{dataset_id}:{converter_id}" to metadata dict

load_dataset_info

load_dataset_info(raw_directory: Path, dataset_id: str) -> dict[str, Any] | None

Load the info.json for a specific raw dataset.

validate_dataset_exists

validate_dataset_exists(raw_directory: Path, dataset_id: str) -> bool

Check if a raw dataset exists and has required files.

list_raw_datasets

list_raw_datasets(raw_directory: Path = Path('datasets/raw')) -> list[str]

List all raw dataset IDs that have converters.

list_real_datasets

list_real_datasets(raw_directory: Path | str = Path('datasets/raw')) -> list[dict[str, Any]]

Summarize every raw dataset for human-readable discovery.

A lightweight discovery view: reads each dataset's info.json and the converter registry, never the data arrays. Useful for browsing which datasets exist, what they are called, and which tasks they support.

Parameters:

Name	Type	Description	Default
raw_directory	Path | str	Directory containing raw datasets with converters. A str is accepted and coerced to Path at this boundary.	Path('datasets/raw')

Returns:

Type	Description
list[dict[str, Any]]	One dict per dataset (sorted by id) with keys id, name,
list[dict[str, Any]]	n_samples, n_features, and tasks (sorted task ids the
list[dict[str, Any]]	dataset has converters for). name falls back to the id, and
list[dict[str, Any]]	n_samples/n_features to None, when info.json lacks them.