Analysis API Reference

Post-run inspection of a completed experiment workdir. These utilities read only the stored results, the output sidecar, and the registries — no algorithm is re-run. They are re-exported at the package top level (from dp_testing_platform import ...); the canonical definitions live in dp_testing_platform.analysis.

Performance profiles

Dolan–Moré-style performance ratios and their CDFs, for comparing algorithms across a set of instances.

dp_testing_platform.analysis.performance_ratios

Compute and plot Dolan–Moré-style performance profiles.

compute_performance_ratios

compute_performance_ratios(df: DataFrame, agg_func: str = 'median', granularity: str = 'run') -> pd.DataFrame

Compute performance ratios relative to the best algorithm per instance.

The denominator is always the best-algorithm aggregate per instance and privacy budget; granularity selects the numerator:

• "run" (default): each per-seed alg_performance divided by the best-algorithm median, so ratios below 1 are admitted — a run-level reliability profile that captures randomized-algorithm variance.
• "aggregate": the per-(algorithm, instance) median divided by the best-algorithm median (the conventional Dolan-Moré profile, ratios ≥ 1).

Cost contract: alg_performance must be a NONNEGATIVE cost — smaller is better, so the best algorithm is the per-instance minimum and ratios are ≥ 0. A larger-is-better score that can go negative (e.g. r2) silently inverts the best = min selection and makes ratios of mixed-sign values meaningless, so negative values are rejected. Profile on a cost (mse or l2_error_to_ols), never on r2 itself.

Parameters:

Name	Type	Description	Default
df	DataFrame	Current-shape results with algorithm, instance, canonical budget, sparse budget-parameter, and performance columns. alg_name is carried through when present.	required
agg_func	str	Aggregation function for per-algorithm-per-instance scores.	'median'
granularity	str	"run" or "aggregate" (see above).	'run'

Returns:

Type	Description
DataFrame	DataFrame with an added performance_ratio column.

Raises:

Type	Description
ValueError	If granularity is invalid, or alg_performance carries negative values (violating the nonnegative-cost contract).

compute_performance_cdfs

compute_performance_cdfs(df: DataFrame, min_tau: float = 1.0, max_tau: float = 10.0, num_points: int = 200) -> pd.DataFrame

Compute per-algorithm CDFs of performance ratios as a tidy frame.

Consumes the performance_ratio column produced by compute_performance_ratios (of either granularity) and returns the curve data in long form, so it renders directly with sns.lineplot / px.line and is inspectable as a table. Each algorithm's CDF weights every instance equally (the per-instance step CDFs are averaged).

Parameters:

Name	Type	Description	Default
df	DataFrame	DataFrame with a performance_ratio column (from compute_performance_ratios).	required
min_tau	float	Lower bound for the CDF x-axis (tau grid).	1.0
max_tau	float	Upper bound for the CDF x-axis (tau grid).	10.0
num_points	int	Number of evenly spaced tau values.	200

Returns:

Type	Description
DataFrame	A tidy DataFrame with columns tau (the evaluation grid), algorithm
DataFrame	(the human-readable alg_name when present — as produced by
DataFrame	load_results — falling back to the raw alg_uuid, with distinct
DataFrame	algorithms sharing a display name disambiguated by a uuid-prefix suffix so
DataFrame	no curve is silently merged), and cdf (the fraction of instances on
DataFrame	which the algorithm is within tau of the best). One row per (algorithm,
DataFrame	tau); num_points rows per algorithm. Empty input yields the empty
DataFrame	frame with these columns.

Raises:

Type	Description
ValueError	If the tau bounds or number of points are invalid.

plot_performance_profile

plot_performance_profile(results: DataFrame | Path | str, *, value: str = 'alg_performance', agg_func: str = 'median', granularity: str = 'run', min_tau: float = 1.0, max_tau: float = 10.0, num_points: int = 200, ax: Any = None) -> Any

Plot a name-labelled performance profile from results in one call.

The best finite aggregate on each instance is the reference. At run granularity, non-finite outcomes remain profile misses; at aggregate granularity, finite outcomes are aggregated and an all-failed pair stays missing. Use :func:summarize_results alongside the profile for explicit failure counts.

Requires matplotlib and seaborn, imported only when called.

Parameters:

Name	Type	Description	Default
results	DataFrame | Path | str	A completed experiment workdir or results DataFrame.	required
value	str	Nonnegative cost column to profile; smaller must be better.	'alg_performance'
agg_func	str	Aggregation for each algorithm-instance pair.	'median'
granularity	str	"aggregate" for a conventional profile or "run" for a randomized-run reliability profile.	'run'
min_tau	float	Lower bound for the performance-ratio axis.	1.0
max_tau	float	Upper bound for the performance-ratio axis.	10.0
num_points	int	Number of evaluation points between the tau bounds.	200
ax	Any	Axes to draw into; creates a new figure when omitted.	None

Returns:

Type	Description
Any	The matplotlib Axes drawn on (typed Any: matplotlib is an optional
Any	extra, so its types are never named in the surface).

Raises:

Type	Description
ImportError	If matplotlib or seaborn is not installed.
ValueError	If the selected value is unavailable, violates the cost contract, or has no finite outcomes.

compare_runs

compare_runs(runs: Sequence[DataFrame | Path | str], *, labels: Sequence[str] | None = None, value: str = 'alg_performance', agg_func: str = 'median', granularity: str = 'run', min_tau: float = 1.0, max_tau: float = 10.0, num_points: int = 200, ax: Any = None) -> Any

Overlay performance profiles from multiple runs.

Algorithms share a colour across runs (hue) while each run gets a distinct line style, so the legend distinguishes both axes of the comparison. Run labels default to a workdir basename, a uniform library version/commit stamp, or a numbered fallback; pass labels for report-ready names.

Requires matplotlib and seaborn, imported only when called.

Parameters:

Name	Type	Description	Default
runs	Sequence[DataFrame | Path | str]	Two or more completed workdirs or results DataFrames.	required
labels	Sequence[str] | None	Optional label corresponding to each run.	None
value	str	Nonnegative cost column to profile; smaller must be better.	'alg_performance'
agg_func	str	Aggregation for each algorithm-instance pair.	'median'
granularity	str	"aggregate" for conventional profiles or "run" for randomized-run reliability profiles.	'run'
min_tau	float	Lower bound for the shared performance-ratio axis.	1.0
max_tau	float	Upper bound for the shared performance-ratio axis.	10.0
num_points	int	Number of evaluation points between the tau bounds.	200
ax	Any	Axes to draw into; creates a new figure when omitted.	None

Returns:

Type	Description
Any	The matplotlib Axes drawn on (typed Any: matplotlib is an optional
Any	extra, so its types are never named in the surface).

Raises:

Type	Description
ImportError	If matplotlib or seaborn is not installed.
ValueError	If fewer than two runs are supplied, labels do not align, or a run cannot produce a profile.

Summaries and privacy–utility plots

Finite-aware health summaries, per-instance tables, and family-aware budget curves.

dp_testing_platform.analysis.summaries

Human-readable summaries and quick-look plotting for experiment results.

Aggregation here is finite-aware: failure rows carry an inf sentinel in alg_performance and errored metrics are NaN, so naive means over a results frame are silently poisoned. Every aggregate in this module is computed over finite values only, while failures stay visible through explicit count columns (n_failed foremost) rather than disappearing into the mask.

All entry points accept either an experiment workdir (Path/str, loaded via load_results) or an already-loaded results DataFrame.

summarize_results

summarize_results(results: DataFrame | Path | str) -> pd.DataFrame

Summarize outcomes per algorithm, instance, and concrete budget.

Groups the results by algorithm and instance identity (uuid — with the human-readable alg_name/instance_name carried alongside when present) and privacy budget, and reports per group:

• n_runs: total rows (seed combinations) in the group.
• n_success: rows with status "success".
• n_failed: every non-success row (principled failures, errors, timeouts, memory kills, non-finite outputs) — failures are loud, never silently dropped by the finite-aware aggregates below.
• n_timeout: the timeout subset of n_failed.
• <col>_mean / <col>_median for alg_performance and every key-metric column: computed over finite values only, so the inf failure sentinel and NaN metric values never poison the aggregates (a group with no finite values yields NaN).

Parameters:

Name	Type	Description	Default
results	DataFrame | Path | str	An experiment workdir (loaded via load_results) or an already-loaded results DataFrame.	required

Returns:

Type	Description
DataFrame	A tidy DataFrame with one row per group, sorted by the group keys.

per_instance_table

per_instance_table(results: DataFrame | Path | str, *, value: str = 'alg_performance', agg: str = 'median', budget: DPBudget | None = None) -> pd.DataFrame

Pivot one value column into an instance-by-algorithm comparison table.

Rows are instances, columns are algorithms (human-readable names when available, disambiguated per uuid), and each cell is the finite-aware agg of value over that pair's rows — the inf failure sentinel and NaN never contribute (an all-failed cell is NaN).

This table intentionally masks non-finite outcomes. Inspect :func:summarize_results alongside it for explicit failure counts.

When no budget is selected, the canonical budget string and its convenience columns stay in the row index, so different families are never mixed.

Parameters:

Name	Type	Description	Default
results	DataFrame | Path | str	An experiment workdir (loaded via load_results) or an already-loaded results DataFrame.	required
value	str	The column to tabulate (alg_performance or any key-metric column).	'alg_performance'
agg	str	Aggregation over each (instance, algorithm) group, e.g. "median" or "mean".	'median'
budget	DPBudget | None	Concrete privacy allowance to select. Its canonical string is matched against the stored budget identity.	None

Returns:

Type	Description
DataFrame	The pivoted DataFrame (instances x algorithms).

Raises:

Type	Description
ValueError	If a budget filter matches no rows or value is not a result value column.

compute_error_vs_budget

compute_error_vs_budget(results: DataFrame | Path | str, *, value: str = 'alg_performance', agg: str = 'median') -> pd.DataFrame

Aggregate the finite-aware agg of value by concrete budget.

Groups by algorithm and canonical budget (over instances and seeds), excluding the inf failure sentinel and NaN so a partially-failing algorithm reports its finite performance rather than a poisoned aggregate. All-failed cells are dropped rather than emitted as NaN. The family and sparse convenience columns travel with each canonical budget; privacy parameters from different families are never placed on one numeric axis.

Parameters:

Name	Type	Description	Default
results	DataFrame | Path | str	An experiment workdir (loaded via load_results) or an already-loaded results DataFrame.	required
value	str	The column to aggregate (alg_performance or any key-metric column).	'alg_performance'
agg	str	Aggregation over each (algorithm, budget) group, e.g. "median" or "mean".	'median'

Returns:

Type	Description
DataFrame	A tidy DataFrame with columns algorithm (human-readable names when
DataFrame	available, disambiguated per uuid), the current budget columns, and the
DataFrame	aggregated value. One row per (algorithm, budget) with a finite result.

Raises:

Type	Description
ValueError	If value is not a result value column.

plot_error_vs_budget

plot_error_vs_budget(results: DataFrame | Path | str, *, value: str = 'alg_performance', agg: str = 'median', ax: Any = None) -> Any

Plot the finite-aware agg of value by concrete budget.

A thin renderer over :func:compute_error_vs_budget. The canonical budget strings form a categorical axis because epsilon, delta, and rho are not one commensurate numeric scale.

Requires matplotlib and seaborn (optional dependencies; imported on call).

Parameters:

Name	Type	Description	Default
results	DataFrame | Path | str	An experiment workdir (loaded via load_results) or an already-loaded results DataFrame.	required
value	str	The column to plot (alg_performance or any key-metric column).	'alg_performance'
agg	str	Aggregation over each (algorithm, budget) group, e.g. "median" or "mean".	'median'
ax	Any	Axes to draw into; a new figure is created when omitted.	None

Returns:

Type	Description
Any	The matplotlib Axes drawn on (typed Any: matplotlib is an optional
Any	extra, so its types are never named in the surface).

Raises:

Type	Description
ImportError	If matplotlib or seaborn is not installed.
ValueError	If value is not a result value column.

Run-archive export

Turn a completed workdir into a portable, hash-verified archive for sharing, integrity-checking, and provenance.

dp_testing_platform.analysis.export

Export a completed experiment workdir as a portable, hash-verified archive.

Takes one completed experiment workdir (a consolidated results.csv plus both registry JSON files) and produces, under a destination directory, a self-describing archive suitable for sharing, integrity-checking, and provenance:

• results/<Algorithm>.csv — the consolidated rows partitioned by algorithm. Partitioning on the algorithm axis keeps each file a readable, replaceable slice (re-run one algorithm, swap one file; small diffs; spreadsheet-sized). Each file is the labelled :func:load_results frame for that algorithm, so it carries human-readable alg_name/instance_name columns.
• both registry JSON files, copied verbatim.
• MANIFEST.json — the archive's descriptor: a content content_hash over the consolidated results + registries, the concrete typed-budget grid and per-cell seed counts, the datasets actually present in the rows, an optional caller-supplied metadata block, the execution environment (hostname, Python version, pip freeze snapshot), and a SHA-256 per result file (integrity check).

The output is deterministic given the inputs except for the execution-environment block, which is provenance and expected to vary by machine.

export_run_archive

export_run_archive(workdir: Path | str, *, dest: Path | str, metadata: Mapping[str, Any] | None = None) -> Path

Export a completed experiment workdir as a portable, hash-verified archive.

Reads the consolidated results.csv and both registry files from workdir and writes, under dest: per-algorithm result CSVs in results/, copies of both registries, and a MANIFEST.json describing the archive (see module docstring for the field list). Task-agnostic — works for any task's workdir.

Parameters:

Name	Type	Description	Default
workdir	Path | str	A completed experiment directory (consolidated results.csv, algorithm_registry.json, instance_generator_registry.json). Must hold no in-flight shards — export a completed run only.	required
dest	Path | str	Destination directory for the archive. Created if absent; its results/ must not already contain foreign CSV files.	required
metadata	Mapping[str, Any] | None	Optional caller-supplied mapping recorded verbatim under metadata in the manifest. The library does not interpret it — use it to attach any caller-side context to the archive.	None

Returns:

Type	Description
Path	The path to the written MANIFEST.json.

Raises:

Type	Description
FileNotFoundError	If workdir holds no consolidated results.csv.
ValueError	If workdir holds in-flight shards (an interrupted or still-running sweep — consolidate it first).
FileExistsError	If dest/results already holds foreign result CSVs.

Post-hoc metric recomputation

Re-derive a metric from each cell's stored released output, without re-running any algorithm.

dp_testing_platform.analysis.post_hoc

Post-hoc metric recomputation from the stored output sidecar.

A sweep stores each cell's released output (its full Output dataclass as a field dict) in an outputs.npz sidecar keyed by the same content-addressed row key as the result row. This module re-derives new metrics from those stored outputs — reconstructing each task.Output and regenerating its instance, then applying a metric — so a metric that wasn't computed during the sweep can be added without re-running any algorithm.

Why regenerate the instance rather than store it? A metric is a function of the released output and the instance's data; the data itself is never stored (only the small output is), and regeneration is deterministic from the instance seed, so the same (output, instance data) pair is reproduced exactly and cheaply.

recompute_metric

recompute_metric(workdir: Path | str, metric_spec: MetricSpec, raw_directory: Path | str, *, column_name: str | None = None) -> pd.DataFrame

Attach a metric recomputed from the stored output sidecar to the results.

Loads the labelled results and the output sidecar from workdir, regenerates each instance's data via the persisted instance-generator registry (reconstructed against raw_directory), and evaluates metric_spec on the reconstructed task.Output for every row whose output was stored. No algorithm is re-run. Rows without a stored output — those whose output exceeded the sidecar size threshold, failure rows, or every row of a workdir with no sidecar — receive NaN, as do rows whose stored output fails to reconstruct or whose metric evaluation raises (each with a warning); one bad row never aborts the whole recompute.

The Output is reconstructed from the stored field dict as task.Output(**fields) (any task — no field name hardcoded); a field stored as a 0-d array is unwrapped to a Python scalar (.item()) while vector fields stay arrays, so scalar-valued outputs round-trip faithfully.

Parameters:

Name	Type	Description	Default
workdir	Path | str	The directory passed as run_experiment(workdir=...).	required
metric_spec	MetricSpec	The metric to compute. Its fn is called as fn(output, evaluation_input=input) and must read only the output and the instance input (the key-metric contract).	required
raw_directory	Path | str	Path to the raw datasets directory on this machine, used to reconstruct real-dataset instance generators.	required
column_name	str | None	Name for the attached column. Defaults to metric_spec.name.	None

Returns:

Type	Description
DataFrame	A copy of load_results(workdir) with the recomputed metric column
DataFrame	added.