Analyzing results

A sweep writes a self-contained workdir. load_results(workdir) reloads it in a fresh process and joins registry-backed algorithm and instance names:

from dp_testing_platform import load_results

df = load_results("results/my_sweep")
print(
    df[["alg_name", "instance_name", "budget", "alg_performance", "status"]]
    .head()
)

Rows still in shards after an interrupted run are included. Persisted numeric values use an explicit dtype schema, and loading does not rerun any algorithm.

Health summary

Start with summarize_results. It groups by algorithm, instance, and concrete budget, reports success and failure counts, and computes summaries over finite outcomes only:

from dp_testing_platform import summarize_results

summary = summarize_results(df)
print(
    summary[
        ["alg_name", "instance_name", "budget", "n_success", "n_failed",
         "alg_performance_median"]
    ]
)

Keep failure counts beside plots: masking non-finite outcomes must not hide an algorithm's reliability.

Result schema and identity

Every task shares these columns:

Column	Meaning
`budget`	Canonical dataclass representation, such as `PureDPBudget(eps=1.0)`; the budget portion of row identity
`budget_family`	Non-identity convenience label: `pure_dp`, `approx_dp`, or `zcdp`
`eps`, `delta`, `rho`	Sparse, non-identity convenience parameters; inapplicable values are blank
`inst_gen_uuid`, `instance_seed`	Instance-generator configuration and data seed
`alg_uuid`, `alg_seed`	Algorithm configuration and algorithm seed
`alg_performance`	Primary metric value
`status`, `failure_reason`	Outcome and failure detail
`library_version`, `library_commit`	Code provenance
`runtime_s`, `peak_rss_mb`	Whole-cell runtime and peak resident memory
`privacy_sound`	False for a diagnostic instance with a data-derived modification

The identity key is (inst_gen_uuid, alg_uuid, budget, alg_seed, instance_seed). The canonical budget string is shared by result and output-sidecar keys. Do not rebuild identity from budget_family and the convenience float columns; those exist only for filtering, grouping, and plotting. Provenance, runtime, and metrics are also not identity fields, so a library upgrade does not silently force recomputation.

Status values

Status	Meaning
`success`	A finite result was produced
`no_release`	Accounted bottom outcome produced by raising `NoRelease`
`timeout`	Cell exceeded `time_limit`
`memory_limit_exceeded`	Cell exceeded `memory_limit_gb`
`error`	Unexpected algorithm failure
`output_infinity`	Output or metric contained infinity
`output_invalid`	Output or metric contained NaN

print(df["status"].value_counts())
ok = df[(df["status"] == "success") & df["privacy_sound"]]

Key metrics

A task may register additional per-cell metrics. Linear regression, for example, adds l2_error_to_ols, normalized_mse, and mse:

print(
    df[["alg_name", "budget", "l2_error_to_ols", "normalized_mse", "mse"]]
    .head()
)

Tasks without key metrics simply omit these columns.

Performance profiles

A performance profile asks how often each algorithm lies within a factor tau of the best on the same instance and budget.

from dp_testing_platform import (
    compute_performance_cdfs,
    compute_performance_ratios,
    plot_performance_profile,
)

ratios = compute_performance_ratios(df, granularity="aggregate")
cdfs = compute_performance_cdfs(ratios)  # columns: tau, algorithm, cdf
ax = plot_performance_profile(df, granularity="aggregate")
ax.figure.savefig("performance-profile.png", bbox_inches="tight")

granularity="aggregate" uses per-(algorithm, instance, budget) medians; "run" retains seed-level outcomes. Exclude a reference algorithm whose metric is exactly zero before taking ratios: a zero denominator makes every competitor's ratio undefined. Plotting requires the [viz] extra.

Family-aware budget curves and tables

The analysis API keeps privacy families distinct. compute_error_vs_budget and plot_error_vs_budget use the canonical budget axis; per_instance_table accepts a concrete typed selector. There is no epsilon-only identity path.

from dp_testing_platform import (
    ApproxDPBudget,
    compute_error_vs_budget,
    per_instance_table,
    plot_error_vs_budget,
)

curves = compute_error_vs_budget(df, value="alg_performance")
print(curves.head())

ax = plot_error_vs_budget(df, value="alg_performance")
ax.figure.savefig("error-vs-budget.png", bbox_inches="tight")

table = per_instance_table(
    df,
    budget=ApproxDPBudget(1.0, 1e-5),
)
print(table.head())

Use budget_family, eps, delta, and rho for convenient exploratory filters, but include the family and every applicable parameter when filtering manually.

Comparing runs

compare_runs overlays performance profiles from completed workdirs without mutating them:

from pathlib import Path
from dp_testing_platform import compare_runs

runs = [Path("results/before"), Path("results/after")]
ax = compare_runs(runs, labels=["before", "after"])
ax.figure.savefig("run-comparison.png", bbox_inches="tight")

Post-hoc metric recomputation

Small released outputs are stored in outputs.npz under the same canonical cell key. recompute_metric regenerates each instance deterministically and applies a new metric without rerunning the algorithm:

from dp_testing_platform import LinearRegression, load_outputs, recompute_metric

stored = load_outputs("results/my_sweep")
spec = LinearRegression.key_metrics["l2_error_to_ols"]
recomputed = recompute_metric(
    "results/my_sweep",
    spec,
    raw_directory="results/my_sweep",
    column_name="l2_error_to_ols_posthoc",
)

Failure rows and outputs above the sidecar element cap receive NaN. Synthetic instances need no raw_directory; real-dataset generators use it to locate data.

Exporting a run archive

export_run_archive packages a completed workdir into per-algorithm CSVs, registries, and a hash-verified MANIFEST.json:

from dp_testing_platform import export_run_archive

manifest_path = export_run_archive(
    "results/my_sweep",
    dest="results/my_sweep_archive",
    metadata={"note": "docs example"},
)

The manifest records the concrete budget grid, algorithm roster, dataset inventory, seed counts, hashes, and execution environment. Export refuses in-flight shards or a destination containing unrelated result files. This is a portable run record, not a leaderboard-submission contract.

See Running evaluations, Hyperparameter tuning, and Adding algorithms.