Hyperparameter Tuning API Reference

The DP hyperparameter-tuning machinery — composed tuners, selection mechanisms, and the optuna-based tuner. These are re-exported at the package top level (from dp_testing_platform import ...); the canonical definitions live in dp_testing_platform.hyperparameter_tuning.

dp_testing_platform.hyperparameter_tuning

Differentially private hyperparameter tuning framework.

Provides base classes and implementations for tuning algorithm hyperparameters while preserving differential privacy guarantees. Also includes a non-DP Optuna-based tuner for convenience.

Available tuners

DPSelectionLiuTalwar: DP selection via Liu & Talwar's Algorithm 2. DPSelectionPapernotSteinke: DP selection via Papernot & Steinke (ICLR 2022). DPExhaustiveCompose: DP selection by exhaustively composing over the grid. tune_algorithm_with_optuna: Non-DP tuning using Optuna's TPE sampler.

DPExhaustiveCompose

Bases: DPHyperparameterTuned

DP hyperparameter selection by exhaustive evaluation under composition.

Enumerates EVERY candidate in the discrete hyperparameter grid (the Cartesian product of the per-parameter value lists), privately evaluates each one, and returns the argmin over the private quality scores. Unlike Liu-Talwar selection (a random number of random draws), this evaluates all m candidates deterministically — the right trade-off on tiny grids, where exhaustively spending a small per-candidate budget beats paying the overhead of a random-stopping selector.

Privacy accounting (add/remove-one neighboring relation):

Per candidate. Each candidate splits the tuning data 80/20 into a train split and an eval split. The algorithm and DP quality metric each receive the same typed per-candidate allowance. Because the two splits are disjoint, parallel composition makes their combined release spend that allowance once.
Across candidates. The m candidates reuse overlapping tuning data, so their releases compose sequentially. Each family is divided in its native additive parameters: epsilon and delta for approximate DP, epsilon for pure DP, and rho for zCDP.
Argmin is free. Selecting the best candidate is post-processing of the m already-private scores, so it costs no additional budget.

Reference

Dwork & Roth, "The Algorithmic Foundations of Differential Privacy", 2014, Theorem 3.16 (basic composition). https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf

Bun & Steinke, "Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds", 2016, arXiv:1605.02065, Lemma 1.7 (zCDP composition).

Hyperparameters

(inherited) alg_class, alg_hyperparam_grid, quality_metric, sample_split, cheating.

DPHyperparameterTuned

Bases: BaseAlgorithm, ABC

Abstract base class for DP hyperparameter tuned algorithms.

Wraps any BaseAlgorithm and tunes its hyperparameters using a DP-safe procedure. The workflow is:

Split the input data into a tuning portion and a final portion.
Call the abstract _tune() method on the tuning data to select the best hyperparameters (subclass-defined DP mechanism).
Run the algorithm with the best hyperparameters on the final data.

Tuning and final fitting each receive the full outer allowance. Their data-independent record partitions are disjoint, so they compose in parallel and jointly spend that allowance once.

If cheating=True, the full dataset is reused for the final run without being accounted for in the privacy budget (useful for debugging/comparison).

Privacy precondition (train/eval disjointness). Subclasses that train a candidate and DP-score it inside one selection iteration split the tuning data into disjoint train/eval portions via :meth:_split_data (one data-independent coin per record; see that method for why per-record assignment, not a fixed-size cut, is what the add/remove-one neighboring relation requires) and give EACH the full per-iteration budget. This is sound only because the two operate on disjoint records: they compose in PARALLEL, so the per-iteration base mechanism spends the MAX of the two per-part budgets, not their sum -- the guarantee the selection theorem then charges against. Every such subclass asserts train.n_samples + eval.n_samples == tuning.n_samples after splitting so the disjointness a selection theorem relies on is enforced structurally rather than merely intended.

init

__init__(alg_class: Union[str, type[BaseAlgorithm]], alg_hyperparam_grid: Mapping[str, list[Any] | dict[str, float]], quality_metric: Union[Metric, str, dict[str, Any]], sample_split: float | None = None, cheating: bool = False, name: str | None = None, description: str | None = None, metadata: dict[str, Any] | None = None) -> None

Initialize the DP tuner.

Parameters:

Name	Type	Description	Default
`alg_class`	`Union[str, type[BaseAlgorithm]]`	Algorithm class to tune. Accepts a class or a dotted import path string (e.g., "my_module.MyAlgorithm").	required
`alg_hyperparam_grid`	`Mapping[str, list[Any] \| dict[str, float]]`	Maps hyperparameter names to discrete candidate lists or explicit continuous numeric ranges.	required
`quality_metric`	`Union[Metric, str, dict[str, Any]]`	DP quality metric for evaluating candidates. Accepts a Metric instance, a metric name string, or a dict with 'name' and 'args' keys (resolved from the task's dp_quality_metrics).	required
`sample_split`	`float \| None`	Fraction of data reserved for tuning.	`None`
`cheating`	`bool`	If True, reuse the full dataset for the final run.	`False`
`name`	`str \| None`	Display name for this tuned algorithm.	`None`
`description`	`str \| None`	Brief description.	`None`
`metadata`	`dict[str, Any] \| None`	Arbitrary metadata dict.	`None`

run

run(metadata: TMetadata, input: TInput, budget: DPBudget, seed: int) -> TOutput

Run the tuned algorithm: split data, tune hyperparameters, then run.

Splits the data in the task's own space, tunes on the tuning portion, then runs the selected inner algorithm on the final portion (all inner algorithms are native to self.TASK).

Parameters:

Name	Type	Description	Default
`metadata`	`TMetadata`	Problem specification in self.TASK format.	required
`input`	`TInput`	The private input data in self.TASK format.	required
`budget`	`DPBudget`	Privacy allowance for tuning and the final fit.	required
`seed`	`int`	Seed for reproducible splitting and tuning.	required

Returns:

Type	Description
`TOutput`	The algorithm's output in self.TASK format.

Raises:

Type	Description
`NoRelease`	If a data-independent split leaves one side empty.

DPSelectionLiuTalwar

Bases: DPHyperparameterTuned

DP hyperparameter selection using Liu and Talwar's Algorithm 2.

Iteratively samples hyperparameter configurations, trains the algorithm, and evaluates using a DP quality metric. In each iteration, a biased coin (probability gamma) decides whether to stop. The best configuration seen so far is returned.

An approximate-DP allowance follows Theorem 3.4's epsilon/delta allocation and finite trial cap. Pure-DP and zCDP allowances use the theorem's pure-DP route: the latter first enters the canonical sound pure-spend conversion, then each iteration receives one third of that epsilon.

Reference

Liu & Talwar, "Private Selection from Private Candidates", 2019. https://arxiv.org/abs/1811.07971

Hyperparameters

gamma: Probability of stopping in each iteration. Default: 0.05.

DPSelectionPapernotSteinke

Bases: DPHyperparameterTuned

DP hyperparameter selection via Papernot & Steinke private selection.

Draws a random repeat count K from a published distribution (the distribution family and its parameters are public; the realized K is secret and never released), runs K i.i.d. candidate configurations sampled data-independently from the fixed grid, DP-scores each on the evaluation split, and returns the best-scoring candidate. The selection surcharge over the base mechanism's per-run privacy is accounted with the paper's closed forms.

Accounting route (generic, one route for every base). The selection is charged through the Papernot-Steinke Renyi-DP surcharge -- Theorem 6 for the Poisson repeat draw, Corollary 4 for the TNB repeat draw -- which builds the tuned RDP curve from the base's per-run rho_run-zCDP curve and converts it to an (eps, delta) guarantee. The selector charges only the route it executes: with a Poisson draw it charges Theorem 6, with a TNB draw Corollary 4. If the route does not fit the budget (its selection overhead exceeds the whole eps_total) the run fails cleanly with NoRelease rather than falling back to a cheaper route it would not execute.

Each base run and quality score receives the exact same ZCDPBudget that the selection theorem charges. Their train/eval inputs are disjoint, so they compose in parallel to that per-run rho. A consumer that cannot realize a zCDP allowance refuses structurally. The outer allowance must be approximate DP because the P&S RDP route yields an approximate-DP guarantee.

Reference

Papernot & Steinke, "Hyperparameter Tuning with Renyi Differential Privacy", ICLR 2022. https://arxiv.org/abs/2110.03620

Hyperparameters

repeat_dist: Repeat-count distribution family ("poisson" or "tnb"). Default: "poisson". mu: Poisson rate (also E[K]) for the Poisson distribution. Default: 10.0 — a house choice; the paper prescribes no value. Larger mu explores more candidates but multiplies per-run compute by ~E[K] and grows the charged selection epsilon. eta: TNB shape parameter. Default: 1.0 (geometric / Liu-Talwar). gamma: TNB stopping parameter. Default: 0.05.

dp_testing_platform.hyperparameter_tuning.optuna_tuner

Optuna-based hyperparameter tuning for algorithms.

tune_algorithm_with_optuna

tune_algorithm_with_optuna(task: Task, alg_class: type[BaseAlgorithm], param_grid: dict[str, Any], instances: Callable[[], Iterable[tuple[TMetadata, TInput, Metric]]], budget: DPBudget, num_trials: int, num_alg_samples: int, seed: int, instance_metrics_aggregator: Callable[[Array1D], float] | None = None, per_instance_sample_aggregator: Callable[[Array1D], float] | None = None, direction: str = 'minimize', progress_callback: Callable[[dict[str, Any]], None] | None = None, time_limit: float | None = None, already_isolated: bool = False) -> tuple[BaseAlgorithm, Any]

Tune an algorithm's hyperparameters using Optuna over a set of instances.

Evaluates each trial by running the algorithm across all instances with repeated runs per instance, then aggregating scores. Uses Optuna's TPE sampler for efficient hyperparameter search.

Parameters:

Name	Type	Description	Default
`task`	`Task`	The Task object (used for lift/push when algorithm's task differs).	required
`alg_class`	`type[BaseAlgorithm]`	Algorithm class to tune.	required
`param_grid`	`dict[str, Any]`	Maps param names to candidate values. Single values (and single-element lists) are fixed; longer lists are searched. An empty list falls back to the algorithm's default grid for that parameter; when that default grid is empty too, the parameter is omitted so the algorithm's own default applies.	required
`instances`	`Callable[[], Iterable[tuple[TMetadata, TInput, Metric]]]`	Callable returning an iterable of (metadata, input, metric) tuples.	required
`budget`	`DPBudget`	Privacy allowance passed to each algorithm run.	required
`num_trials`	`int`	Number of Optuna trials (distinct hyperparameter configurations).	required
`num_alg_samples`	`int`	Repeated runs per instance for averaging stochastic results.	required
`instance_metrics_aggregator`	`Callable[[Array1D], float] \| None`	Aggregator across instances (default: mean).	`None`
`per_instance_sample_aggregator`	`Callable[[Array1D], float] \| None`	Aggregator across repeated runs (default: mean).	`None`
`seed`	`int`	Seed for Optuna's sampler and per-trial randomness.	required
`direction`	`str`	Optimization direction, 'minimize' or 'maximize'.	`'minimize'`
`progress_callback`	`Callable[[dict[str, Any]], None] \| None`	Called with a status dict at the start of each trial.	`None`
`time_limit`	`float \| None`	Maximum seconds per algorithm run. None disables the limit.	`None`
`already_isolated`	`bool`	When True, run each inner algorithm call directly in the current process instead of spawning a per-call subprocess. Set this only when the tuning loop itself runs inside an already isolating, resource-capped worker — e.g. a `run_experiment` loky cell or the caller's own `joblib.Parallel` worker — where the per-call `spawn` (a fresh interpreter + re-import every trial on macOS) is pure redundant overhead. The outer worker still provides crash isolation and the resource cap; only the redundant inner spawn is collapsed. Leave False for an un-isolated tuning call so the per-call time limit is still enforced via a subprocess.	`False`

Returns:

Type	Description
`BaseAlgorithm`	A (tuned_algorithm, study) tuple. The algorithm instance has best-found
`Any`	hyperparameters and tuning metadata attached. The study is the optuna
`tuple[BaseAlgorithm, Any]`	Study (typed Any: optuna is an optional extra, so its types are never
`tuple[BaseAlgorithm, Any]`	named in the surface).

Raises:

Type	Description
`ValueError`	If param_grid is empty or direction is invalid.
`RuntimeError`	If Optuna does not find a best trial.
`ImportError`	If the `optuna` package is not installed.