Optuna when every trial is expensive

Hyperparameter search is easy to waste. If each trial means a long train and a score pass, grid search and random search burn budget on regions that already look bad. You want a searcher that remembers what happened and spends the next trial where it might still help.

That is what Optuna is for.

What Optuna is

Optuna keeps a study: a search space, one or more objectives, and a sampler that suggests the next parameters from past trials. You write an objective function that calls suggest_* for each parameter, runs your eval, and returns the metric(s). Then you call optimize. Later trials tend to cluster near better regions. For more than one objective you get a Pareto front, not a single "best" point.

A common setup is multi-objective TPE (Tree-structured Parzen Estimator): minimize validation error and inference latency together. Note for engineers: with directions=["minimize", "minimize"], Optuna's default sampler is NSGA-II, not TPE. If you want multi-obj TPE, pass it explicitly — for example sampler=optuna.samplers.TPESampler().

The same pattern works for any slow eval: define the space, score, fold results back into the study, repeat.

Hello Optuna (see the effect in minutes)

Before you wire a heavy pipeline, run a toy so you can see learning.

Minimize (x − 2)² + (y + 1)² with x, y drawn from Uniform(−5, 5). Use TPE for 30–50 trials. Then:

• Plot trial number vs objective value — it should fall over time.
• Scatter (x, y) colored by value — points should cluster near (2, −1).

Create the study with SQLite storage and a named study so the dashboard can reopen the same file unambiguously:

import optuna

def objective(trial):
    x = trial.suggest_float("x", -5.0, 5.0)
    y = trial.suggest_float("y", -5.0, 5.0)
    return (x - 2) ** 2 + (y + 1) ** 2

study = optuna.create_study(
    study_name="toy",
    direction="minimize",
    storage="sqlite:///toy.db",
)
study.optimize(objective, n_trials=50)

Optional dashboard:

optuna-dashboard sqlite:///toy.db

If you cannot watch trials improve on a two-variable bowl, you will not trust the same loop on a real training run.

Multi-objective: two metrics, one Pareto front

Same API, two directions — and an explicit sampler if you want TPE rather than the NSGA-II default:

study = optuna.create_study(
    directions=["minimize", "minimize"],
    sampler=optuna.samplers.TPESampler(),
)

You no longer get one champion trial. You get a front of tradeoffs. In the worked example below, the two objectives are validation error and inference latency. A trial that is great on accuracy and slow at inference is not automatically "better" — it sits on the front until you decide how to pick.

Same pattern, real cost: propose → score → fold → challenge

When each trial is a full train + measure pass, use this loop:

1. Propose hyperparameters (learning rate, depth, regularization, batch size, and so on) with history-informed multi-obj TPE — minimize validation error and latency.
2. Score them on the validation split with the same train/eval path every time.
3. Fold trial metrics back into the Optuna study so the next round proposes from real history, not from a cold start.
4. Challenge leads against your current baseline on a held-out test split — alert only, never auto-promote.

An illustrative lead might look strong on validation (say lower error and lower latency than baseline). That is interesting. It is not a ship call.

What validation measures — and what can fool you

Validation error and latency are averages (or aggregates) over the val set. A mean win on validation can still hide a slice — for example a rare class, a hard domain, or a latency tail that only shows up under load.

Gates that close that gap (so Optuna cannot "win" on average alone): slice metrics by class or cohort, check p95/p99 latency, and keep a test set Optuna never sees during search.

• Fool mode: promote on validation mean.
• Honest mode: the lead must beat or match baseline on held-out test (and on the slices you care about), or you stay with the baseline.

Holdout decides; Optuna does not

Illustrative only (not a real run):

	val error ↓	latency (ms) ↓
Optuna lead	0.12	8.4
Baseline	0.15	9.1

On held-out test the same lead might still lose on one hard slice, or p99 latency might flip the call. Treat that as HOLD on the baseline and keep the lead as alert-only — interesting, not shippable.

The validation table is what the searcher optimized. The test / slice checks are the expansion that hardens the gate — they are not the same numbers Optuna saw while proposing.

Closing

Optuna is how you spend expensive trials where history says they might help. Multi-obj TPE is how you search when you care about more than one metric at once. A held-out challenge is how you keep a search win from becoming a ship decision.

Optuna finds candidates. Holdout decides.