Skip to content

03 — Search and optimization with Study

Study runs hyperparameter searches: define a search space, pick a strategy (grid, random, bayesian), and hand it a trial function. Every trial is tracked in .soma/runs/<id>/ (crash-safe, resumable).

tracking=False is used here to keep the notebook self-contained.

from soma import Study

def objective(trial):
    x = trial["x"]
    return {"score": 1.0 - abs(x - 0.7)}   # best at x = 0.7

grid = Study(
    "grid-demo",
    search_space=[{"type": "float", "name": "x", "low": 0.0, "high": 1.0}],
    strategy="grid",
    n_trials=9,
    objectives=[("score", "maximize")],
    tracking=False,
)
grid.run(objective)
best = grid.best_trial
print(f"best x = {best['params']['x']:.3f}  score = {best['metrics']['score']:.3f}")

best x = 0.750  score = 0.950

3.1 — Random and Bayesian (TPE) search

For more than 1–2 dimensions, grids explode. random samples uniformly; bayesian builds a model of good regions (ask/tell TPE) and needs a seed for reproducibility.

def objective_2d(trial):
    x, y = trial["x"], trial["y"]
    return {"score": -(x - 0.3) ** 2 - (y - 0.6) ** 2}

space = [
    {"type": "float", "name": "x", "low": 0.0, "high": 1.0},
    {"type": "float", "name": "y", "low": 0.0, "high": 1.0},
]

for strategy in ("random", "bayesian"):
    study = Study(f"{strategy}-demo", search_space=space, strategy=strategy,
                  n_trials=30, objectives=[("score", "maximize")],
                  seed=42, tracking=False)
    study.run(objective_2d)
    best = study.best_trial
    print(f"{strategy:9s} best: x={best['params']['x']:.3f} "
          f"y={best['params']['y']:.3f} score={best['metrics']['score']:.4f}")

random    best: x=0.241 y=0.660 score=-0.0071
bayesian  best: x=0.296 y=0.593 score=-0.0001

3.2 — Experiment seeds: `seeds=[...]`

The variance question — “is this config actually better, or was the seed lucky?” — is first-class: pass seeds=[...] and every sampled config runs once per seed. Each (config, seed) pair is an independent trial with trial["seed"] in its params, and an independent cache line (a crash after 3 of 5 seeds resumes with 3 exact hits).

import statistics, random as _random

def train(trial):
    rng = _random.Random(trial["seed"])           # wire the seed into YOUR framework
    noise = rng.gauss(0.0, 0.05)                  # (torch.manual_seed(trial["seed"]), etc.)
    return {"score": 1.0 - abs(trial["x"] - 0.7) + noise}

seeded = Study(
    "seeded-demo",
    search_space=[{"type": "float", "name": "x", "low": 0.0, "high": 1.0}],
    strategy="grid",
    n_trials=3,
    objectives=[("score", "maximize")],
    seeds=[11, 22, 33],
    tracking=False,
)
seeded.run(train)
print(f"{seeded.n_trials} trials = 3 configs x 3 seeds\n")

by_config = {}
for t in seeded.trials:
    by_config.setdefault(round(t["params"]["x"], 3), []).append(t["metrics"]["score"])
for x, scores in sorted(by_config.items()):
    print(f"x={x}: mean={statistics.mean(scores):+.3f}  std={statistics.stdev(scores):.3f}")

9 trials = 3 configs x 3 seeds

x=0.0: mean=+0.267  std=0.051
x=0.5: mean=+0.767  std=0.051
x=1.0: mean=+0.667  std=0.051

3.3 — Trials that drive a Graph

In real studies the trial function builds a Graph from the sampled params. Thanks to the persistent cache, pipeline stages shared across trials (fixed preprocessing on fixed data) are computed once for the whole study — and survive crashes.

from soma import Filter, Graph

class Center(Filter):
    _cache_version = "nb03-v1"
    def fit(self, x, y=None):
        return {"mean": sum(x) / len(x)}
    def forward(self, x, state):
        return [v - state["mean"] for v in x]

class Ridge(Filter):
    _cache_version = "nb03-v1"
    def __init__(self, alpha=1.0, **kwargs):
        super().__init__(alpha=alpha, **kwargs)
    def fit(self, x, y=None):
        n = len(x)
        xy = sum(a * b for a, b in zip(x, y))
        xx = sum(a * a for a in x)
        return {"w": xy / (xx + self.alpha * n)}
    def forward(self, x, state):
        return [state["w"] * v for v in x]

X = [1.0, 2.0, 3.0, 4.0, 5.0]
Y = [2.1, 3.9, 6.2, 7.8, 10.1]

def ridge_objective(trial):
    # Fluent DSL: >> chains center into ridge (same as node() + edge()).
    g = Graph.somatize(Center() >> Ridge(alpha=trial["alpha"]))
    g.fit(X, Y)
    pred = g.forward(X)
    mse = sum((p - yv) ** 2 for p, yv in zip(pred, Y)) / len(Y)
    return {"mse": mse}

study = Study(
    "ridge-demo",
    search_space=[{"type": "float", "name": "alpha", "low": 0.001, "high": 1.0, "scale": "log"}],
    strategy="bayesian", n_trials=15,
    objectives=[("mse", "minimize")], seed=7, tracking=False,
)
study.run(ridge_objective)
best = study.best_trial
print(f"best alpha = {best['params']['alpha']:.4f}  mse = {best['metrics']['mse']:.4f}")

best alpha = 0.0010  mse = 36.2618

What’s next

Tracking (run dirs, events, metrics) and pruning: notebook 06.
Crash-resume of a killed study: Study.load(run_dir).run(fn, resume=True).