Hyperparameter Optimization

Core Principle: Search Spaces Live in the Filter

Unlike external optimization frameworks where you define search spaces separately, Soma’s search spaces are co-located with the parameters they describe:

#[derive(SomaFilter)]
struct MyClassifier {
    #[soma(search(low = 0.001, high = 100.0, scale = "log"))]
    C: f64,

    #[soma(search(choices = ["linear", "rbf", "poly"]))]
    kernel: String,

    // Not searchable -- fixed value
    verbose: bool,
}

This co-location provides:

• Compile-time type validation: The macro verifies that search annotations match field types
• No string references: No risk of typos in parameter names
• Single source of truth: Change the parameter, change the search space, in one place
• Automatic aggregation: The graph collects all search spaces from its filters

Search Dimensions

#[derive(Serialize, Deserialize, Clone)]
pub enum SearchDimension {
    /// Continuous range
    Float {
        name: String,
        low: f64,
        high: f64,
        scale: Scale,
        default: Option<f64>,
    },

    /// Integer range
    Int {
        name: String,
        low: i64,
        high: i64,
        scale: Scale,
    },

    /// Set of discrete choices
    Categorical {
        name: String,
        choices: Vec<Value>,
    },

    /// Active only when parent has specific values
    Conditional {
        name: String,
        parent: String,
        parent_values: Vec<Value>,
        dimension: Box<SearchDimension>,
    },
}

#[derive(Serialize, Deserialize, Clone)]
pub enum Scale {
    Linear,
    Log,
    ReverseLog,
}

Type Inference from Rust Types

The derive macro infers the dimension type from the field’s Rust type:

Rust Type	Inferred Dimension	Required Annotation
f32, f64	Float	low, high (required), scale (optional)
i8..i64, u8..u64, usize	Int	low, high (required)
bool	Categorical { [true, false] }	None (automatic)
String + choices	Categorical	choices (required)
enum	Categorical	Declare the variants explicitly as choices

Compile-Time Validation

The macro rejects invalid combinations at compile time:

// ✗ Float range on String field
#[soma(search(low = 0.1, high = 10.0))]
method: String,
// compile_error!("Float range on String field. Use `choices` instead.")

// ✗ low > high
#[soma(search(low = 10.0, high = 0.1))]
lr: f64,
// compile_error!("`low` must be less than `high`")

// ✗ Log scale on integer (ambiguous)
#[soma(search(low = 1, high = 10, scale = "log"))]
n_layers: usize,
// compile_error!("Log scale not supported for integer fields. Use f64 or remove scale.")

Categorical choices

List the choices explicitly. #[soma(search)] with no arguments auto-detects only bool (→ Categorical[true, false]); there is no derive that turns enum variants into choices.

#[derive(SomaFilter)]
struct MySVM {
    #[soma(search(choices = ["linear", "rbf", "polynomial"]))]
    kernel: String,

    #[soma(search(low = 0.001, high = 100.0, scale = "log"))]
    C: f64,

    #[soma(search)]  // bool: becomes Categorical[true, false]
    shrinking: bool,
}

The Searchable Trait

Generated automatically by #[derive(SomaFilter)]:

pub trait Searchable {
    /// The search space defined by field annotations
    fn search_space() -> SearchSpace;

    /// Construct an instance from sampled parameters
    fn from_sample(params: &HashMap<String, Value>) -> Result<Self>
    where Self: Sized;

    /// Extract current parameters as key-value pairs
    fn current_params(&self) -> HashMap<String, Value>;
}

Graph Search Space Aggregation

Aggregation happens Python-side, not in soma-core: a Rust Node stores only the implementation name (NodeKind::Filter { filter_name }) — live instances exist in Python (graph.filters(), graph.steps()) or in the runtime’s NodeCatalog, so the core graph cannot reach their search spaces. graph.search_space() (installed by soma/_study.py) aggregates three sources, prefixing dimension names with the node id to avoid collisions:

• Filters declare theirs as class attributes (lr = search(...)).
• Agents and judges declare theirs at the call site (Agent(model=search(choices=[...]))), because an agent’s hyperparameters are its constructor arguments. Both land as the same kind of dimension, so a study cannot tell them apart.
• Optional edges (g.optional(a, b)) each contribute a categorical edge:<source>-><target>: [True, False] — whether a connection should exist at all is a dimension like any other, the edge optimization that agentic-graph search papers (GPTSwarm, AFlow) do by hand.

graph.apply_params() writes a sampled configuration back onto the live instances, and keeps or cuts the optional edges:

g = Graph.somatize(MyScaler(scale=2.0) >> MySVM(kernel="rbf", C=1.0))

g.search_space()
#  scaler.scale: Float[0.1, 10.0] log
#  svm.kernel:   Categorical[linear, rbf, poly]
#  svm.C:        Float[0.001, 100.0] log

g.apply_params({"svm.C": 3.2, "scaler.scale": 0.7})

study = g.study("tune", strategy="grid", n_trials=4,
                objectives=[("f1", "maximize")])

(Rust-side, SearchSpace::merge_with_prefix exists for the same purpose should catalog-level aggregation ever be needed in Rust; today it has no production caller, and the Searchable trait below is derived but not yet invoked by any runner.)

Search Strategies

#[derive(Serialize, Deserialize)]
pub enum SearchStrategy {
    /// Exhaustive grid search
    Grid { points_per_dim: usize },

    /// Random sampling
    Random { n_trials: usize, seed: Option<u64> },

    /// Bayesian optimization (Tree-Parzen Estimator)
    Bayesian {
        n_trials: usize,
        n_startup: usize,       // random trials before modeling
        seed: Option<u64>,
    },

    /// Successive halving with early stopping
    Hyperband {
        max_resource: usize,     // max epochs/iterations
        reduction_factor: usize, // halving factor (typically 3)
    },

    /// Multi-objective optimization (NSGA-II or similar)
    MultiObjective {
        n_trials: usize,
        objectives: Vec<Objective>,
    },

    /// Agent-guided exploration (future: agent decides what to try)
    AgentGuided { agent_id: String },
}

Pruning Strategies

Pruning stops unpromising trials early, saving computation:

#[derive(Serialize, Deserialize)]
pub enum PruningStrategy {
    /// No pruning
    None,

    /// Prune if metric is below median of completed trials at same step
    Median { n_warmup_steps: usize },

    /// Prune if metric is below given percentile
    Percentile {
        percentile: f64,
        n_warmup_steps: usize,
    },

    /// Bracket-based pruning (used with Hyperband)
    Hyperband,
}

Pruning integrates with the filter via ctx.report_metric():

// Inside a filter's fit() method:
for epoch in 0..self.epochs {
    let loss = train_epoch(&model, &data);
    let val_f1 = evaluate(&model, &val_data);

    // Report to the study. If pruned, this returns Err(Pruned).
    ctx.report_metric("f1", val_f1, epoch)?;
}

The Study

A Study orchestrates the full optimization:

#[derive(Serialize, Deserialize)]
pub struct Study {
    pub id: StudyId,
    pub name: String,
    pub graph: Graph,
    pub search_space: SearchSpace,     // aggregated from graph
    pub strategy: SearchStrategy,
    pub pruning: PruningStrategy,
    pub objectives: Vec<Objective>,
    pub trials: Vec<Trial>,
    pub best_trials: Vec<TrialId>,
}

#[derive(Serialize, Deserialize)]
pub struct Objective {
    pub metric: String,
    pub direction: Direction,
}

#[derive(Serialize, Deserialize)]
pub enum Direction { Minimize, Maximize }

Trial Lifecycle

#[derive(Serialize, Deserialize, Clone)]
pub struct Trial {
    pub id: TrialId,
    pub params: HashMap<String, Value>,
    pub state: TrialState,
    pub metrics: Vec<MetricRecord>,
    pub duration: Option<Duration>,
}

#[derive(Serialize, Deserialize, Clone)]
pub enum TrialState {
    Pending,
    Running,
    Completed,
    Pruned { step: usize, reason: String },
    Failed { error: String },
}

Freezing Parameters

Users can fix some parameters and only search over the rest:

study = Study(
    graph=g,
    freeze={"MySVM.kernel": "rbf"},  # fix kernel, search only C
    strategy=Bayesian(n_trials=50),
    objectives=[("f1", "maximize")],
)

Python API

from soma import Graph, Study, Bayesian, Median

g = Graph.somatize(
    MyScaler(scale=2.0) >> MySVM(kernel="rbf", C=1.0)
)

# View the auto-collected search space
print(g.search_space())
# MyScaler.scale:  Float[0.1, 10.0] log
# MySVM.kernel:    Categorical[Linear, Rbf, Polynomial]
# MySVM.C:         Float[0.001, 100.0] log

# Run optimization
study = Study(
    name="svm_optimization",
    graph=g,
    strategy=Bayesian(n_trials=100),
    pruning=Median(n_warmup_steps=5),
    objectives=[("f1", "maximize")],
)

study.run(train_data, eval_data)

# Results
print(study.best_trial)
print(study.best_trial.params)
# {'MyScaler.scale': 1.23, 'MySVM.kernel': 'rbf', 'MySVM.C': 12.5}

# Visualization (see design/visualization.md; needs `somatize[viz]`)
study.plot_optimization_history()
study.plot_parallel_coordinate()
study.plot_param_importances()
study.plot_intermediate_values()   # learning curves, pruned dashed
study.plot_timeline()              # trial gantt
study.trials_dataframe()           # pandas projection

Events Produced

During a study, the following events flow:

StudyStarted { study summary }
  TrialStarted { trial_001, params: {scale: 0.5, C: 10.0} }
    RunStarted { run_001 }
      NodeStarted { scaler }
      NodeCompleted { scaler, 0.02s }
      NodeStarted { svm }
        TrialMetric { f1: 0.72, step: 1 }
        TrialMetric { f1: 0.78, step: 2 }
        TrialMetric { f1: 0.81, step: 3 }
      NodeCompleted { svm, 1.2s }
    RunCompleted { run_001, 1.22s }
  TrialCompleted { trial_001, f1: 0.81 }
  BestUpdated { trial_001 }
  StudyProgress { completed: 1, total: 100, best: 0.81 }

  TrialStarted { trial_002, params: {scale: 5.0, C: 0.1} }
    ...
    TrialMetric { f1: 0.45, step: 1 }
    TrialMetric { f1: 0.46, step: 2 }
  TrialPruned { trial_002, step: 2, reason: "below median" }
  StudyProgress { completed: 2, total: 100, best: 0.81 }

  ...

StudyCompleted { best_trials: [trial_042] }