Publishing has never been verified end to end. This is the one item
that stands between the workspace and a version a stranger can install.
release.yml publishes on a v* tag; until a tag runs green with the
crates and the wheels actually landing, “it works” is a claim about a
local checkout. Trusted publishing (OIDC) now replaces the stored tokens
on both crates.io and PyPI, which means the publisher must be configured
registry-side before the next tag.
Refused, and the message names the path that works. That mode is the local loop — the caller drives context/backward/step and owns when the parameters move — and driving it remotely would need distributed autograd. Training a differentiable graph on workers is set_strategy("data_parallel"), which is a complete round
A backward pass on a worker
soma-worker/src/python_process.rs
Fixed 2026-08-05. A remote fit of a DifferentiableFilter runs forward/loss/backward and leaves the gradients on the parameters, so data_parallel trains. Verified against a hand-computed reference: same init, each shard’s gradient taken separately, the two averaged, one SGD step — an exact match, and different from what either shard alone gives
Gradients as an opaque torch blob
soma-worker/src/python_process.rs
Fixed 2026-08-05. They cross the wire as JSON. The aggregator is in Rust, and the mean of two pickles is not a thing that can be computed: the round died at the aggregation step having done all the work
Targets not sharded
soma-runtime/src/strategy.rs
Fixed 2026-08-05.shard_pair splits inputs and targets together. Sharding only x sent each replica the whole y — shapes that broadcast rather than fail, so every replica trained on pairs that were never pairs and the round reported success
ModelParallel
soma-runtime/src/strategy.rs
Written 2026-08-05. Partitions tile the graph and each is a stage on its pinned worker, threading the activation. A node claimed twice, claimed by nobody, or interleaved with another stage is refused rather than run
PopulationBased
soma-runtime/src/strategy.rs
Refuses by design, not for want of an implementation: every member needs different hyperparameters applied to the graph, and a worker is sent a plan rather than a way to build one. PBT lives where Study lives — soma.Pbt(...).run(train, evaluate), added 2026-08-05
run_pipeline, run_study
soma-mcp/src/exec.rs
Written 2026-08-05. They build the graph a model described out of the project’s own filters and run it in a Python subprocess rooted there. The claim that “the server cannot load user code” was true of the server and beside the point: soma-worker has always run Python in a subprocess
Seed dropped on the remote path
soma-worker/src/ws_transport.rs
Fixed 2026-08-05.Transport::execute now takes the run’s seed and WsTransport puts it on the wire; it was hardcoded None, so a remote sweep shared one cache line across every seed
The empty filters list on that same transport is not a gap: the
worker rebuilds a Python filter by unpickling
SerializedFilter::pickled_filter, and those bytes live only in the
Python layer. A NodeCatalog holds live filters and their states, never
the pickle, so this transport cannot supply them and sending empty
pickles would be worse than sending none. The path that can supply them
builds its own SerializedPlan in soma-python/src/graph.rs.
The strategy layer is no longer a blank: Federated, DataParallel and
ModelParallel all run across workers, with the caller in GraphSession::fit, a
StrategyContext over one transport per worker, and FedAvg and AllReduce
written. PopulationBased stays refused on purpose, with soma.Pbt as
the thing that actually evolves a population.
One trap is worth knowing while developing against a worker: it builds an
isolated venv per pipeline and installs somatize from PyPI, so a
working tree ahead of the last release ran an older Soma on the worker
than the one that pickled the filters. soma.Worker(...) now points its
environments at the calling interpreter’s own package; the standalone
binary takes SOMA_LOCAL_PACKAGE.
Documented where each belongs, not here:
soma ui and the rest of the visualization
deferrals (fANOVA importances, NodeProgress/ParetoUpdated emitters,
a Python-implementable EventSink, parquet compaction), and the
experiment-pool ones (warm-starting a
study from the pool, dedup by cache key, ChronosVector as a real vector
index once an Embedder exists).
