CU12 — A slice that runs in another process

# A. the worker is your own code, already on that machine
g = Graph.somatize(Encode() >> Classify().at("gpu-box"))
g.forward(x, workers={"gpu-box": Worker.at("gpu-box:7000")})

# B. the worker is a bare node: `pip install somatize` and nothing else
#    python -m somatize.worker --listen 0.0.0.0:7000
g.forward(x, workers={"gpu-box": Worker.at("gpu-box:7000",
                                           mode="network", send=["my_package"])})

Status: closed. A crate of its own — transport, which left for soma-fabric/wire after CU27 — and the generic worker in somatize.worker. 74 tests in it — 49 of the worker, 22 of the protocol, 3 of the artifact — plus test_remote.py (37), test_integration.py (17), test_manifest.py (13) and test_fingerprint.py (13) in Python.

It comes in two halves. The first is in the core and adds three things: Host — a name, and not a Device, which is a place inside a machine — Plan::Remote, and the Transport hole. The second is the crate that fills it, and it lives outside the core for the same reason python/ does: here there are child processes, sockets and a byte format, three things a core has no business knowing.

The benefit is measurable here, with one machine and no cluster: two Python nodes in a wave interleave but do not overlap, because of the GIL, and test_waves.py already said so as a known limitation. In two processes they overlap, and there is a test that makes them prove it by meeting in a file.

The question: who resolves the implementations?

A plan travels; a Catalog does not — an Arc<dyn Node> has no way of crossing a wire. So somebody over there has to turn a name into something executable, and there are exactly two answers:

	where it gets the code	who it is for
`Serving::own`	it brings it: the same binary, the same clone	you control the infrastructure
`Serving::provisioned`	the client sends it, in an `Artifact`	a bare node, and you do not

Two constructors and not an optional parameter, because they reject different things: offering the first an artifact is an error, and not offering the second one is too. An Option would have turned both refusals into a branch somebody forgets.

What travels, and what deliberately does not

The plan, the values the slice reads and does not produce, and the placement — three things that are data. And, for an empty worker, an artifact this crate does not look at, which is where the nodes and the driver ride: whoever packs one packs the other.

Not the catalog as such. Not the environment: that torch is installed over there is the business of whoever stood the worker up, and putting it in cost the original soma 420 lines of environment manager and a hot pip install. And not a Value::Opaque, which points into the process that made it — it fails at encoding time, with the host in front of you.

Host is a name and Device is a place inside a machine, and they are independent: .on("cuda:0") and .at("gpu-box") can be written in either order. What a name resolves to is said by whoever executes, in forward(workers=…), so the same graph spreads across two processes here or two machines there without touching a line of what was declared.

The two strategies, asked by `kind` and not tried in a chain

`mode=` here	`kind` on the wire	what travels	what the worker supplies	size
`"project"` (default)	`project`	names, versions and state	the code, from its own clone	48 bytes for two nodes
`"network"`	`pickle`	the code as well, via `cloudpickle`	nothing	megabytes

Two vocabularies for the two sides of the same choice, and they are not the same word: what you write is what the artifact is for — a worker on the network that has nothing — and what travels is what it is, a pickle. send= names the modules of yours that have to go inside it, because cloudpickle serializes by reference anything importable, which leaves out exactly the case a generic worker exists for.

project is what you want when the worker runs in a clone of the project: tens of bytes per node, no coupling between interpreters, and it checks the version. pickle removes all friction when the worker does not have your code at all, and pays for it by demanding that both interpreters look very much alike.

Versioning, which project needs and gets almost for free. A class’s fingerprint is the hash of its AST, plus transitively whatever of yours that code names — a helper in the same module, a base class, a module constant. Not comments and not docstrings: comparing text would make versioning noise. It stops at what is installed, at what has no source, and at somatize itself, which goes into the client’s identity instead so that upgrading the library does not invalidate every class at once.

Where the check lands is the good part: pickle’s own find_class. The whole policy is one method and pickle does the rest, nested objects included. --strict (the default) stops with both versions in front of you; --lucky runs whatever it has and says so on stderr — running a different version in silence is what gets discovered three days later.

Decisions taken

1. The name is announced before the artifact is sent. Hello { runtime, offering } carries the artifact’s id, not the artifact. The saving is real — a pickled catalog with weights is megabytes and asking “do you have sha256:abc…?” is forty bytes — but it is not the point. The point is that the day there is a store, a worker answers Ready off its own shelf and the protocol does not change a line. It is git’s have/want and docker’s layer exchange. (That day came in CU13.)

2. The client identifies itself in the greeting. This is the original’s lesson written as a method: its worker chose an interpreter with $SOMA_PYTHON or python3 from the PATH, and a pickled filter rebuilt by an interpreter that is not close enough comes back as the class’s __dict__ instead of an instance — surfacing as 'dict' object is not callable from inside a subprocess, with nothing pointing at the version gap. Refusing on connect, with both runtimes in front of you, is cheaper than anything afterwards.

3. An artifact’s id is set by whoever produces it. Hashing the bytes here would be the natural thing and it is wrong: without interpreting the content there is no criterion for saying when two artifacts are the same one, and two pickles of one catalog can differ byte for byte. Whoever produces it knows what identifies it; here we compare strings.

4. The driver travels with the nodes, in the same artifact and by the same strategy. A node that answers Await has to be served where it runs. Declared versus injected is about the graph — a node is in it, a driver is not — and not about how either one gets there.

5. One thread per conversation. See below; it is what the tests found.

6. A worker holds one catalog. A second client provisioning it with a different artifact would pull it out from under the first, and an id present in both would silently run the wrong implementation. Each session remembers what it greeted with and is told so, rather than executing somebody else’s nodes. Checked at the job and not at the greeting, because that is where it can go wrong and there is no race to lose there.

7. Chunks with a length in front, and a cap. A pipe has no messages, it has bytes. The cap is not a limit of the format — the length is a u32 — it is a safety net: four ASCII characters on the worker’s stdout read as a length of between 500 MB and 2 GB, and without it a stray print() is a hung process with no message.

8. There is no authentication, and there will not be. Whoever reaches that port runs code on that machine as that user — pickle artifacts are opened with cloudpickle.loads and project ones resolve classes out of that clone. That is ssh’s job and srun’s, not a framework’s, and it is said plainly in somatize.worker’s own docstring rather than left to be discovered.

What the tests found

Serving one conversation at a time deadlocked. Two branches of a wave open two connections; the second sits in the accept queue and the first does not release its own until its forward finishes. The integration test caught it, which is where it had to show. The lesson was not “do not serialize” but serialize at message granularity, not session granularity.

A failing test that had stood a worker up hung cargo test for ten minutes. The kill was the last line of each test, and a test that fails never reaches its last line — so the orphan kept the test binary’s inherited stderr open and cargo waited on that pipe instead of reporting the failure. Twenty milliseconds of failure, six hundred seconds of wait. It is a Drop now.

A stray print() in a user’s node is on the wire. stdout is the protocol, so there is not one println! in a worker and talking goes to stderr. In Python this is more dangerous, because the print can be in a library on import, and it is why the frame cap exists.

Questionnaire

The core’s seam (soma-core/tests/unit/execution.rs, with a double that never leaves its seat)

what a slice reads and does not produce is what travels with it, and no more
the placement travels; the host half does not, having already done its job
a host nobody resolves is not executed here just in case
what comes back is merged as if it had been produced here

The transport (soma-fabric/wire/tests/unit/)

Python (test_remote.py, test_integration.py, test_manifest.py, test_fingerprint.py)

.at() sends the whole piece, the innermost one wins, and the order of .on() and .at() does not matter
a worker with the project receives only names and state, and the code does not go over the wire
a worker without the project says so instead of guessing
another version of the code stops with --strict and is reported with --lucky
the driver’s state travels with it, and it runs over there and not here
one driver serves both sides of the same run
the fingerprint changes with a helper, a base class or a module constant, and not with a comment or a docstring

The cluster, which came later and is where this is really tested

Every test above runs its workers as subprocesses of the test. That proves the protocol and cannot prove the rest: those workers share this filesystem, this interpreter and these installed packages, so “the worker cannot import your module” is a trick with sys.path and “another version of the code” is the file rewritten underneath.

soma-python/tests/cluster/ is the same thing without the tricks: four containers declared in its own docker/compose.yaml, each with the wheel and nothing else, and the client outside. What only becomes provable there:

	as subprocesses	as containers
the worker has none of your code	a trick with `sys.path`	it really has none
another version of it	the file rewritten	another mount, another image
two hosts	two processes on the loopback	two network namespaces
a store shared between workers	a common `/tmp`	a volume between machines
a worker with a GPU	—	a device, and torch, in one of them

It is opt-in — SOMA_CLUSTER=1 python -m pytest tests/cluster -q — because the first build takes minutes, and the suite has to keep taking seconds.

Writing it turned up something missing: Serving::store and Serving::keeping existed in Rust and were not reachable from the shipped worker. The generic worker now takes --store DIR, which is what lets two containers share a shelf: one keeps what a node produced, and the other reads it.

Training across the cut, which is where a wire really shows

What crosses a wire is the value, not the graph that made it. So a node with parameters that runs on another machine gets no gradient here, and — this is the part worth writing down — nothing failed when that happened: the loss came down, because whatever was downstream of it was learning, and half the net never moved. Silently wrong numbers, with nobody asking the question.

The question is asked now, and at the level that can ask it. After the first backward(), if the optimizer is about to update a parameter that received no gradient, Trainer stops with NoGradient and names the node. It is more general than the case that prompted it: a slice on another host, an output read back from a store, a branch the loss never reads — one symptom, one check.

And .at() is not a refusal to train, which is why the check is about gradients and not about hosts. The far half can perfectly well train itself — that is split learning — and it worked here with no framework at all: a node told which half of the pass it was in, branching on it, keeping its activation between the two calls.

Three things that were already there made it fall out: a worker keeps its catalog, so the node object survives between calls and its activation stays alive on the far side; a node is one contract, so it dispatches on its input instead of needing a kind of its own; and a gradient is a tensor like any other, so it crosses as data. Those three are why CU14 could take it over without a new variant anywhere — the mechanism was already right, it was the user who was being asked to drive it. Nobody writes that node any more; it survives in the tests as the control, because the strongest thing a framework can show is the loop it replaced, running beside it, producing the same losses step for step.

What did NOT go in

Authentication and encryption, on purpose and for good (decision 8) · installing the environment, which is what cost the original 420 lines · scheduling: which host gets what is declared, not decided — there is no placement policy and no load balancing · retrying a failed slice, because a node that already ran half of itself is not idempotent and nobody has said what it means to run it again · a protocol version, since both sides are the same binary from the same cargo build; the day they stop being so, the place for one is the Hello, which already negotiates the runtime · and a store, which is CU13 and where the have/want finally got its have.