CU10 — Where a node runs

Generated from the requirements document

This section is part of docs/use-cases.md, the document the project is written from. Each section records a decision at the moment it was taken: what it says is what was true when that slice closed.

g = Graph.somatize(
    Tokenize()
    >> (Encoder().on("cuda:0") | Other().on("cuda:1"))
    >> Join()
)
g.devices()   # {"encoder": "cuda:0", "other": "cuda:1"}
g.plan()      # the usual one: the plan says when, not where

Status: closed. 110 tests in Rust, 119 in Python.

The question: where does the device live?

The first design put it in the plan, as Plan::On { device, inner } wrapping a subplan — the same shape Plan::Remote is going to have. It was rejected for a reason that knocks it down entirely: the plan determines the execution order and the concurrency, and placing changes neither. They are two different axes and putting them in the same type ties them together needlessly.

Taking it out of the plan paid off immediately:

• plan.rs is not touched in the whole use case.
• The rule for collapsing contiguous runs of the same device disappears, and it was the most fragile part of the design: the one part that had to be canonical and could stop being so.
• “Placing does not change the plan” stops being something to check and becomes true by construction, because compile does not see the placement. The test is still written, but as a warning for the day someone tries to put it there.

Also rejected, and why:

where	why not
in the Node (fn device(&self))	it puts an orchestration decision inside the implementation’s contract. The node does not choose where it runs; and it also becomes invisible: it cannot be printed or reasoned about
in the Graph	a Graph is topology only. And the engine does not look at it — every plan step has been self-contained since CU3 — so it would have forced passing the graph to the Executor
in the Catalog	it was the runner-up: the engine already has it to hand. It loses because the catalog is the half that is not data, and a placement is. When a subgraph travels to another machine, the placement travels with it and the implementations do not
a generic Metadata	it is the generic name for Placement, and generic is paid for dearly: an id → dict sack cannot hold a typed Device

Decisions taken

1. Placement is a type of its own, and it is given to the engine the way the driver is. That leaves a fourth orthogonal fact, which is what the question made visible:

piece	answers
Graph	what exists and how it connects
Catalog	who executes it
Placement	where
Plan	when, and with what concurrency

It fits what Executor had already written about itself: “executing needs context — today the store and the driver — and tomorrow it will need more”.

2. Device is an enum, not a shape-validated String. The argument that decided it is not exhaustiveness but this one: with an enum, a typo is an error at declaration time. .on("cude:0") fails where it was written; a Device(String) that only checked the shape would accept it and the failure would surface inside torch halfway through a run.

The cost of the vocabulary becoming ours can be paid because the core does not match on a Device anywhere else: it decides nothing based on which one it is, it only carries it. Adding a variant is three lines — the enum, a FromStr arm and a Display arm — and nowhere else stops compiling.

3. The cuda index is mandatory. In torch, bare "cuda" means “the current GPU”, which is thread state. To whoever is placing, “the current one” is not a placement: .on("cuda") is rejected asking for cuda:0. An ambiguous declaration cannot be written.

4. meta goes in as a variant. It is the only device that lets us prove end to end that a placement arrives and is obeyed on any machine. The development machine has a single GPU, so without meta half the questionnaire would depend on the hardware.

5. Unplaced ≠ placed on cpu. The first is “wherever it already is”, the second is an order to move. That is why Placement::of returns an Option rather than a default Cpu.

6. The device arrives via the Ctx, and the one that obeys is the node. It is the consequence of the core not knowing what a GPU is: its role is to carry the declaration to the point of execution. ctx.device arrives written the way torch writes it — "cuda:0" — so it can be handed to .to() without translating.

7. .on() in the DSL, place() with the id, and a single door. .on() is handed out to the leaves without a place and the innermost one wins: (a.on("cuda:0") >> b).on("cuda:1") leaves a on 0 and b on 1. But .on() needs the object inside an expression, and there are two cases where only the id is left: the graph built in a loop, and — the one that really matters — the placement decided afterwards, from whatever is on the machine:

for i, nid in enumerate(g.nodes()):
    g.place(nid, f"cuda:{i % torch.cuda.device_count()}")

They are not two paths: .on() ends up calling place(), so the validation is written once and the DSL inherits it. No orphan id is possible — .on() only names nodes of its own Wire, and place() validates against the graph.

What .on() is not

.on("cuda:1") is not torch.cuda.set_device(1). For a node to compute on a GPU three things have to happen, and the ambient context only affects the third:

what	how	when
the parameters are there	module.to(dev)	once
the input is there	x.to(dev)	every forward
what is created inside is born there	explicit device=	every forward

The counterexample was already in the repo: test_pipeline_torch.py creates the index tensor with torch.tensor(rows) inside the forward. With the Embedding moved to cuda, that blows up with “Expected all tensors to be on the same device”, and no set_device fixes it.

Hence obeying is the node’s job, and the pattern is written by hand — it is five lines, and until they repeat three times there is nothing to pull out into a base class:

def forward(self, x, ctx):
    if ctx.device:
        if self.placed != ctx.device:
            self.lin.to(ctx.device)   # the parameters, once
            self.placed = ctx.device
        x = x.to(ctx.device)          # the input, every time
    return Done(Opaque(self.lin(x)))

The postcondition, which is what prevents the silence

A node that ignores its ctx.device would run in the wrong place without anyone noticing, and that is exactly what this project does not tolerate. From outside there is only one thing to look at: where what it returned ended up. If it does not match, it is a named error:

node `encoder` failed: it declared `cuda:0` but returned a value on `cpu`

Whatever has a .device to look at is checked — a tensor, loose or inside an Opaque. A placed node that returns a list of strings is not checked, and placing it did not make much sense anyway.

The case it flags without it being an error: a node that runs on the GPU and deliberately finishes with a .cpu(). It is accepted knowingly — it is the rare case, the message says exactly what happened, and the alternative was silence.

Why this came after the waves

A wave’s branch runs whole on one thread (CU9’s decision), so a device per branch means something. The other way round did not work: grouping by topological level would have made a branch hop threads, and torch’s device is thread-local.

And what makes the whole use case cheap: .to() between devices is differentiable, so autograd crosses the hop and Opaque has not had to change a line. There is a test: two layers, one on cuda:0 and one on cpu, training end to end.

Questionnaire

Rust (soma-core/tests/unit/device.rs, placement.rs, execution.rs)

• cpu, cuda:N and meta parse, and the round trip gives the same thing
• cude:0 is an unknown kind; cuda asks for an index; cuda:, cuda:x, cuda:1:2, cpu:0 and "" are not shaped like a device
• .on() spreads over the whole piece and the innermost one wins
• each branch of a | in its own place, and what is unplaced stays unplaced
• the node sees its own and only its own — nobody catches the neighbour’s
• a wave’s branches see different devices, each on its own thread
• placing changes neither the plan, nor the graph, nor what it produces

Python (soma-python/tests/test_device.py)

• .on() and place() give the same graph, and .named and .on commute
• placing afterwards in a loop, and replacing overwrites the previous one
• placing a node that does not exist fails, and each bad name with its warning
• ctx.device arrives, and shows up in the Ctx’s repr
• the postcondition fires, and says which node — without torch, with any old object that knows how to say where it is
• with torch: meta end to end without hardware; a node that ignores its device is caught
• with a GPU: cuda:0 → cpu in the same graph, and the backward pass crosses the hop while training

What did NOT go in

Choosing the device automatically. Balancing, “auto”, looking at how much memory is left: that is a policy, and there is nobody asking for it yet.

Splitting a node across devices, and g.to("cuda") for the whole graph.

somatize.torch. The pattern for obeying a placement is written by hand in the test, which is where it is documented until it repeats. (It opens in CU11.)

Generalizing Placement to “a place, local or remote” to get ahead of a worker. (CU12 decided against it: a Host is a name and a Device is a place inside a machine, and they are independent.)

Measured, not asserted

With a single GPU on the development machine, cuda:1 does not exist: spreading across two GPUs can be declared and cannot be executed here. The tests say so in their names rather than leaving it implicit.

And CU9’s warning still stands: do not justify this with a benchmark of two branches on two GPUs. CUDA launches asynchronously and the two already overlap when executed in sequence; what the waves buy is host time.