An aside: micro-batches, and what became of the open questions

Generated from the requirements document

This section is part of docs/use-cases.md, the document the project is written from. Each section records a decision at the moment it was taken: what it says is what was true when that slice closed.

The plan was left open here on 16 August 2026, with what was then called CU12 in doubt. It is worth keeping because the doubt was right: “micro-batches” covers three problems that have neither the same owner nor the same value.

problem	what solves it	whose it is	consumer?
the batch does not fit in memory	splitting it and accumulating gradients	the Trainer’s, five lines	yes, and it is 80% of cases
the bubble: cuda:1 idle while cuda:0 computes	chaining micro-batches	the graph’s	doubtful
bounding the live activations	a real 1F1B scheduler	nobody’s, and that is the problem	no

The bubble may already not exist. CUDA launches asynchronously, so a micro-batch loop on the host already overlaps the devices without a scheduler — nothing synchronizes along the way, since Opaque wraps the tensor and there is no .item() in the seam. What a scheduler would add has to be measured before it is written.

Real 1F1B is not ours. Its value is bounding how many micro-batches have their activations live, and for that the backward passes have to be interleaved with the forwards. The backward pass is fired by the Trainer, not by the engine, so a 1F1B scheduler would require the plan to know about the backward pass — i.e. putting training inside the graph, which is exactly what CU11 decided against.

What happened next: the first row is the Trainer’s every= (After CU14), the second and third never opened, and the local worker that was the third candidate became CU12, whose benefit was measurable with one machine — two Python nodes in a wave serialize against the GIL; in two processes they do not.

Training from Rust, researched and never opened

It waits on a consumer with a name: a federated client that trains without a CPython loaded. Four results from 16 August 2026, kept so the work is not done twice:

• tch::Tensor is Send but not Sync, so it does not fit in Value::Opaque, whose bound is Arc<dyn Any + Send + Sync>. tch is ruled out short of wrapping every tensor in a Mutex.
• candle_core::Tensor is Send + Sync — an Arc<RwLock<Storage>> inside, and its own code says the RwLock was chosen for exactly that — so it would fit today without touching the core. Verified by compiling.
• the limit that does not move: a graph is all-Python or all-Rust for the tensors. An Opaque put there by Python carries a PyObject, and a Rust node doing downcast_ref::<candle::Tensor>() gets None. Converting for real would mean copying the raw data and losing the autograd graph, which is what Opaque exists to prevent.
• the order, if the day comes: first a Rust node with parameters, then the collection, and the Trainer last. Never starting with the Trainer.