somatize.torch

Generated from the package

Rendered from the docstrings of somatize 1.0.1 itself, so every signature and every paragraph below is the one the installed package carries. Nothing on this page is written twice.

What only makes sense with torch in front of you: training.

The core does not know what a loss is, nor a gradient, nor an optimizer, and it is not going to — writing this neutrally would ask for a Backend with a single implementor. So it lives here, in Python, and core/ does not change a line:

from somatize import Graph
from somatize.torch import Trainer, parameters

g = Graph.somatize(Encoder().on("cuda:0") >> Head().on("cuda:0"))
t = Trainer(g, objective=cross_entropy,
            optimizer=torch.optim.Adam(parameters(g), lr=1e-3))
t.fit(data, epochs=10)

Split — and whatever else is written into the Learning hole — is for the half that cannot be trained from here. A node on another machine gets no gradient from a backward() run in this process, so a trainer travels to it: it keeps the activation where its autograd graph is, is handed dL/d(what the node produced) and carries on under an optimizer of its own. The node is not asked to know any of it. Split learning, greedy, forward-forward and synthetic gradients are the same hole answered four ways:

Trainer(g, objective=cross_entropy, optimizer=Adam(parameters(g), lr=1e-3),
        trains={"body": Split(SGD, lr=0.1)}, broker=broker)

Importing this also says how a tensor is written down, which is what lets a graph keep what it produces: see somatize.torch._codec.

Training does not touch the graph: afterwards its nodes, its edges, its plan and its placement are the same. What changes are the weights, which live inside the nodes and always did.

That the package is called torch does not shadow the real one: in Python 3 imports are absolute, so import torch in here brings the usual one.

Classes

Audit

Audit(*, every: int = 1, snapshot: int = 50, channels: bool = False, groups: Mapping[str, Mapping[str, Iterable[int]]] | None = None, window: int = 20, inside: Any = None, most: int = 32)

Watches the nodes of a graph and says what it saw. Built by the Trainer when auditing= is given; little reason to make one by hand except to choose a cadence or to declare groups, the channel partitions a node keeps apart:

Trainer(g, ..., auditing=Audit(every=10, channels=True))
Audit(groups={"encoder": {"audio": range(0, 64), "text": range(64, 128)}})

Audit.observed

Audit.observed(graph: Graph) -> list[Fact]

One health fact per node, for the step that just finished.

Called after backward, the only moment when this step’s activations and the gradients for them are both in hand. Empty on an unmeasured step.

Audit.release

Audit.release() -> None

Takes the hooks off. A hook nobody removed is a graph nobody can garbage-collect, and a second watch would double every count.

Audit.watch

Audit.watch(graph: Graph) -> Audit

Hooks every node of this graph, and — with inside= — what is in it.

What comes back is keyed by node, and by node.path.to.submodule for anything inside one. The dot is what lets a figure colour the node while the hover says which layer of it.

Learning

Learning(optimizer: Callable[..., Any], *, every: int | None = None, **how: Any)

What trains one node, on the machine that node runs on.

A technique writes learn(signal, ctx) and nothing else: handed dL/d(what the node produced), it gives back dL/d(what it was given), or None. It reaches the node through held — this step’s activation — and given — the leaf its input became; both are dropped after each learn.

Methods

Learning.accumulating

Learning.accumulating(every: int | None) -> Learning

How many steps go into one update, unless this one already said. A technique that named its own wins, the same rule trains follows for who trains whom.

Learning.beside

Learning.beside() -> tuple[Enters, Learning]

Its two positions in a graph: the one that leafs the input, and itself, which keeps what the node produced.

Learning.closes

Learning.closes() -> bool

Whether this step ends one, which is where the optimizer moves.

Learning.done

Learning.done() -> Any

Lets go of this step’s activation, and gives back what the node was given so a gradient can be read off it. Every learn ends here.

Learning.entering

Learning.entering(value: Any, ctx: Ctx) -> Any

The input as a leaf, remembered, which is what makes dL/d(input) a thing that exists. Called from the other position.

Learning.forward

Learning.forward(value: Any, ctx: Ctx) -> Any

As a node, in the position after the one it trains: an ordinary value is the activation to keep, and an envelope is the gradient to learn from.

Learning.learn

Learning.learn(signal: torch.Tensor | None, ctx: Ctx) -> Any

What to do with dL/d(what the node produced), and what to give back.

The hole. Whatever a technique is, it is this method — and signal being None is nobody owing it anything this step.

Learning.of

Learning.of(node: Any) -> Learning

The node it trains. Said when the graph is put together, because that is the only moment somebody has both.

Learning.opens

Learning.opens() -> bool

Whether this step starts a group, which is where gradients are cleared rather than added to.

Learning.training

Learning.training() -> list[torch.nn.Parameter]

Which parameters it updates: the node’s, and whatever else a technique brought with it — a decoder, a guesser — which is why it is a method and not a line.

Learning.waiting

Learning.waiting() -> Any

The activation this step left, or OutOfStep if there is none. Not a None walking into an optimizer: a gradient for an activation that is not there means the two halves are a step apart.

Also from somatize.Node: at, cached, frozen, mapped, named, on.

Properties

Learning.optimizer

Its optimizer, built the first time it is asked for — over the parameters of wherever it ended up, which is the whole reason it is built here and not by whoever declared it.

NoGradient

Raised, not constructed.

Something the optimizer holds never got a gradient. Its own type because it is worth catching: with a cut on purpose — split learning — this is what you expect, and you say so by taking those parameters out of the optimizer.

OutOfStep

Raised, not constructed.

A gradient arrived for an activation that is not there any more.

Through a Graph it comes back as the text of a ValueError: a node’s failure crosses as a message and not as a type.

Result

Result(history: list[float])

What a training run leaves behind: the loss, step by step.

Result.loss

The last loss, or None if not a single step was taken.

Split

Split(optimizer: Callable[..., Any], *, every: int | None = None, **how: Any)

Split learning: carry on with the chain rule from the gradient of the seam, step, and hand back the gradient of the input. With a group of more than one step the same three movements are taken apart — cleared where the group opens, added to in between, stepped where it closes.

Methods

Split.learn

Split.learn(signal: torch.Tensor | None, ctx: Ctx) -> Any

What to do with dL/d(what the node produced), and what to give back.

The hole. Whatever a technique is, it is this method — and signal being None is nobody owing it anything this step.

Also from somatize.Node: at, cached, frozen, mapped, named, on.

Also from somatize.torch.Learning: accumulating, beside, closes, done, entering, forward, of, opens, training, waiting.

Properties

Also from somatize.torch.Learning: optimizer.

Trainer

Trainer(graph: Graph, *, objective: Objective, optimizer: Any = None, trains: dict[str, Learning] | None = None, every: int = 1, micro: int = 1, store: Store | str | None = None, broker: Broker | None = None, watching: Any = None, auditing: Any = None)

Trains a graph, without the graph finding out — no g.fit(...), so the same graph can be trained three ways without touching it:

t = Trainer(g, objective=cross_entropy,
            optimizer=torch.optim.Adam(parameters(g), lr=1e-3))

The optimizer is the caller’s, which keeps a name registry (optimizer="adam") out. store is a directory, and makes a settled .cached() prefix run once per batch. broker says who knows where each host is, as in Graph.forward.

Training a graph with a slice on another machine is not training that slice: what crosses a wire is the value and not the graph that made it. trains is how that half gets trained anyway, said here because it is a fact of this training run and not of the graph:

Trainer(g, objective=cross_entropy,
        optimizer=Adam(parameters(g), lr=1e-3),   # the half that is here
        trains={"body": Split(SGD, lr=0.1)},      # the half that is not
        broker=Broker.embedded({"gpu": Worker.at("node3:7000")}))

That puts a trainer beside the node rather than inside it, so the node is never asked to know it is being trained. Those weights are that trainer’s, so they come out of this optimizer — parameters(g, without=trains) — and holding both is refused rather than quietly updating them twice.

every is how many steps go into one update and micro how many pieces one step is cut into; they multiply rather than compete. watching is told what happens and is handed on to every forward, so one stream carries both the engine’s vocabulary and this level’s loss and updated.

Trainer.export

Trainer.export() -> Weights

What this training run learnt: its weights, node by node, as {node_id: {key: tensor}}, by the same two ducks everything here asks by.

A snapshot and not a view: detached and copied, so the next step does not move it under whoever is holding it. The optimizer’s state is not in it — momentum is this client’s. Refused for a node trained and running elsewhere: the copy here never learnt anything.

Trainer.fit

Trainer.fit(data: Iterable[Batch], epochs: int = 1) -> Result

Takes one step per batch, for as many epochs as you say.

data is walked once per epoch, so with more than one it has to be re-iterable: a generator is exhausted on the first.

Trainer.load

Trainer.load(weights: Weights) -> None

The mirror of export. Every node it names has to be here with the weights and shapes it says, and nothing is copied in until all of that is true, so a refusal leaves the net as it was rather than half loaded.

Trainer.step

Trainer.step(batch: Batch) -> float

One step: forward, loss, backward, and update when the group closes. Returns the loss whole — divided for the backward pass and not for whoever is reading, or a history would change shape with every.

The primitive, and fit is sugar on top: whatever does not fit in an epoch loop is written as a while over this.

Trainer.update

Trainer.update() -> bool

Applies what has been accumulated and starts a new group, for the group a run ends in the middle of. Does nothing, and says so, if none is open.

Across a cut it costs one pass over the transposed stages and not a step: what travels is the fact that the group is over, in an empty envelope.

Functions

architecture

architecture(graph: Graph, example: Any = None, *, most: int = 48, depth: int = 0, broker: Broker | None = None) -> dict[str, Inside]

What each node is made of, as {node: Inside} — ready for a figure.

g.figure(inside=architecture(g, x))

The graph is run once, with hooks on everything every node holds. Not an optimisation: a node in the middle is handed what the nodes above produced, and tracing it on the graph’s own input feeds a fan-in the wrong thing — which it did, until a picture with an empty box said so. Then, module by module, torch.fx is asked for the same thing; where it can answer it wins, and the seam is a module boundary rather than a judgement call.

example is one input to run the graph on. A composite everybody recognises is one box and is not opened; depth= opens them, and blocks that are the same block collapse to one and a ×N.

kind_of

kind_of(what: Any) -> str

What kind of thing this is, by role and never by exact class name.

A class this table has never heard of that ends in Norm is a normalisation. Guessing by suffix is a guess and a good one — the alternative is calling half of everybody’s models other.

traced

traced(module: Any, example: Any = None) -> Inside | None

What this module is made of. fx if it can, a real forward if it cannot.

example is an input to run it on; without one only the symbolic path is available, and a module that needs a forward answers None.

envelope

envelope(gradient: Any, closing: bool = False) -> Envelope

A gradient in its envelope, which is how one crosses an edge. closing says this gradient ends a group of accumulated ones, so whoever is accumulating applies what it has — it rides on the envelope because it is the same fact seen from the other end.

fedavg

fedavg(exports: Iterable[Export], sizes: Sequence[float] | None = None) -> Export

The average of what several training runs exported, weight for weight. sizes is how many samples each one saw, which is what FedAvg weights by; left out, they weigh the same:

average = fedavg([client.export() for client in clients], sizes=[900, 100])

What is not averaged is whatever is not a floating-point number: a num_batches_tracked is a count and the mean of two counts is not one. The first one’s is kept, which every implementation does and none says out loud.

freeze

freeze(graph: Graph, *node_ids: str) -> None

Settles these nodes — or whatever was already declared .frozen().

With ids it declares and obeys; with none it only obeys, which is what Trainer calls so a .frozen() in the expression is true before the first step rather than after somebody notices.

gather

gather(store: Store, what: Any, *, run: str, round: int, clients: int, mine: int, size: float | None = None, within: float = 600.0, asking: float = 1.0) -> Any

Puts this client’s round in, waits for everybody else’s, and gives back the average. run names the training run, round which round, clients how many there are and mine which one this is. size is how much data this client saw, and it travels in the record so whoever averages can weigh by it. Raises TimeoutError after within seconds, naming who never turned up.

gradient

gradient(value: Any, device: str | None = None) -> torch.Tensor | None

What an envelope carries, as a tensor — or the sum of what a map of them carries. None for something that is not a backward message at all, and None too for an envelope carrying nothing.

parameters

parameters(graph: Graph, without: Container[str] = ()) -> list[torch.nn.Parameter]

The parameters of every node in the graph that has any.

It asks for .parameters() and skips whoever lacks it, so a tokenizer does not stop being a node for having nothing to train. Without repeats by identity — two nodes can share a module — and in declaration order.

without names the nodes to leave out — the ones somebody else updates:

trains = {"body": Split(SGD, lr=0.1)}
Adam(parameters(g, without=trains), lr=1e-3)

Leaving them in is refused by the Trainer: a node trained where it runs and also held by this optimizer is updated twice when where it runs is here.

probe

probe(graph: Graph, example: Any, *, depth: int = 0, most: int = 48, probes: int = 24, watching: Any = None, broker: Broker | None = None) -> dict[str, dict[str, Any]]

What this graph looks like at initialisation, as {where: numbers}.

where is a node, or node.path.to.submodule — the same keys the audit uses and the same scope the figure draws, so a finding from a probe lands on the box a finding from a run would. The answer is the shape somatize.health.seen returns from a store. watching= takes a Recorder or anything callable.

proxies

proxies(graph: Graph, example: Any, *, target: Any = None, objective: Objective | None = None) -> dict[str, float]

Every proxy that can be taken with what it was given, as {name: score}.

Without a target and an objective the three that read a loss are not in the answer, rather than being in it as None: a score that is missing and a score that is bad have to look different.

proxy

proxy(graph: Graph, example: Any, of: str, *, target: Any = None, objective: Objective | None = None) -> float

One cheap score for one candidate, higher being better. of names which — see EVERY. snip and grasp want a target and an objective.

The units are nobody’s, and comparing two runs of the same proxy is the only thing it is for, which is why every one that spans decades comes back as a logarithm.