somatize.health

Whether what happened was healthy. An opinion, and it says so.

from somatize.health import diagnose, about
from somatize.torch import Audit

t = Trainer(g, ..., auditing=True, watching=Recorder(store, run="tuesday"))
diagnose(store, run="tuesday")           # {"body": ["VANISHING"]}
about("VANISHING")                       # what to do about it

Three things get called observability and they are not the same: the declaration drawn, the record of what happened, and this — a judgement about that record, at thresholds somebody chose. The line between the last two is an invariant, and it is a test:

A diagnosis has to be reproducible from the stored record, without training again.

So the verdict lives in a Rust crate with no dependencies at all and this module carries a store to it. Change a bound and ask again.

flag	what it means
`NAN` / `INF`	a number stopped being one; nothing below it means anything
`VANISHING` / `EXPLODING`	the parameter gradients are too small to train on, or too big to step on
`DEAD` / `SATURATED`	the output is off, or pinned where the derivative is nothing
`STALLED` / `OVERSTEPPING`	the update is tiny, or enormous, next to the weights it moves
`DEAD_CHANNELS(n)` / `IGNORED_CHANNELS(n)`	part of the width does nothing, or does something nobody asks for
`LEAKAGE`	two groups meant to stay apart carry the same information
`NARROWING`	the update has collapsed into a few directions
`MISSING_NORMALISATION`	the signal grows over a stretch with nothing normalising it
`LOSING_PLASTICITY`	weights growing, rank falling and units going quiet, all at once

about(flag) says what to do about each, beside the thresholds because they are one opinion.

The same three functions read a probe — somatize.torch.probe(g, x), one forward that was recorded and never trained — because a probe writes run/<id>/0 and nothing here ever asked what made a record.

profile(store, run=...) is the picture, because vanishing is a shape over depth and not a property of a layer; flags(store, run=...) is the table. They are the only figures here where colour may mean good-or-bad, because they are the only ones drawing opinions.

Classes

`Thresholds`

Thresholds(**over)

The bounds a verdict is taken at. The whole of the opinion, and it is data: change one and the same record answers differently, without training again.

`Thresholds.as_dict`

Thresholds.as_dict()

What they are, as a dict — for writing them down beside a diagnosis, so that an alarm from last week can be read with the bounds it was taken at.

Functions

`alerts`

alerts(store: Store, *, run: str, thresholds: Thresholds | None = None, last: int | None = None) -> Alerts

What is wrong, loudly. See [Alerts].

`about`

about(flag)

What a flag means and what to do about it, by name.

Beside the flag and not in whoever draws it: the bounds and the advice are one opinion, and splitting them is how a dashboard ends up saying something this library never said.

`diagnose`

diagnose(store: Store, *, run: str, thresholds: Thresholds | None = None, last: int | None = None) -> dict[str, list[str]]

What is wrong, as {where: [flag, ...]}. where is a node, or node.path.to.submodule when inside= was asked to look in.

A node with nothing wrong is not in the answer. Empty would say this was checked and is fine, and no flags does not mean that: a metric nobody measured cannot raise one, and seen is what says what was measured.

A fetch per forward looked at, because the numbers are in the blobs.

`flags`

flags(store: Store, *, run: str, thresholds: Thresholds | None = None, last: int | None = None) -> Figure

What is wrong with each node, and what to do about it. A table and not a list, because a diagnosis without its advice is a word somebody has to look up. A node with nothing wrong is not a row: empty would read as checked and fine, and a metric nobody measured cannot raise a flag.

`history`

history(store: Store, *, run: str, node: str, of: str = 'grad_norm', last: int | None = None) -> list[tuple[int, float]]

One measurement of one node over the run, as (forward, value) pairs.

What a curve is drawn from — a gradient norm falling away, an update ratio drifting. A fetch per forward, so last= is worth using.

`overlaid`

overlaid(graph: Graph, store: Store, *, run: str, thresholds: Thresholds | None = None, last: int | None = None, inside: Inside | None = None) -> Figure

The graph, with what is wrong marked on the nodes it is wrong in — the answer to where, which is the question a diagnosis of a distributed graph raises.

Health gets a channel of its own: the fill goes on saying where a node runs, the outline turns to the alarm colour, and the flags are a badge. Recolouring the fill would let is this unhealthy eat where does this run.

inside is what somatize.torch.architecture gives back, and passing it is what makes a finding land on the layer it is about. Findings from a node whose architecture was not drawn land on the node.

`profile`

profile(store: Store, *, run: str, of: str = 'grad_norm', thresholds: Thresholds | None = None, last: int | None = None) -> Figure

One measurement per node, across the graph — the shape over depth. of is any number the audit wrote down. A node that raised a flag is drawn in the alarm colour and says which on its hover. Nodes are in declaration order, which for a chain is the order the gradient travelled, backwards.

`seen`

seen(store: Store, *, run: str, last: int | None = None) -> dict[str, dict[str, float | bool]]

The numbers a verdict would be taken over — the latest of each.

Keyed by node, and by node.path.to.submodule for anything inside= was asked to look at. The dot is what lets a figure colour the node while the detail says which layer of it: this node is unhealthy is not an answer when the node is twenty layers deep.

For looking at what was measured rather than at what somebody thinks of it, and for taking the verdict yourself with verdict(seen[where], bounds).

`where`

where(store: Store, *, run: str, thresholds: Thresholds | None = None, last: int | None = None, inside: Inside | None = None) -> Overlay

A diagnosis folded onto the nodes of a graph: {node: [flag, ...]}.

What overlaid hands to the figure, and what to hand to graph.figure() yourself if you are composing something else. A finding inside a node is named on its flag — LEAKAGE in net.2 — because the box it lands on is the node.

`within`

within(where: str) -> str

The node a key belongs to, whether or not it names a submodule of it.