05 — The health of a network

The third of the three things observability was split into, and the one that is an opinion: the record says what happened, this says what somebody thinks of it, at thresholds somebody chose.

The line between them is an invariant, and it is what this notebook is really about:

A diagnosis has to be reproducible from the stored record, without training again.

So nothing below trains twice. Each pathology is built, trained once, written down — and then diagnosed from the store, argued with by moving a bound, and diagnosed again.

import tempfile

import torch

import somatize.torch  # noqa: F401
from somatize import Graph, Node, Opaque, Recorder, Store
from somatize.health import (
    Thresholds,
    about,
    alerts,
    diagnose,
    flags,
    history,
    overlaid,
    profile,
    seen,
    where,
)
from somatize.record import gantt, progress
from somatize.torch import Audit, Trainer, architecture, parameters

torch.manual_seed(0)


class Block(Node):
    """One layer and a non-linearity, which is all any of these need."""

    def __init__(self, width=16, activation="relu", bias=None, gain=None):
        self.net = torch.nn.Linear(width, width)
        if bias is not None:
            torch.nn.init.constant_(self.net.bias, bias)
        if gain is not None:
            with torch.no_grad():
                self.net.weight *= gain
        self.after = {
            "relu": torch.nn.ReLU(),
            "sigmoid": torch.nn.Sigmoid(),
            "tanh": torch.nn.Tanh(),
            "none": torch.nn.Identity(),
        }[activation]

    def forward(self, x, ctx):
        return Opaque(self.after(self.net(x)))

    def parameters(self):
        return list(self.net.parameters())


def chain(blocks):
    named = [block.named(f"b{i}") for i, block in enumerate(blocks)]
    wired = named[0]
    for one in named[1:]:
        wired = wired >> one
    return Graph.somatize(wired)


def trained(g, store, run, *, steps=20, lr=0.05, auditing=True, width=16):
    """One short run, audited and written down. Nothing here is trained twice."""
    t = Trainer(
        g,
        objective=torch.nn.functional.mse_loss,
        optimizer=torch.optim.SGD(parameters(g), lr=lr),
        auditing=auditing,
        watching=Recorder(store, run=run, summarising=["loss"]),
    )
    for _ in range(steps):
        t.step((torch.randn(32, width), torch.randn(32, width)))
    return t


store = Store(tempfile.mkdtemp())

Measuring is opt-in, and it decides nothing

auditing=True puts hooks on every node that holds a torch module. What comes out is a health fact per node per step — numbers, in the same record as the loss — and whether those numbers are bad is somebody else’s job.

The cheap half is a handful of reductions on tensors already in memory. The expensive half is an SVD and runs on a cadence: Audit(every=4, snapshot=50).

Vanishing is a shape over depth, not a property of a layer

σ′(z) ≤ 0.25 per layer, so with unit-gain init the backpropagated signal shrinks geometrically with depth. The early layers go quiet while the last one still learns — which is why one number per node says nothing and the profile says all of it.

torch.manual_seed(0)
deep = chain([Block(activation="sigmoid") for _ in range(9)])
trained(deep, store, "deep-sigmoid")

profile(store, run="deep-sigmoid", of="grad_norm")

Figure from cell 4

Nine orders of magnitude across nine layers, and it is a straight line on a log axis — geometric decay, exactly as the theory says.

The same profile in the update-to-weight ratio is the more actionable one, because it has the learning rate in it: it says how far each layer actually moves per step, against its own size. Practice puts a healthy layer near 1e-3, and the dotted line is that.

profile(store, run="deep-sigmoid", of="update_ratio")

Figure from cell 6

The last two layers sit on the line and the first seven are between four and nine orders below it. That number is the cheapest signal in the whole audit, and the original framework measured it without ever saying anything about it.

flags(store, run="deep-sigmoid")

Figure from cell 8

And loudly

A table is for reading; this is for noticing. A finding nobody saw is a finding nobody had, which is why the original framework put these on the screen as toasts.

alerts(store, run="deep-sigmoid")

deep-sigmoid — 8 finding(s)

⚠ VANISHING

this node is barely being trained — look at the depth profile, not at this node alone: it is the early layers that go quiet first

⚠ STALLED