Enum Flag 

pub enum Flag {
    Nan,
    Inf,
    Vanishing,
    Exploding,
    MissingNormalisation,
    Dead,
    Saturated,
    Stalled,
    Overstepping,
    DeadChannels(usize),
    IgnoredChannels(usize),
    Leakage,
    Narrowing,
    IgnoredInput(String),
    SoleReliance(String),
    LosingPlasticity,
}

One thing that looks wrong with a node.

An enum because the set is closed and named: a diagnosis is only useful if two runs of it say the same word for the same thing. Every variant is an opinion at a threshold and none of them is a fact — the facts are in the record; these are what somebody thinks of them.

Variants

Nan

A number stopped being a number. Nothing below it means anything.

Inf

Or stopped being finite.

Vanishing

The parameter gradients are so small this node is not being trained. The classic depth pathology, and a profile over depth rather than a property of a network: the early layers go quiet while the last learns.

Exploding

The parameter gradients are so large the next step will not be a step.

MissingNormalisation

The signal has grown over a stretch nobody is normalising.

A conjunction, and both halves are load-bearing: the structural half is baked into the measurement, which counts the gain from the last normalisation upstream. A badly-initialised stack with a norm layer drifts under 3x and trains.

One-sided, and measured rather than assumed: it says nothing about a signal that shrank, because Adam is scale-invariant per parameter. See health/tests/normalisation.py.

Dead

Most of what this node outputs is zero, on at least one step.

Read off the maximum over the window: a layer that dies one step in four is dead, and the mean is exactly what hides it.

Saturated

Most of what it outputs is pinned at the far end of its non-linearity, where the derivative is nothing. Also read off the maximum.

Stalled

It is moving, but by so little relative to its own weights that it will not arrive. The ratio of update to weight, which practice puts near 1e-3 — the cheapest signal there is.

Overstepping

It is moving so much relative to its own weights that each step throws away where it was.

DeadChannels(usize)

How many channels are dead — output near zero across the window. Separate from Flag::Dead: a layer can be alive with a quarter of its width doing nothing, which is a width problem and not a layer problem.

IgnoredChannels(usize)

How many channels are alive and never asked for: they compute something and no gradient comes back. Gradient starvation, and a dormant channel is not the same thing — it is computing nothing to be ignored.

Leakage

Two groups of channels the architecture means to keep apart are carrying the same information, by linear CKA.

Narrowing

The update has collapsed into a few directions compared with what this run was doing before. The earliest warning there is: it moves thousands of steps before the loss does.

IgnoredInput(String)

An input the model is not using: taking it away costs nothing. A network with a perfectly healthy gradient can be ignoring an input all afternoon without a single other flag firing.

SoleReliance(String)

One input carries everything, and nothing else would take over.

Not a failure and not always wrong: sometimes one channel really is the signal. It is worth knowing before the day that channel is missing.

LosingPlasticity

The weights keep growing, the representation keeps narrowing and the units keep going quiet — all three at once, which is what tells a network that has stopped being able to learn from one that is training.

Implementations

impl Flag

pub fn name(&self) -> &'static str

The word this flag is written down as, which is what a record keeps and what somebody greps for.

pub fn family(&self) -> &'static str

Which family of trouble this is. A closed set, so a figure can give each family a colour instead of painting everything one red. By what to do about them and not by what was measured: VANISHING and EXPLODING are both answered by looking at depth and initialisation.

pub fn about(&self) -> &'static str

What to do about it, in one line. Part of the flag and not of whoever draws it: the thresholds and the advice are the same opinion.

Trait Implementations

impl Clone for Flag

fn clone(&self) -> Flag

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Flag

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Flag

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Flag

fn eq(&self, other: &Flag) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for Flag

impl StructuralPartialEq for Flag