somatize_health/
seen.rs

1//! What was measured about one node over one window.
2
3/// The numbers a verdict is taken over.
4///
5/// **`None` means nobody measured it**, which is not zero and not healthy. A
6/// node with no parameters has no gradient norm; a window with no snapshot in
7/// it has no effective rank; per-channel statistics are opt-in because they
8/// cost a reduction per step. A metric that was not taken cannot raise a flag,
9/// and it must not silently pass for one that was taken and came out fine.
10///
11/// It is `Option<f64>` and not a `NaN` sentinel, which is what the original
12/// used. `NaN < x` being false does make an unobserved metric quietly not
13/// flag — elegant, and it works right up until somebody sums a column.
14///
15/// # A window, not a step
16///
17/// Every field here is already reduced over however many steps the caller
18/// decided to look at. Which steps those were, and how many, is the caller's:
19/// a verdict that quietly needed a particular cadence would be a threshold
20/// hiding in a schedule.
21#[derive(Debug, Clone, Default, PartialEq)]
22pub struct Seen {
23    /// Anything in the window was not a number.
24    pub nan: bool,
25    /// Anything in the window was not finite.
26    pub inf: bool,
27    /// The mean L2 norm of this node's parameter gradients.
28    pub grad_norm: Option<f64>,
29    /// The **largest** fraction of the output that was off, over the window.
30    pub zero_frac_max: Option<f64>,
31    /// The **largest** fraction that was pinned, over the window.
32    pub sat_frac_max: Option<f64>,
33    /// The mean ratio of the size of a step to the size of the weights it
34    /// moved.
35    pub update_ratio: Option<f64>,
36    /// How many channels were off across the whole window.
37    pub dead_channels: usize,
38    /// How many were alive and never asked for.
39    pub ignored_channels: usize,
40    /// What fraction of the channels are dormant on Sokar's normalised score.
41    pub dormancy_frac: Option<f64>,
42    /// The largest linear CKA between two groups the caller declared separate.
43    pub group_cka: Option<f64>,
44    /// The effective rank of the representation at the end of the window.
45    pub eff_rank: Option<f64>,
46    /// How the effective rank is moving, per step and relative to itself.
47    pub eff_rank_slope: Option<f64>,
48    /// How the norm of the parameters is moving, likewise.
49    pub param_norm_slope: Option<f64>,
50    /// The stable rank of the update over the window: how many directions this
51    /// node actually moved in.
52    pub update_rank: Option<f64>,
53    /// And what that usually was for this run, which is the only reference a
54    /// single training run has.
55    pub update_rank_usual: Option<f64>,
56    /// The scale of the signal here against where the last normalisation
57    /// upstream left it.
58    ///
59    /// Measured before a step is taken, by a probe rather than by an audit: a
60    /// probe knows what feeds what, because it traced it. Where the last
61    /// normalisation **is** is structure and not a bound, which is why it is
62    /// baked into the number and the threshold stays on this side of the wall.
63    pub signal_gain: Option<f64>,
64    /// The factor a gradient at the output arrives here by:
65    /// `sqrt(E||J^T v||^2)` over random probes, from this layer to the output.
66    ///
67    /// Deliberately not a parameter-gradient norm. At initialisation there is
68    /// no loss, so a gradient norm would be taken against a target somebody
69    /// made up and would land in [`Seen::grad_norm`] at a different scale, to
70    /// be judged by the same bound. This one is a ratio and needs no target.
71    ///
72    /// **It raises nothing, and that was measured.** Walked across criticality,
73    /// the worst network that still trained read 1.41 and the best one that did
74    /// not read 1.95: a factor of 1.4 is where the sampling landed and not a
75    /// bound. See `health/tests/isometry.py`. The number is recorded and drawn,
76    /// because its profile over depth is the vanishing picture and a person can
77    /// read one.
78    pub jacobian_gain: Option<f64>,
79    /// How spread that Jacobian's spectrum is: `s_max / s_rms` of a random
80    /// sketch of it.
81    ///
82    /// Dynamical isometry (Pennington et al., 2017) is a claim about the
83    /// spectrum's **shape** and not its size — a flat one trains dramatically
84    /// faster than one with the same mean and a long tail — and a mean cannot
85    /// see the difference.
86    ///
87    /// **It raises nothing either, and the inversion is the reason**: a network
88    /// reading 1.87 trained and one reading 1.76 did not, so the failing one
89    /// had the tighter spectrum.
90    ///
91    /// There is a rule underneath both of these and it is worth more than
92    /// either. [`Flag::MissingNormalisation`](crate::Flag::MissingNormalisation)
93    /// separates because the forward scale is a **runaway** — a geometric
94    /// process either stays put or leaves by decades, and there is nothing in
95    /// between to be wrong about. These two vary **continuously** with how well
96    /// a network turns out, and something continuous is a ranking. A ranking
97    /// belongs at level 3, beside the proxies, where a number only ever means
98    /// something next to another candidate's.
99    pub jacobian_spread: Option<f64>,
100}
somatize_health/seen.rs

somatize_health/
seen.rs
