pub struct Thresholds {Show 16 fields
pub grad_low: f64,
pub grad_high: f64,
pub dead_eps: f64,
pub dead_frac: f64,
pub saturated_at: f64,
pub saturated_frac: f64,
pub update_low: f64,
pub update_high: f64,
pub dormant_tau: f64,
pub dormant_frac: f64,
pub leakage_cka: f64,
pub narrowing_of_usual: f64,
pub ignored_input: f64,
pub sole_reliance: f64,
pub plasticity_growth: f64,
pub gain_drift: f64,
}Expand description
The bounds a verdict is taken at.
The whole of the opinion, and it is data. Nothing else in this crate holds a number, which is what lets a record be judged again tomorrow with other bounds and lets two people disagree about a network by comparing two of these rather than two codebases.
The defaults come from the original soma, which tuned them for LayerNorm-ish activations and Adam-sized steps, plus the literature for the three it did not have. They are a starting point and they are meant to be argued with.
Fields§
§grad_low: f64Below this parameter-gradient norm, nothing is being learnt here.
grad_high: f64Above it, the next step will not be a step.
dead_eps: f64A value this close to zero counts as off.
dead_frac: f64More of the output than this being off, on any one step, is dead.
saturated_at: f64A value this large counts as pinned.
saturated_frac: f64More of the output than this being pinned, on any one step, is saturated.
update_low: f64Below this update-to-weight ratio a node is not going to arrive.
Practice puts a healthy one near 1e-3; a decade either side of that is
where the two bounds sit, because the useful signal is an order of
magnitude and not a percentage.
update_high: f64And above it, each step throws away where it was.
dormant_tau: f64A channel whose mean normalised activation is under this is dormant (Sokar et al., ICML 2023).
dormant_frac: f64This much of a layer dormant is part of what says it is losing plasticity.
leakage_cka: f64Linear CKA above this, between two groups meant to stay apart, is leakage (Kornblith et al., 2019).
narrowing_of_usual: f64The update’s stable rank falling below this fraction of its own recent
median is narrowing. 0.0 by default, which never fires.
It is off because it was measured and the measurement did not support
it. Huang et al. (2026) monitor the spectrum of W_t - W_{t-d} and find
it collapses thousands of steps before the loss — but their certificate
is the deviation from a healthy baseline run, and a single training
run has no baseline. Against its own recent median, on a 4-layer GELU
net learning a fixed teacher, three healthy runs dipped to 0.69-0.71 of
their own median and six destabilised ones ranged 0.43-0.86: the two
overlap in both directions, so no bound separates them. The numbers are
in docs/use-cases.md.
What is kept is the metric, which is recorded and drawn: the collapse is visible to a person looking at the curve, and that is a weaker and honest claim. Set this yourself if you have a baseline to compare against, which is what the paper actually asks for.
ignored_input: f64Below this share of what all the inputs are worth, the model is not using that input.
A twentieth: with four inputs an even split is a quarter each, so this is five times below its fair share before anything is said. Generous, because the finding is loud and being wrong about it sends somebody looking at their data for a week.
sole_reliance: f64And above this share, one input is carrying everything.
plasticity_growth: f64How fast a thing has to be moving, per step and relative to itself, to count as growing or shrinking rather than wobbling.
gain_drift: f64How far the signal may grow over a stretch nobody normalises before that is worth saying. One side only, and it is the upper one.
A decade, because the drift is geometric and the useful signal is an order of magnitude rather than a percentage. Measured: everything that trained sat at 2.8x or below and everything that did not was at 100x or above, so the bound has 3.6x of margin below it and 10x above.
There is no lower bound and that is a finding, not an omission. A
plain stack whose signal arrives five ten-thousandths of the size it
went in trained as well as the healthy one — Adam is scale-invariant per
parameter, so a signal that shrank does not stop a step being taken. See
health/tests/normalisation.py.
Trait Implementations§
Source§impl Clone for Thresholds
impl Clone for Thresholds
Source§fn clone(&self) -> Thresholds
fn clone(&self) -> Thresholds
1.0.0 · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read more