somatize.data

Generated from the package

Rendered from the docstrings of somatize 1.0.1 itself, so every signature and every paragraph below is the one the installed package carries. Nothing on this page is written twice.

Where the data comes from, and whether the model is learning what you meant.

Two halves that meet on the same word. Where it comes from:

from somatize import Graph, Store
from somatize.data import Parquet, settle, to_polars

sms = Parquet(Store("/data"), "sms/train")
g = Graph.somatize(sms.named("sms").frozen() >> Clean().named("clean").cached())
settle(g)
g.forward({"at": 0, "take": 64}, store="/data")

A source is a node, what it answers with is Arrow, and what the graph is handed is a coordinate — which is what stops a cache weighing the batch on every step. See somatize.data._source.

And whether it is learning what you meant, which is not about the network at all:

from somatize.data import contribution, leaning, shares

said = contribution(g, batches, objective=mse, over=("symptoms", "text"))
shares(said)     # {"symptoms": 0.01, "text": 0.99}
leaning(said)    # {"symptoms": ["IGNORED_INPUT(symptoms)"], ...}

somatize.health asks whether a network is learning: gradients, activations, channels, the update. This asks whether it is learning what you meant — which no amount of looking at a gradient will ever say.

It exists because of a real project: symptom channels for detecting a mental-health condition, where interpretability and performance could be had one at a time and never together. Months went into the architecture. The signal was in the self-disclosure and not in the symptoms, and one afternoon of taking inputs away would have said so.

Classes

Parquet

Parquet(store: Store, name: str)

A parquet file in a store, answering spans of rows.

Parquet(store, name) where name is what the file is bound under. It resolves the name and reads nothing: a graph that names a dataset has not opened it.

Methods

Parquet.forward

Parquet.forward(input: Any, ctx: Ctx) -> Frame

The rows that span names, as a Frame.

Also from somatize.Node: at, cached, frozen, mapped, named, on.

Properties

Parquet.name

The name it was declared under, which is the graph’s word.

Parquet.version

What this dataset is: the digest of its content, which the store had already worked out. It is the duck settle and somatize.torch.freeze both look for.

Functions

contribution

contribution(graph: Graph, batches: Iterable[Batch], *, objective: Callable[[Any, Any], float], over: Iterable[str] | None = None, repeats: int = 3, seed: int = 0, broker: Broker | None = None) -> dict[str, float]

How much worse the score gets without each input, as {name: drop}.

batches is an iterable of (input, target) — the shape a Trainer takes — and input is a mapping, because a graph with two branches is fed a map and the keys are what gets taken away. over names which to try.

repeats is how many shuffles each gets, averaged: three is enough to stop one unlucky permutation deciding an afternoon’s conclusion. Nothing is trained and nothing changed.

leaned

leaned(drops: dict[str, float], *, thresholds: Thresholds | None = None, title: str = 'what it leaned on') -> Figure

The share each input was worth, biggest first.

leaning

leaning(drops: dict[str, float], thresholds: Thresholds | None = None) -> dict[str, list[str]]

What is wrong with what the model is leaning on.

{name: [flag, ...]}, and a name with nothing wrong is not in it — for the same reason a healthy node is not in a diagnosis: no flags is not a clean bill, it is nothing tripped.

settle

settle(graph: Graph, *node_ids: str) -> None

Says what each declared-frozen source is settled at. The same shape as somatize.torch.freeze: the graph declares that a node’s state does not change and whoever knows what is inside makes it true — here by repeating what the store already knew, so it costs nothing.

Without it a dataset is not in the key of anything computed from it and two datasets share a name, which the graph refuses before running.

shares

shares(drops: dict[str, float], thresholds: Thresholds | None = None) -> dict[str, float]

What each input is worth, as {name: share} — the drops divided by what they add up to, so they read as how much of what matters is this.

shuffled

shuffled(what: Any, order: Sequence[int]) -> Any

One input, with its rows put in that order. An Opaque is unwrapped and wrapped again, since shuffling the wrapper rather than the tensor would be a quiet way of measuring nothing.

Torch tensors, lists and tuples. Anything else is left alone: shuffling a thing whose first axis is not the batch is a different question by accident.

to_arrow

to_arrow(frame: Frame) -> pyarrow.Table

That frame as a pyarrow.Table.

to_polars

to_polars(frame: Frame) -> Any

That frame as a polars.DataFrame.

Any and not polars.DataFrame, because polars is not a dependency of this package and annotating a type nobody here can import would make the checker’s answer depend on what happens to be installed.