somatize.study

Generated from the package

Rendered from the docstrings of somatize 1.0.1 itself, so every signature and every paragraph below is the one the installed package carries. Nothing on this page is written twice.

Level 3: what is above one training run.

The graph is a network — one forward. The Trainer is a training run — an afternoon. This is the level above and, like a federated round, it has no type: N training runs are a for:

from somatize.study import Partition, Pruner, Sampler, Space
from somatize.torch import Trainer

space = Space().real("lr", 1e-5, 1e-1, log=True).choice("opt", ["adam", "sgd"])
sampler, finished = Sampler.tpe(goal="min"), []

for trial in range(50):
    point = sampler.ask(space, trial, finished)
    g = build(**point)                       # a Point is a mapping
    t = Trainer(g, objective=cross_entropy, optimizer=Adam(parameters(g)))
    finished.append((point, t.fit(data, epochs=10).loss))

What lives here are the pieces that for asks for, all of one shape: numbers in, a decision out — never a tensor. That is what lets it be Rust while the loop stays in Python.

A Pruner says whether a trial going badly is worth another epoch, and it stops nothing — it answers and the loop stops calling:

for epoch in range(50):
    reported.append(t.fit(data, epochs=1).loss)
    if why := pruner.verdict(reported, finished):
        break

The three schemes differ in what they judge against: median/percentile the other trials, threshold a constant, patience the trial itself. The samplers differ in what they look at: grid at the space’s shape and it is the one that runs out, random/halton/sobol at nothing, tpe at what already happened. All but tpe derive their point from the seed and the index, so a machine that claimed trial 7 gets the same point without replaying six.

Partition is five schemes and not sklearn’s fifteen, because stratifying is a k-fold inside each class and grouping a k-fold over the groups; the rest are parameters. It is not called Split: somatize.torch.Split is already split learning.

Classes

Partition

Not constructed directly — use grouped, kfold, stratified, stratified_grouped, time_series.

somatize.study.Partition — how the samples are cut into folds.

Constructors

Partition.grouped

Partition.grouped(k)

k folds where all the samples of a group land on the same side. Needs groups=, and takes no seed: it places the biggest groups first, which is what keeps the folds comparable.

Partition.kfold

Partition.kfold(k, *, shuffle=None)

k folds over the samples, each held out in turn. shuffle is the seed, and without one the order they came in is kept.

Partition.stratified

Partition.stratified(k, *, shuffle=None)

k folds where every class keeps the share it has in the whole. Needs classes=.

Partition.stratified_grouped

Partition.stratified_grouped(k)

Groups whole, and among the ways of doing that the one leaving the classes most even. Needs both.

Partition.time_series

Partition.time_series(k, *, gap=0)

k growing prefixes, so nothing is ever trained on its own future. gap drops that many samples between the two sides.

Methods

Partition.folds

Partition.folds(n, *, classes=None, groups=None)

The folds as (train, test) pairs of indices — sklearn’s shape, so a loop written against KFold().split() reads the same.

classes= and groups= are one small integer per sample. They are numbers, not labels: turn y into them where y already is, with .tolist() or a dictionary, and no tensor crosses.

Properties

Partition.k

How many folds it produces, without producing them.

Point

Not constructed directly — handed to you.

somatize.study.Point — one configuration.

It behaves as a mapping, so build(**point) and point["lr"] both work, and str(point) is the trial’s name — derived from the values in the space’s order, so two machines that never spoke file it identically.

Point.items

Point.items()

Both, paired.

Point.keys

Point.keys()

The knobs, in the space’s order. What makes **point work.

Point.values

Point.values()

The values, in the same order.

Pruner

Not constructed directly — use diverged, median, patience, percentile, threshold.

somatize.study.Pruner — whether a trial that is going badly is worth another epoch.

Constructors

Pruner.diverged

Pruner.diverged()

Only what blew up: no bounds, so nothing goes but a loss that is not a number.

Pruner.median

Pruner.median(*, goal='min', warmup=0, startup=1)

Prune what is behind the median of the trials that already finished. Pruner.percentile(50, …) with a name on it.

Pruner.patience

Pruner.patience(steps, *, min_delta=0.0, goal='min')

Prune what has stopped improving on its own best — early stopping. steps cannot be zero.

Pruner.percentile

Pruner.percentile(p, *, goal='min', warmup=0, startup=1)

The same with the share that survives said out loud: smaller prunes more. At 50 the better half stays.

Pruner.threshold

Pruner.threshold(*, lower=None, upper=None)

Prune what leaves bounds you already know are hopeless. The only scheme that needs no other trial, so it works on the very first.

Methods

Pruner.verdict

Pruner.verdict(mine, others=None)

Why this trial is not worth another epoch, or None to carry on.

mine is what it has reported so far, in order; others the same for the trials that already finished. A “step” is the n-th report, so trials have to report on the same schedule for the comparison across them to mean anything.

Nothing is stopped here. You stop calling the trainer:

if why := pruner.verdict(reported, finished):
    break

Sampler

Not constructed directly — use grid, halton, random, sobol, tpe.

somatize.study.Sampler — where to look for the next configuration.

Constructors

Sampler.grid

Sampler.grid(steps=5)

Every combination, then nothing. steps says how finely a continuous knob is cut; an int narrower than that is taken whole.

Sampler.halton

Sampler.halton(*, seed=0)

Cover the space evenly instead of drawing from it evenly, one prime per knob.

A uniform draw is even in expectation: nothing stops the next two trials from landing on top of each other, it is only unlikely. This is even for every prefix, which is what a study handed out of a shared folder wants — two machines taking different numbers do not collide, and not because collision is improbable.

Its cover thins once there are many knobs, and it has no ceiling.

Sampler.random

Sampler.random(*, seed=0)

Uniform in every knob, looking at nothing else — and over a space where only a few knobs matter, that beats a grid on the same budget.

Sampler.sobol

Sampler.sobol(*, seed=0)

The same, without the seam — and with a ceiling of 32 knobs.

Every knob is read in base two and told apart by a table of direction numbers (Joe and Kuo, 2008), so nothing thins out as the knobs grow. Past what the table reaches, ask answers None from the very first trial.

Sampler.tpe

Sampler.tpe(*, goal='min', startup=10, candidates=24, quantile=0.25, seed=0)

Guided by what already worked: model the good trials, model the bad ones, and propose where the first is likely and the second is not.

Random until startup trials have finished. Unlike the other two its answer depends on what the asking machine has seen, which is what being guided means.

Methods

Sampler.ask

Sampler.ask(space, trial, seen=None)

Where to look for the trial-th time, or None when there is nowhere left — which is a grid saying it is done, and how a for stops without being told a number.

seen is (point, score) for the places somebody has already been. A score of None means the trial is still running — another machine is trying it and nobody knows yet how it will do, which is what in_flight gives back. Four of the five schemes ignore the whole argument, and that is the point of having five.

Sampler.total

Sampler.total(space)

How many combinations there are, for the one scheme that has an answer — None for the two that never run out. What a range() wants.

Space

Space()

somatize.study.Space — what is being searched over.

Built up, and every call gives back a new space: the one you had is still the one you had, which is what makes handing the same base to two studies safe.

Space.choice

Space.choice(name, options)

A knob that is one of these, by name.

Space.int

Space.int(name, low, high)

A knob that is a whole number between the two, both included.

Space.names

Space.names()

The knobs, in declaration order.

Space.read

Space.read(said)

The point that text names, read against these knobs.

The other half of str(point), and it needs the space in front of it: batch=64 on its own does not say whether 64 is a whole number or an option spelt "64".

It is what makes a study’s history come back in one scan of the shared folder: a trial keeps its configuration as text next to its score, so nothing has to be fetched to know where it looked.

Space.real

Space.real(name, low, high, *, log=False)

A knob that is anything between the two.

log=True draws it evenly in the logarithm, which is the only sane way to search a learning rate: drawn linearly, four fifths of 1e-5..1e-1 sits above 0.02.

Functions

abandoned

abandoned(store: Store, *, study: str, stale: float = 3600.0) -> list[tuple[int, int]]

Which trials have stopped moving, as (trial, attempt) pairs.

It decides nothing: whether a quiet trial is dead, preempted or on a very long epoch is not something a folder can tell. So this reports and the loop chooses:

for trial, attempt in abandoned(store, study=STUDY):
    take(store, point, study=STUDY, trial=trial, me=me, attempt=attempt + 1)

Taking the next attempt rather than writing over the old record, for the same reason claim uses a link. Being wrong is cheap: too eager is a trial run twice, and a claim still cannot collide.

coordinates

coordinates(store: Store, space: Space, *, study: str, goal: str | None = None) -> Figure

Every finished trial as a curve across the knobs, coloured by score. The one picture that shows a region of the space rather than one knob at a time.

Drawn by hand out of splines rather than with plotly’s Parcoords, which only draws straight segments. That costs its brushing and buys a trial reading as one continuous thing. A curve is an interpolation between axes, where there is nothing to be wrong about — a point exists only where it crosses an axis — and still drawn gently, because a curve bulging past the top of an axis reads as a value beyond its range.

No colour scale beside it: the score is the last axis. Unlike table there is no fallback, so a study that recorded no direction and a caller that names none is an error rather than a guess.

curves

curves(store: Store, *, study: str) -> list[list[float]]

The reports of every trial that ran to the end — what a Pruner wants. The reader that pays: a curve grows, so it lives in the blob and this is a scan plus one fetch per trial.

direction

direction(store: Store, *, study: str) -> str | None

Which way is better in this study — "min", "max", or None. One scan and no fetches.

None means no trial said, and it is the honest answer rather than "min". When records disagree the newest wins — the direction is what the person running the study currently means. Ties break by the higher trial number, because a study writes its first records inside the same second.

finished

finished(store: Store, space: Space, *, study: str) -> list[tuple[Point, float]]

Every trial that ran to the end, as (point, score) — what ask wants, in one scan and no fetches.

Pruned trials are left out on purpose: a pruned score is real but was measured after fewer epochs, so a sampler that treats it as a bad configuration learns something untrue.

importance

importance(store: Store, space: Space, *, study: str) -> list[tuple[str, float]]

How decisive each knob was, as (name, |rho|), biggest first.

Spearman’s rho: how well the score follows a knob monotonically, without assuming a shape. Ranks rather than values, so a knob searched in log needs no special case, and only the trials that ran to the end.

A categorical knob is ranked by its own options in order, which is honest for two and thin beyond that — answered anyway, because leaving it out would be this deciding what you may look at. 0.0 where a knob never varied: no evidence, which is not no effect.

in_flight

in_flight(store: Store, space: Space, *, study: str, stale: float = 3600.0) -> list[tuple[Point, float | None]]

The trials another machine is holding, each with no score.

Hand these to a sampler beside finished and a guided one stops proposing next to what somebody else is already trying:

point = sampler.ask(space, trial,
                    finished(store, space, study=STUDY)
                    + in_flight(store, space, study=STUDY))

That is constant liar (Ginsbourger, Le Riche and Carraro, 2010) without the lie, and the difference was measured. Handing the sampler a made-up bad score backfires: Tpe sizes the pile it imitates as a share of everything it is handed, so one more point raises the quota and can promote a trial out of the bad pile — one proposal in two hundred landed on the occupied region without it, thirty-nine with it. So None says running and does not vote.

One scan and no fetches. stale is how far behind a running trial may fall before it counts as stopped, measured against the newest write in this study and not against this machine’s clock — two machines sharing a folder are two clocks that disagree by minutes.

influence

influence(store: Store, space: Space, *, study: str) -> Figure

How decisive each knob was: |rho| against the score, biggest first.

A rank correlation, so it says this knob orders the results and not this knob is worth these many points.

report

report(store: Store, point: Point, reports: Sequence[float], *, study: str, trial: int, me: object, attempt: int = 0, state: str = 'running', score: float | None = None, because: str | None = None, took: float | None = None, goal: str | None = None) -> None

Writes down where this trial has got to, as often as there is something to say — once an epoch makes a curve watchable from another machine while it is still being drawn.

Only the machine that claimed it writes, and nothing has to enforce that: nobody else could have got the claim. goal goes beside the score because a score without it is not readable by anybody without this script.

table

table(store: Store, space: Space, *, study: str, goal: str | None = None) -> Figure

Every scored trial, best first, with the configuration that got it. One scan and no fetches. Pruned trials are here too and say so.

Best needs a direction, which comes from the record; goal overrides it. When neither says, the trials come back in the order they were run and the title says so — a table headed best first sorted the wrong way round is worse than one that is not sorted.

take

take(store: Store, point: Point, *, study: str, trial: int, me: object, attempt: int = 0, goal: str | None = None) -> bool

Claims the trial-th trial of study. True when it is this machine’s; False means somebody else got there first and the loop goes on to the next number. goal is written here as well as on every report, so a trial claimed and never reported still says which way it was looking.

trials

trials(store: Store, space: Space, *, study: str) -> list[Trial]

Every trial of this study, whatever state it is in, as records. The one for looking rather than deciding: what is still running, and whether the study is done.

Constants

• DONE — 'done'
• FAILED — 'failed'
• MAX — 'max'
• MIN — 'min'
• PRUNED — 'pruned'
• RUNNING — 'running'
• STALE — 3600.0