Language Language English 简体中文

unilab.training.soak

Resource and progress monitoring for long-duration training soaks.

The monitor deliberately uses /proc and nvidia-smi instead of importing Torch: it must remain usable when the supervised trainer, its collector subprocesses, and their accelerator contexts are the only CUDA owners. The artifact is intended to be self-contained enough to diagnose a failed soak even after all child processes have exited.

Functions

run_soak(*, command, progress_dir, output_path)	Run a command in its own process group and produce a soak artifact.
runtime_snapshot(root)	Record source, GPU, software, and environment provenance.

Exceptions

SoakFailureError

Raised when a supervised soak violates a fail-closed condition.

exception unilab.training.soak.SoakFailureError

Bases: RuntimeError

Raised when a supervised soak violates a fail-closed condition.

unilab.training.soak.runtime_snapshot(root)

Record source, GPU, software, and environment provenance.

Parameters:

root (Path)

Return type:

dict[str, Any]

unilab.training.soak.run_soak(*, command, progress_dir, output_path, sample_interval_seconds=5.0, startup_timeout_seconds=600.0, stale_progress_seconds=120.0, post_shutdown_grace_seconds=5.0, min_duration_seconds=0.0, cwd=None, env=None, metadata=None)

Run a command in its own process group and produce a soak artifact.

progress_dir is an owner-controlled training log directory. File growth is used as a backend-independent liveness signal; once progress is first observed, no file change for stale_progress_seconds fails the soak rather than allowing a deadlocked worker to run forever.

Parameters:

• command (Sequence[str])

• progress_dir (Path)

• output_path (Path)

• sample_interval_seconds (float)

• startup_timeout_seconds (float)

• stale_progress_seconds (float)

• post_shutdown_grace_seconds (float)

• min_duration_seconds (float)

• cwd (Path | None)

• env (Mapping[str, str] | None)

• metadata (Mapping[str, Any] | None)

Return type:

dict[str, Any]