unilab.training.soak.run_soak¶
- unilab.training.soak.run_soak(*, command, progress_dir, output_path, sample_interval_seconds=5.0, startup_timeout_seconds=600.0, stale_progress_seconds=120.0, post_shutdown_grace_seconds=5.0, min_duration_seconds=0.0, cwd=None, env=None, metadata=None)[source]¶
Run a command in its own process group and produce a soak artifact.
progress_diris an owner-controlled training log directory. File growth is used as a backend-independent liveness signal; once progress is first observed, no file change forstale_progress_secondsfails the soak rather than allowing a deadlocked worker to run forever.