Runtime Model

The detailed runtime contract is in ADR-0001 Runtime Model And Layer Boundaries and Developer 指南. This page keeps the English summary close to the code paths.

Runtime Shapes

Synchronous PPO Paths

src/unilab/scripts/train_rsl_rl.py composes Hydra config, calls registry bootstrap, constructs the strict TorchEnv through registry.make(...), and runs the learner in the same process. The RSL-RL path adapts that tensor boundary through src/unilab/rl/.

Async APPO And Off-Policy Paths

APPO and off-policy runners use an environment-to-learner split:

Environment tensor loop -> shared IPC carrier -> learner
        ^                                      |
        +------------- SharedWeightSync -------+

  • APPO uses APPORunner, TensorRolloutRingBuffer, and SharedWeightSync.

  • SAC and FlashSAC use one off-policy execution path: ReplayBuffer provides bounded host ingress, the complete ring lives on one CUDA/MPS learner device, and SharedWeightSync publishes actor weights.

  • AsyncRunner in unilab.rl.runtime.async_runner owns collector process startup, stop signaling, and shared-resource cleanup.

Boundary Rules

  • The Manager-Based runtime returns TorchEnvState; CPU-authoritative physics backends cross their declared host-bridge H2D/D2H boundaries, while CUDA-native backends remain device-resident.

  • Trainer-only tensors such as optimizer state and rollout storage belong to learner code, not env code.

  • Collector/learner protocols must reuse the existing IPC primitives instead of creating ad-hoc parallel protocols in scripts.

Evidence In Repo

  • PPO entrypoint: src/unilab/scripts/train_rsl_rl.py

  • APPO runner: unilab.rl.algos.appo.runner

  • Off-policy runner: unilab.rl.runtime.offpolicy.double_buffer_runner

  • IPC primitives: unilab.rl.runtime.async_runner, unilab.rl.runtime.ipc.tensor_rollout_ring_buffer, unilab.rl.runtime.ipc.replay_buffer, unilab.rl.runtime.ipc.weight_sync