unilab.algos.mlx.ppo.ppo.PPOConfig

class unilab.algos.mlx.ppo.ppo.PPOConfig[source]

Bases: object

PPOConfig(num_learning_epochs: ‘int’ = 4, num_mini_batches: ‘int’ = 4, clip_param: ‘float’ = 0.2, gamma: ‘float’ = 0.99, lam: ‘float’ = 0.95, value_loss_coef: ‘float’ = 0.5, entropy_coef: ‘float’ = 0.0, learning_rate: ‘float’ = 0.0003, use_clipped_value_loss: ‘bool’ = True, max_grad_norm: ‘float’ = 1.0, log_ratio_clip: ‘float’ = 20.0, schedule: ‘str’ = ‘fixed’, desired_kl: ‘float’ = 0.01, min_learning_rate: ‘float’ = 1e-05, max_learning_rate: ‘float’ = 0.01, normalize_advantage_per_mini_batch: ‘bool’ = False, adaptive_kl_beta: ‘float’ = 0.9, adaptive_lr_decay: ‘float’ = 1.5, adaptive_lr_growth: ‘float’ = 1.2, adaptive_lr_update_interval: ‘int’ = 1, target_kl_stop: ‘float | None’ = None, metrics_interval: ‘int’ = 8, finite_check_interval: ‘int’ = 8, enable_compile: ‘bool’ = False, warmup_strict_iters: ‘int’ = 0, warmup_metrics_interval: ‘int’ = 1, warmup_finite_check_interval: ‘int’ = 1, disable_finite_checks: ‘bool’ = False)

Parameters:

  • num_learning_epochs (int)

  • num_mini_batches (int)

  • clip_param (float)

  • gamma (float)

  • lam (float)

  • value_loss_coef (float)

  • entropy_coef (float)

  • learning_rate (float)

  • use_clipped_value_loss (bool)

  • max_grad_norm (float)

  • log_ratio_clip (float)

  • schedule (str)

  • desired_kl (float)

  • min_learning_rate (float)

  • max_learning_rate (float)

  • normalize_advantage_per_mini_batch (bool)

  • adaptive_kl_beta (float)

  • adaptive_lr_decay (float)

  • adaptive_lr_growth (float)

  • adaptive_lr_update_interval (int)

  • target_kl_stop (float | None)

  • metrics_interval (int)

  • finite_check_interval (int)

  • enable_compile (bool)

  • warmup_strict_iters (int)

  • warmup_metrics_interval (int)

  • warmup_finite_check_interval (int)

  • disable_finite_checks (bool)

Methods

__init__([num_learning_epochs, ...])

Attributes

adaptive_kl_beta

adaptive_lr_decay

adaptive_lr_growth

adaptive_lr_update_interval

clip_param

desired_kl

disable_finite_checks

enable_compile

entropy_coef

finite_check_interval

gamma

lam

learning_rate

log_ratio_clip

max_grad_norm

max_learning_rate

metrics_interval

min_learning_rate

normalize_advantage_per_mini_batch

num_learning_epochs

num_mini_batches

schedule

target_kl_stop

use_clipped_value_loss

value_loss_coef

warmup_finite_check_interval

warmup_metrics_interval

warmup_strict_iters

num_learning_epochs: int = 4
num_mini_batches: int = 4
clip_param: float = 0.2
gamma: float = 0.99
lam: float = 0.95
value_loss_coef: float = 0.5
entropy_coef: float = 0.0
learning_rate: float = 0.0003
use_clipped_value_loss: bool = True
max_grad_norm: float = 1.0
log_ratio_clip: float = 20.0
schedule: str = 'fixed'
desired_kl: float = 0.01
min_learning_rate: float = 1e-05
max_learning_rate: float = 0.01
normalize_advantage_per_mini_batch: bool = False
adaptive_kl_beta: float = 0.9
adaptive_lr_decay: float = 1.5
adaptive_lr_growth: float = 1.2
adaptive_lr_update_interval: int = 1
target_kl_stop: float | None = None
metrics_interval: int = 8
finite_check_interval: int = 8
enable_compile: bool = False
warmup_strict_iters: int = 0
warmup_metrics_interval: int = 1
warmup_finite_check_interval: int = 1
disable_finite_checks: bool = False
__init__(num_learning_epochs=4, num_mini_batches=4, clip_param=0.2, gamma=0.99, lam=0.95, value_loss_coef=0.5, entropy_coef=0.0, learning_rate=0.0003, use_clipped_value_loss=True, max_grad_norm=1.0, log_ratio_clip=20.0, schedule='fixed', desired_kl=0.01, min_learning_rate=1e-05, max_learning_rate=0.01, normalize_advantage_per_mini_batch=False, adaptive_kl_beta=0.9, adaptive_lr_decay=1.5, adaptive_lr_growth=1.2, adaptive_lr_update_interval=1, target_kl_stop=None, metrics_interval=8, finite_check_interval=8, enable_compile=False, warmup_strict_iters=0, warmup_metrics_interval=1, warmup_finite_check_interval=1, disable_finite_checks=False)
Parameters:

  • num_learning_epochs (int)

  • num_mini_batches (int)

  • clip_param (float)

  • gamma (float)

  • lam (float)

  • value_loss_coef (float)

  • entropy_coef (float)

  • learning_rate (float)

  • use_clipped_value_loss (bool)

  • max_grad_norm (float)

  • log_ratio_clip (float)

  • schedule (str)

  • desired_kl (float)

  • min_learning_rate (float)

  • max_learning_rate (float)

  • normalize_advantage_per_mini_batch (bool)

  • adaptive_kl_beta (float)

  • adaptive_lr_decay (float)

  • adaptive_lr_growth (float)

  • adaptive_lr_update_interval (int)

  • target_kl_stop (float | None)

  • metrics_interval (int)

  • finite_check_interval (int)

  • enable_compile (bool)

  • warmup_strict_iters (int)

  • warmup_metrics_interval (int)

  • warmup_finite_check_interval (int)

  • disable_finite_checks (bool)