IRPO

Official code for “Iterative Regularized Policy Optimization with Imperfect Demonstrations” (ICML2024).

Prepare python environment

conda create --name IRPO python=3.8
conda activate IRPO
conda install --file requirements.txt

Generate Demonstrations

Generate demonstrations with PID controller

Sample from $V\times U\times X=[v_{min}:v_{max}:v_{interval}]\times [{\mu }_{min}:{\mu }_{max}:{\mu }_{interval}]\times [{\chi }_{min}:{\chi }_{max}:{\chi }_{interval}]V\; \backslash times\; U\; \backslash times\; X\; =\; [v\_\{min\}:v\_\{max\}:v\_\{interval\}]\; \backslash times\; [\backslash mu\_\{min\}:\backslash mu\_\{max\}:\backslash mu\_\{interval\}]\; \backslash times\; [\backslash chi\_\{min\}:\backslash chi\_\{max\}:\backslash chi\_\{interval\}]$ with PID controller and save sampled trajectories in demonstrations/data/{step-frequence}hz_{$v_{interval}v\_\{interval\}$}_{${\mu }_{interval}\backslash mu\_\{interval\}$}_{${\chi }_{interval}\backslash chi\_\{interval\}$}_{data-dir-suffix}.

# sample trajectories by single-processing
python demonstrations/rollout_trajs/rollout_by_pid.py --data-dir-suffix v0 --step-frequence 10 --v-min 100 --v-max 300 --v-interval 10 --mu-min -85 --mu-max 85 --mu-interval 5 --chi-min -170 --chi-max 170 --chi-interval 5

# sample trajectories by multi-processing (base on Ray)
python demonstrations/rollout_trajs/rollout_by_pid_parallel.py --data-dir-suffix v1 --step-frequence 10 --v-min 100 --v-max 300 --v-interval 10 --mu-min -85 --mu-max 85 --mu-interval 5 --chi-min -170 --chi-max 170 --chi-interval 5

Update demonstrations with policy

Update demonstrations in {demos-dir} with policy in {policy-ckpt-dir}.

python demonstrations/rollout_trajs/rollout_by_policy_and_update_demostrations.py --policy-ckpt-dir checkpoints/sac_her/best_model --env-config-dir configs/env/env_config_for_ppo.json --demos-dir demonstrations/data/10hz_10_5_5_v1

Augment demonstrations

Augment trajectories based on $\chi \backslash chi$‘s symmetry.

python demonstrations/utils/augment_trajs.py --demos-dir demonstrations/data/10hz_10_5_5_v1

Train policies

Pre-train policy with Behavioral Cloning

Train policy by Behaviroal Cloning with config file in {config-file-name}

python train_scripts/train_with_bc_ppo.py --config-file-name configs/train/iteration_1/lambda_1e-3/ppo_bc_config_10hz_128_128_1.json

Train policy with PPO

Train policy by PPO with config file in {config-file-name}

python train_scripts/train_with_rl_ppo.py --config-file-name configs/train/iteration_1/lambda_1e-3/ppo_bc_config_10hz_128_128_1.json

Fine-tune a pre-trained policy with PPO

Fine-tune a pre-trained policy with PPO with config file in {config-file-name}

python train_scripts/train_with_rl_bc_ppo.py --config-file-name configs/train/iteration_1/lambda_1e-3/ppo_bc_config_10hz_128_128_1.json

More experiments

• Refer to exp_on_d4rl/ for experiments on HalfCheetah and Hopper.
• Refer to exp_on_panda/ for experiments on Reach.

Citation

@inproceedings{gong2024iterative,
  title={Iterative Regularized Policy Optimization with Imperfect Demonstrations},
  author={Xudong, Gong and Dawei, Feng and Kele, Xu and Yuanzhao, Zhai and ChengKang, Yao and Weijia, Wang and Bo, Ding and Huaimin, Wang},
  booktitle={International Conference on Machine Learning},
  year={2024},
  organization={PMLR}
}