Summary: Advice on Reinforcement Learning Experimentation

During the BeNeRL seminar talks researchers also briefly share their approach to RL experimentation. The below list keeps track of their main advice.
(BE = Benjamin Eysenbach, DP=Daniel Palenicek, AA=Ademi Adeniji, TD=Tal Daniel, HJ=Hojoon Lee)

Managing experiments

1. Maintain a lab/experiment journal (BE)

• Number every experiment you perform

• Rule of thumb: a paper requires in total ~200-250 experiments (unfruitful research directions will stop earlier)

2. Before each experiment, write down your hypotheses (BE)

• Determine what you want to get out of the next experiment

• Think beyond "does my method outperform the baseline?" --> there are many more (interesting) questions

3. Add reminders to yourself (BE)

• In your journal, add notes to yourself: when to check back to a certain experiment (with a date), what to look for, what to do next, etc. 

4. Change one thing at a time (DP)

• Don't change 5 things at once, since you will not know what broke your system. 

Interpreting experiments

1. Log as much data as possible (BE, DP, TD, HL)

• Don't only log learning curves. The more information you log, the more you can analyse

2. Dig deep into your results

• Analysis > Coding. Spend a significant part of your time on analyzing the output of an experiment (BE)

• Be curious, try to learn what is going on. Don't only look at learning curves. (BE)

• Use your debugger for careful inspection during training (note: jax.debug.breakpoint() ) (DP)

• Analyse sources of failures. (TD)

3. Visualize (BE, DP, AA)

• Try to visualize your results as much as possible

• Automate this, possibly through you own plotting pipeline, such as: https://github.com/danielpalen/wandb_plot

• Name your experiments thoughtfully with the relevant hyperparameters and use wandb to record command-line flags and save a code snapshot

4. Don't draw premature conclusions from single seeds (DP, AA)

• Have scripts ready to deploy you experiments to a cluster, ensure that you always run multiple seeds

• Unless debugging, always run experiments over multiple seeds, at least 4 

Implementations

1. Just use good implementations 

• Good impelementations can matter more than model-based vs model-free or on-policy vs off-policy, e.g. DreamerV3 requires minial tuning. (AA)

• Start simple with the cleanest code possible (ClearnRL, PufferLib). (TD)

2. For offline RL - policy extraction temperature and conservative regularization term matters (AA)

3. For model-based RL- first make sure your algorithm works with groud-truth state (TD)

4. If time permits, try to re-implement algorithms from scratch to understand the details (HL)

5. Iterate fast, utlize GPU-based simulators: IssacLab, BRAX / PufferLib (TD)

6. Think big, start small (HL)