Configure Your Experiments

To run your custom experiment other than the default, you need to create YAML configuration files for the experiment, as this package is using the YAML-based config system Hydra. You can put all configs in one YAML file then you can run the custom experiment by specifying the path and name of your entrance config file.

However, using a single YAML file to accommodate all configs for a continual learning experiment is very unmanagable. We strongly recommend using the hierarchical system like the default example does, and that’s what Hydra is meant for. As a starter, you can simply copy and custom your experiment on top of the the example configs.

1 Use Hierarchical Configs

The command cltrain example --config-dir <path/to/example_configs> to run the default experiment locates the path and name as entrance.yaml in the example configs. The directory contains a hierarchical structure of config files, and each sub-directory is an indivisible element of the CL system, which make it possible for different combinations of them. It is modulised, more organised and handy than single file.

./clarena/example_configs
β”œβ”€β”€ __init__.py
β”œβ”€β”€ entrance.yaml
β”œβ”€β”€ experiment
β”‚   β”œβ”€β”€ example.yaml
β”‚   └── ...
β”œβ”€β”€ cl_dataset
β”‚   β”œβ”€β”€ permuted_mnist.yaml
β”‚   └── ...
β”œβ”€β”€ cl_algorithm
β”‚   β”œβ”€β”€ finetuning.yaml
β”‚   └── ...
β”œβ”€β”€ backbone
β”‚   └── ...
β”œβ”€β”€ optimizer
β”œβ”€β”€ trainer
β”œβ”€β”€ lightning_loggers
β”œβ”€β”€ callbacks
β”œβ”€β”€ hydra
└── misc
example_configs/entrance.yaml
defaults:
  - experiment: null # must specify the experiment when running relative command

In the entrance config, We can literally find nothing but the experiment field. It is only an entrance (similar to β€œindex.html” in webpages), while the actual main thing is in the experiment sub-directory. For example, example_configs/experiment/example.yaml is the main config for the default experiment:

example_configs/experiment/example.yaml
# @package _global_
# make sure to include the above commented global setting!

cl_paradigm: TIL
num_tasks: 10

defaults:
  - /cl_dataset: permuted_mnist.yaml
  - /backbone: mlp.yaml
  - /cl_algorithm: finetuning.yaml
  - /optimizer: sgd.yaml
  - /trainer: cpu.yaml
  - /lightning_loggers: default.yaml
  - /callbacks: default.yaml
  - /hydra: default.yaml
  - /misc: default.yaml

test: True
output_dir_name: example # the name of output directory of this experiment

# overrides
trainer:
  max_epochs: 2
Note
  • __init__.py only works inside the package so you don’t have to include it in your external config directory.
  • Make sure to include the comment# @package _global_ at the first line of the experiment configs.

Now to run your custom experiment, you should specify the path and name of the entrance file (entrance.yaml in this case) to locate, and which experiment to run in the experiment sub-directory:

cltrain <your-experiment-name> --config-dir=</path/to/config/directory> --entrance=<your-entrance-config-name>

We set the default value --config-dir as configs/and --entrance as entrance, so if you store your configs right in the configs/ directory of your current path with an entrance.yaml, simple run:

cltrain <your-experiment-name>

2 Understand YAML Config Fields

We take the example configs as an example to understand what is included in YAML configs. There are many fields (entries) in YAML files, where each field is a β€œkey: value” pair. They can be loaded by Hydra into a config dict and then parsed by the source code. Generally, these fields fall into three types:

  • Single Entries: These fields contain basic values like integers or strings. E.g., num_tasks=10 in example_configs/experiment/example.yaml;
  • References to Other YAML Files: These fields link to other YAML files in sub-directories that contain collection of sub-configs. E.g., /cl_dataset: permuted_mnist.yamlin the defaults list in example_configs/experiment/example.yaml links to example_configs/cl_dataset/permuted_mnist.yaml;
  • Class Instantiators: some YAML files have the _target_ field at the beginning. The _target_ field specifies the class to instantiate from the source code, with arguments defined in the following fields. E,g. in the sub-config example_configs/cl_dataset/permuted_mnist.yaml, the _target_ field links to the classclarena.cl_dataset.PermutedMNIST in source codes and instantiates it. The following fields specifies the arguments of that class.
example_configs/cl_dataset/permuted_mnist.yaml
_target_: clarena.cl_datasets.PermutedMNIST
root: data/MNIST
num_tasks: 10
validation_percentage: 0.1
batch_size: 64
num_workers: 10

3 Write Your Experiment Config

To write your own create a new YAML file in example_configs/experiment/ and follow the rules below.

3.1 Required Config Fields

Your configuration must include all necessary fields for the experiment. Missing any required fields will trigger errors by our sanity check which ensures config integrity.

The required config fields for an experiment, as outlined in example_configs/experiment/example.yaml, include:

  • CL Paradigm: specifies the type of continual learning, either TIL (Task-Incremental Learning) or CIL (Class-Incremental Learning).
  • Number of Tasks: total number of tasks in the experiment.
  • Training/Testing Mode: specifies if it’s a test or training-only session.
  • Output Directory Name: the directory name of output directory in outputs/
  • Components:
  • Configs for Hydra: keep the default unless for special reasons.

3.2 Full Class Instantiation

In any sub-config involving class instantiation, all fields without default values must be explicitly defined to fully instantiate the class.

For example, the data parameter is required for the class PermutedMNIST, so data field must be included in example_configs/cl_dataset/permuted_mnist.yaml. Please refer to the documentation of each class to know what fields are required for the class.

There is one exception that optimizers can be left without the parameters specified, as we can’t do that in the config before the model is instantiated. In this case,_partial_: True field should be included (check example_configs/optimizer/sgd.yaml for example).

3.3 Better Organise through Overriding

The idea of hierarchize configs is to make the huge single-file config more readable. We try to divide the components of CL system as fine-grained as possible, but we don’t want the over-grained configs to make tons of sub-config categories which is unbelievably unmanagable, so we keep the sub-categories in the class-wise level.

Also, we don’t want to manage tons of config-files from the various choices and combinations of the class argument. So we highly recommend adhering to the rules that:

  • Keep as few as possible, ideally only one config for each class or for each major functionality;
  • The config should be set as default as possible;
  • Override in the experiment config, if you have different field values for it. This is a feature provided by Hydra. For example, trainer: max_epochs: 2 in the experiment example config overrides what is defined in /trainer: cpu.yaml.
Back to top