Configure CL Main Experiment

Modified

August 16, 2025

This section will guide you through configuring custom continual learning main experiments (CL Main). Please refer to Continual Learning (CL) for information about CL Main.

Prepare Configs

Experiments in CLArena are configured using YAML configuration files within a config folder. The configs/ directory must contain:

  • An entrance.yaml file, which serves as the entry point for the Hydra config system.
  • An experiment/ subfolder, which contains YAML files storing the experiment configurations. Each YAML file corresponds to a complete configuration for an experiment.

If you are unsure how to proceed, you can simply use the example configs and customize them. To run a custom experiment, you need to create a YAML file in the experiment/ subfolder.

Usage of clarena train clmain

The command clarena train clmain locates the config folder configs/, parses the configuration of the specified continual learning main experiment, and runs the experiment:

clarena train clmain experiment=<experiment-name>

Please ensure that the configs/ folder that meets the requirements above exists in the directory where you run the commands. The <experiment-name> corresponds to the path of the YAML file within the experiment/ subfolder. For example, if the YAML file til_pmnist_finetuning.yaml is located in the experiment/clmain_train/ subfolder, the <experiment-name> would be clmain_train/til_pmnist_finetuning.

Understanding Hierarchy of Configs

The experiment configs in experiment/ have a hierarchical structure, which is a feature of Hydra. We use example.yaml from the example configs as a demonstration:

example_configs/experiment/example_clmain_train.yaml
# @package _global_
# make sure to include the above commented global setting!

cl_paradigm: TIL
train_tasks: 10
eval_after_tasks: 10
global_seed: 1

defaults:
  - /cl_dataset: permuted_mnist.yaml
  - /backbone: clmlp.yaml
  - /cl_algorithm: finetuning.yaml
  - /optimizer: sgd.yaml
  - /lr_scheduler: reduce_lr_on_plateau.yaml
  - /trainer: cpu.yaml
  - /metrics: cl_default.yaml
  - /lightning_loggers: default.yaml
  - /callbacks: cl_default.yaml
  - /hydra: default.yaml
  - /misc: default.yaml

output_dir: outputs/example_clmain_train/${misc.timestamp}

# overrides
trainer:
  max_epochs: 2

Fields represented as <field>: <value> in the YAML file include:

  • Single entries assigned with values such as integers, strings, etc. For example, train_tasks: 10.
  • References to other YAML files (in the defaults list). For example, /cl_dataset: permuted_mnist.yaml refers to the YAML file permuted_mnist.yaml in the cl_dataset/ subfolder. We call these sub-configs. Please note that the field name must match the subfolder name that contains the referenced YAML file.
example_configs/cl_dataset/permuted_mnist.yaml
_target_: clarena.cl_datasets.PermutedMNIST
root: data/MNIST
num_tasks: 10
validation_percentage: 0.1
batch_size: 128
permutation_mode: first_channel_only

The fields in sub-configs can also include these two types, where the latter can refer to sub-sub-configs. This creates a hierarchical structure of configs, which results in the organization of YAML files in a directory structure:

./configs
├── entrance.yaml
├── experiment
│   ├── example.yaml
│   └── ...
├── cl_dataset
│   ├── permuted_mnist.yaml
│   └── ...
├── cl_algorithm
│   ├── finetuning.yaml
│   └── ...
├── backbone
│   └── ...
├── optimizer
│   └── ...
├── lr_scheduler
│   └── ...
├── trainer
│   └── ...
├── metrics
│   └── ...
├── lightning_loggers
│   └── ...
├── callbacks
│   └── ...
├── hydra
│   └── ...
├── misc
│   └── ...

We have divided the full experiment configuration into the above different components, each of which is stored in a separate YAML file under a config category subfolder. The uppermost YAML files (except entrance.yaml) are those in experiment/, which we call experiment index configs. For details on each config category, please refer to the Experiment Index Config (CL Main) section.

Override

Hydra provides an override mechanism to override config fields at higher levels. There are two approaches:

  1. Override in higher level YAML files: For example, we can find trainer: max_epochs: 2 in the example.yaml above. This overrides the value of the field max_epochs from the sub-config cpu.yaml in trainer/ with the new value 2.
  2. Override in command line: For example, the following command also overrides the value of the field max_epochs from the sub-config cpu.yaml in trainer/ with the new value 2.
clarena train clmain experiment=example trainer.max_epochs=2

Required Config Fields

To write our own experiment configs, we need to know which config fields are required. Essentially, this is determined by what is defined in the code, which parses the configs and passes them to the runtime. The experiment config must include all required fields for the experiment. Missing any required fields will trigger errors from our sanity check that ensures config integrity.

But don’t worry, we won’t make you figure it out through the code on your own :) All required fields are listed in the following Experiment Index Config (CL Main) section. For sub-configs, their required fields are also listed in the corresponding sections. Please check the next section first.

Back to top