Configure Pipelines

CLArena supports four machine learning paradigms:

• Continual Learning (CL): Learning new tasks sequentially while retaining previous knowledge. See Continual Learning (CL) section.
• Continual Unlearning (CUL): Selectively forgetting tasks in continual learning scenarios. See Continual Unlearning (CUL) section.
• Multi-Task Learning (MTL): Learning multiple tasks simultaneously. See Multi-Task Learning (MTL) section.
• Single-Task Learning (STL): Traditional supervised learning for individual tasks. See Single-Task Learning (STL) section.

CLArena provides experiment and evaluation pipelines for each paradigm. Experiments are complete pipelines that include training and evaluation. Evaluations involve model evaluation for trained model. This section guides you through how to configure experiment and evaluation pipelines and run them in CLArena.

Prepare Configs

Eexperiment and evaluation pipelines in CLArena are configured using YAML configuration files within a config folder configs/. The configs/ directory must contain:

• An entrance.yaml file, which serves as the entry point for the Hydra config system.
• An index/ subfolder, which contains YAML files storing the pipeline configurations. Each YAML file corresponds to a complete configuration for an experiment or evaluation pipeline.

If you are unsure how to proceed, you can simply use the example configs and modify them. To run a custom pipeline, you need to create a YAML file in the index/ subfolder.

Usage of clarena

The command clarena locates the configs/ folder, parses the configuration of the specified experiment, and runs the experiment:

clarena pipeline=<pipeline-indicator> index=<index-config-name>

Ensure the configs/ folder meeting the above requirements exists in the directory where you run the command. The <pipeline-indicator> is a string that identifies the pipeline to run (see Supported Pipelines below). The <index-config-name> corresponds to the relative path of the YAML file within the index/ subfolder. For example, if the YAML file example_cl_main_expr.yaml is located in the index/ subfolder, then <index-config-name> is example_cl_main_expr.

Understanding Configuration Hierarchy

The experiment configs in index/ have a hierarchical structure, which is a feature of Hydra. We use example_cl_main_expr.yaml from the example configs as a demonstration:

configs/index/example_cl_main_expr.yaml

# @package _global_
# make sure to include the above commented global setting!

# experiment info
pipeline: CL_MAIN_EXPR
expr_name: example_cl_main_expr
output_dir: outputs/${expr_name}/${misc.timestamp}
global_seed: 1

# paradigm settings
cl_paradigm: TIL
train_tasks: 10
eval_after_tasks: 10 

# components
defaults:
  - /cl_dataset: permuted_mnist.yaml
  - /backbone: clmlp.yaml
  - /cl_algorithm: finetuning.yaml
  - /optimizer: adam.yaml
  - /lr_scheduler: reduce_lr_on_plateau.yaml
  - /trainer: cpu.yaml
  - /metrics: cl_default.yaml
  - /lightning_loggers: default.yaml
  - /callbacks: cl_default.yaml
  - /hydra: default.yaml
  - /misc: default.yaml

# overrides
trainer:
  max_epochs: 2

Fields represented as <field>: <value> in the YAML file include:

• Single entries assigned with values such as integers, strings, etc. For example, train_tasks: 10.
• References to other YAML files (in the defaults list). For example, /cl_dataset: permuted_mnist.yaml refers to the YAML file permuted_mnist.yaml in the cl_dataset/ subfolder. We call these sub-configs. The field name must match the subfolder name that contains the referenced YAML file.

example_configs/cl_dataset/permuted_mnist.yaml

_target_: clarena.cl_datasets.PermutedMNIST
root: data/MNIST
num_tasks: 10
validation_percentage: 0.1
batch_size: 128
permutation_mode: first_channel_only

Fields in sub-configs can also include these two types, where the latter can refer to sub-sub-configs. This creates a hierarchical structure of configs, resulting in the following directory organization:

./configs
├── entrance.yaml
├── index
│   ├── example_cl_main_expr.yaml
│   └── ...
├── cl_algorithm
│   ├── finetuning.yaml
│   └── ...
├── cul_algorithm
│   └── ...
├── mtl_algorithm
│   └── ...
├── stl_algorithm
│   └── ...
├── cl_dataset
│   ├── permuted_mnist.yaml
│   └── ...
├── mtl_dataset
│   └── ...
├── stl_dataset
│   └── ...
├── backbone
│   └── ...
├── optimizer
│   └── ...
├── lr_scheduler
│   └── ...
├── trainer
│   └── ...
├── metrics
│   └── ...
├── lightning_loggers
│   └── ...
├── callbacks
│   └── ...
├── hydra
│   └── ...
├── misc
│   └── ...

We divide the full configuration into the components above, each stored in a separate YAML file under a config category subfolder. The uppermost YAML files (except entrance.yaml) are in index/; we call them experiment index configs. For details on each config category, see Components.

Override

Hydra provides an override mechanism to override config fields at higher levels. There are two approaches:

1. Override in higher level YAML files: For example, we can find trainer: max_epochs: 2 in the example_cl_main_expr.yaml above. This overrides the value of the field max_epochs from the sub-config cpu.yaml in trainer/ with the new value 2.
2. Override in command line: For example, the following command also overrides the value of the field max_epochs from the sub-config cpu.yaml in trainer/ with the new value 2.

clarena pipeline=CL_MAIN_EXPR index=example_cl_main_expr trainer.max_epochs=2

Supported Pipelines & Required Config Fields

To write your own experiment configs, you need to know which config fields are required. Essentially, this is determined by what is defined in the code, which parses the configs and passes them to the runtime. The config must include all required fields for the pipeline. Missing any required fields will trigger errors from our sanity checks that ensure config integrity.

In CLArena, we have implemented many experiment and evaluation pipelines, all of which are listed below. Their required fields are listed in each pipeline documentation (link on their names). For sub-configs, their required fields are listed in Components.

Experiments

Below is the full list of experiments supported in CLArena.

Indicator	Experiment Pipeline	Required Config Fields
CL_MAIN_EXPR	Continual Learning Main Experiment	See here
CL_REF_JOINT_EXPR	Reference Joint Learning Experiment (Continual Learning)
CL_REF_INDEPENDENT_EXPR	Reference Independent Learning Experiment (Continual Learning)
CL_REF_RANDOM_EXPR	Reference Random Learning Experiment (Continual Learning)
CL_FULL_EXPR	Continual Learning Full Experiment
CUL_MAIN_EXPR	Continual Unlearning Main Experiment
CUL_REF_RETRAIN_EXPR	Reference Retrain Experiment (Continual Unlearning)
CUL_REF_ORIGINAL_EXPR	Reference Original Experiment (Continual Unlearning)
CUL_FULL_EXPR	Continual Unlearning Full Experiment
MTL_EXPR	Multi-Task Learning Experiment
STL_EXPR	Single-Task Learning Experiment

Evaluations

Below is the full list of evaluations supported in CLArena.

Indicator	Evaluation	Command
CL_MAIN_EVAL	Continual Learning Main Evaluation
CL_FULL_EVAL	Continual Learning Full Evaluation
CUL_MAIN_EVAL	Continual Unlearning Main Evaluation
CUL_FULL_EVAL	Continual Unlearning Full Evaluation
MTL_EVAL	Multi-Task Learning Evaluation
STL_EVAL	Single-Task Learning Evaluation