Single-Task Learning Experiment

Single-task learning experiment is the experiment in CLArena for training and evaluating single-task learning algorithms. This section defines its pipeline and guides you through configuring custom experiments.

Experiment Pipeline

1. Training Phase

• The model is trained on the union of training sets ${\mathcal{D}}_{\text{train}}$ for the specified number of epochs
• The single-task learning algorithm’s mechanisms are applied during this phase

2. Validation Phase

• At the end of each training epoch, the model is validated on ${\mathcal{D}}_{\text{val}}$
• Validation results can optionally guide model selection and hyperparameter tuning

3. Test Phase

• After completing training and validation, the model is evaluated on the test dataset ${\mathcal{D}}_{\text{test}}$
• Note: This phase may be skipped based on configuration settings (see test in required config fields)

Running

To run a single-task learning experiment, specify the STL_EXPR indicator in the command:

clarena pipeline=STL_EXPR index=<index-config-name>

Configuration

To run a custom single-task learning experiment, create a YAML file in the index/ folder as index config. Below is an example.

Example

configs/index/example_stl_expr.yaml

# @package _global_
# make sure to include the above commented global setting!

# pipeline info
pipeline: STL_EXPR
expr_name: example_stl_expr
eval: true
global_seed: 1

# components
defaults:
  - /stl_dataset: stl_mnist.yaml
  - /stl_algorithm: single_learning.yaml
  - /backbone: mlp.yaml
  - /optimizer: adam.yaml
  - /lr_scheduler: reduce_lr_on_plateau.yaml
  - /trainer: cpu.yaml
  - /metrics: stl_default.yaml
  - /lightning_loggers: default.yaml
  - /callbacks: stl_default.yaml
  - /hydra: default.yaml
  - /misc: default.yaml

# outputs
output_dir: outputs/stl_mnist/${misc.timestamp}

Required Config Fields

Field	Description	Allowed Values
pipeline	The default pipeline that clarena use the config to run	Choose from supported pipeline indicators Only STL_EXPR is allowed
expr_name	The name of the experiment	A string value
eval	Whether to include evaluation phase	Boolean value
global_seed	The global seed for the entire experiment	Same as seed argument in lightning.seed_everything()
/stl_dataset	The single-task dataset on which the experiment is conducted	Choose from sub-config YAML files in stl_dataset/ folder See Configure STL Dataset
/stl_algorithm	The single-task learning algorithm	Choose from sub-config YAML files in stl_algorithm/ folder See Configure STL Algorithm
/backbone	The backbone network on which the single-task learning algorithm is based	Choose from sub-config YAML files in backbone/ folder See Configure Backbone Network
/optimizer	The optimizer for learning the single task	Choose from sub-config YAML files in optimizer/ folder See Configure Optimizer
/lr_scheduler	The learning rate scheduler for learning the single task	Choose from sub-config YAML files in lr_scheduler/ folder See Configure Learning Rate Scheduler
/trainer	The PyTorch Lightning Trainer object that contains all configs for training, validation and test process	Choose from sub-config YAML files in trainer/ folder See Configure Trainer
/metrics	The metrics to be monitored, logged or visualized	Choose from sub-config YAML files in metrics/ folder See Configure Metrics
/lightning_loggers	The Lightning Loggers used to log metrics and results	Choose from sub-config YAML files in lightning_loggers/ folder See Configure Lightning Loggers
/callbacks	The callbacks applied to this experiment (other than metric callbacks). Callbacks are additional actions integrated in different points during the experiment	Choose from sub-config YAML files in callbacks/ folder See Configure Callbacks
/hydra	Configuration for Hydra	Choose from sub-config YAML files in hydra/ folder See Other Configs
/misc	Miscellaneous configs that are less related to the experiment	Choose from sub-config YAML files in misc/ folder See Other Configs
output_dir	The folder storing the experiment results	Relative path to where you run the clarena command We recommend including experiment names and timestamps in the path to distinguish multiple runs, e.g. outputs/{expr_name}/${misc.timestamp}

Note

The single-task learning experiment is managed by a STLExperiment class. To learn how these fields work, please refer to its source code.