Multi-Task Learning Experiment
Multi-task learning experiment is the experiment in CLArena for training and evaluating multi-task learning algorithms. This section defines its pipeline and guides you through configuring custom experiments.
Experiment Pipeline
For all tasks
- Training Phase
- The model is jointly trained on all tasks’ training data
for the specified number of epochs - The multi-task learning algorithm’s mechanisms are applied during this phase
- Model parameters are updated to maximize intertask knowledge transfer
- Validation Phase
- At the end of each training epoch, the model is validated on
- Validation results can optionally guide model selection and hyperparameter tuning
- Evaluation Phase
- After completing training and validation, the model is evaluated on the test datasets of all tasks:
(this is customizable, seeeval_tasks
in required config fields) - The evaluation assesses both task-specific and overall performance
Running
To run a multi-task learning experiment, specify the MTL_EXPR
indicator in the command:
clarena pipeline=MTL_EXPR index=<index-config-name>
Configuration
To run a custom multi-task learning experiment, create a YAML file in the index/
folder as index config. Below is an example.
Example
example_configs/index/example_mtl_expr.yaml
# @package _global_
# make sure to include the above commented global setting!
# pipeline info
pipeline: MTL_EXPR
expr_name: example_mtl_expr
train_tasks: 5
eval_tasks: 5
global_seed: 1
# components
defaults:
- /mtl_dataset: from_cl_split_mnist.yaml
- /mtl_algorithm: joint_learning.yaml
- /backbone: mlp.yaml
- /optimizer: adam.yaml
- /lr_scheduler: reduce_lr_on_plateau.yaml
- /trainer: cpu.yaml
- /metrics: mtl_default.yaml
- /lightning_loggers: default.yaml
- /callbacks: mtl_default.yaml
- /hydra: default.yaml
- /misc: default.yaml
# outputs
output_dir: outputs/${expr_name}/${misc.timestamp}
# overrides
trainer:
max_epochs: 2
Required Config Fields
Below is the list of required config fields for the index config of multi-task learning experiment.
Field | Description | Allowed Values |
---|---|---|
pipeline |
The default pipeline that clarena use the config to run |
|
expr_name |
The name of the experiment |
|
train_tasks |
The list of task IDs1 to train |
|
eval_tasks |
The list of task IDs to evaluate |
|
global_seed |
The global seed for the entire experiment |
|
/mtl_dataset |
The multi-task learning dataset on which the experiment is conducted |
|
/mtl_algorithm |
The multi-task learning algorithm |
|
/backbone |
The backbone network on which the multi-task learning algorithm is based |
|
/optimizer |
The optimizer for learning multiple tasks |
|
/lr_scheduler |
The learning rate scheduler for learning multiple tasks |
|
/trainer |
The PyTorch Lightning Trainer object that contains all configs for training, validation and test process |
|
/metrics |
The metrics to be monitored, logged or visualized |
|
/lightning_loggers |
The Lightning Loggers used to log metrics and results |
|
/callbacks |
The callbacks applied to this experiment (other than metric callbacks). Callbacks are additional actions integrated in different points during the experiment |
|
/hydra |
Configuration for Hydra |
|
/misc |
Miscellaneous configurations that are less related to the experiment |
|
output_dir |
The folder storing the experiment results |
|
The multi-task learning experiment is managed by a MTLExperiment
class. To learn how these fields work, please refer to its source code.
Footnotes
The task IDs are integers starting from 1, ending with number of tasks of the MTL dataset. Each corresponds to a task-specific dataset in the MTL dataset.↩︎