Configure Callbacks
Under the PyTorch Lightning framework, we use callbacks to add additional actions at different points in the experiment or evaluation, including before, during, or after training, validation, or testing.
Each callback type is designed for a specific set of actions, such as saving sample images or generating logging information. We can apply multiple callbacks to enable different sets of actions at the same time.
This section is only about configuring callbacks other than metric callbacks. To configure metric callbacks, please refer to Configure Metrics section.
Callbacks are a sub-config under the index config of:
- Continual learning main experiment and evaluation
- Continual learning full experiment and the reference experiments
- Continual unlearning main experiment and evaluation
- Continual unlearning full experiment, the reference experiments and evaluation
- Multi-task learning experiment and evaluation
- Single-task learning experiment and evaluation
To configure custom callbacks, create a YAML file in the callbacks/
folder. Note that callbacks are shared in different continual learning tasks. Below is an example of the callbacks config.
Example
configs
βββ __init__.py
βββ entrance.yaml
βββ index
β βββ example_cl_main_expr.yaml
β βββ ...
βββ callbacks
β βββ cl_default.yaml
...
example_configs/index/example_cl_main_expr.yaml
defaults:
...
- /callbacks: cl_default.yaml
...
The callbacks config is a list of callback objects:
example_configs/callbacks/cl_default.yaml
- _target_: clarena.callbacks.CLPylogger
- _target_: clarena.callbacks.CLRichProgressBar
- _target_: clarena.callbacks.SaveFirstBatchImages
save_dir: ${output_dir}/samples/
- _target_: clarena.callbacks.SaveModels
save_dir: ${output_dir}/saved_models/
Supported Callbacks & Required Config Fields
All Lightning built-in callbacks are supported. In CLArena, we have also implemented many callbacks as Python classes in the clarena.callbacks
module that you can use for your experiments and evaluations.
To choose a callback, assign the _target_
field to the corresponding class name, such as lightning.pytorch.callbacks.EarlyStopping
for EarlyStopping
. Each callback has its own hyperparameters and configurations, which means it has its own required fields. The required fields are the same as the arguments of the class specified by _target_
. The arguments for each callback class can be found in the PyTorch Lightning documentation and CLArena API documentation.
PyTorch Lightning Documentation (Built-In Callbacks) API Reference (Callbacks) Source Code (Callbacks)
Below is the full list of supported callbacks. Note that the names in the βCallbackβ column are the exact class names that you should assign to _target_
.
Continual Learning Callbacks
These callbacks can be applied to continual learning and unlearning.
Callback | Description | Required Config Fields |
---|---|---|
SaveModels | Saves the model at the end of each training task. See Save and Evaluate Model (CL Main) | Same as SaveModels class arguments |
CLPylogger | Provides additional logging messages for during continual learning progress | Same as CLPylogger class arguments |
SaveFirstBatchImages | Saves images and labels of the first batch of training data into files. Applies to all tasks | Same as SaveFirstBatchImages class arguments |
CLRichProgressBar | Customised RichProgressBar for continual learning |
Same as CLRichProgressBar class arguments |
Multi-Task Learning Callbacks
These callbacks can be applied to multi-task learning.
Callback | Description | Required Config Fields |
---|---|---|
SaveModels | Saves the model at the end of training. See Save and Evaluate Model (MTL) | Same as SaveModels class arguments |
SaveFirstBatchImages | Saves images and labels of the first batch of training data into files | Same as SaveFirstBatchImages class arguments |
MTLPylogger | Provides additional logging messages for during multi-task learning progress | Same as MTLPylogger class arguments |
Single-Task Learning Callbacks
These callbacks can be applied to single-task learning.
Callback | Description | Required Config Fields |
---|---|---|
SaveModels | Saves the model at the end of training. See Save and Evaluate Model (STL) | Same as SaveModels class arguments |
STLPylogger | Provides additional logging messages for during single-task learning progress | Same as STLPylogger class arguments |
SaveFirstBatchImages | Saves images and labels of the first batch of training data into files | Same as SaveFirstBatchImages class arguments |