Configure Callbacks (STL)
Under the PyTorch Lightning framework, we use callbacks to add additional actions at different points in the experiment, including before, during, or after training, validation, or testing.
Each callback type is designed for a specific set of actions, such as saving sample images or generating logging information. We can apply multiple callbacks to enable different sets of actions at the same time.
This section is only about configuring callbacks other than metric callbacks. To configure metric callbacks, please refer to Configure Metrics (CL Main) section.
Callbacks are a sub-config under the experiment index config (CL Main) and experiment index config (CL Main Eval). To configure custom callbacks, create a YAML file in the callbacks/ folder. At the moment, we only support a uniform callback setting across all tasks. Below is an example of the callbacks config.
Example
configs
├── __init__.py
├── entrance.yaml
├── experiment
│ ├── example_clmain_train.yaml
│ └── ...
├── callback
│ ├── cl_default.yaml
...configs/experiment/example_clmain_train.yaml
defaults:
...
- /callbacks: cl_default.yaml
...The callbacks config is a list of callback objects:
configs/callbacks/cl_default.yaml
- _target_: clarena.callbacks.CLPylogger
- _target_: clarena.callbacks.CLRichProgressBar
- _target_: clarena.callbacks.SaveFirstBatchImages
save_dir: ${output_dir}/samples/
- _target_: clarena.callbacks.SaveModels
save_dir: ${output_dir}/saved_models/Supported Callbacks & Required Config Fields
All Lightning built-in callbacks are supported. In CLArena, we have implemented many callbacks as Python classes in the clarena.callbacks module that you can use for your experiments.
To choose a callback, assign the _target_ field to the corresponding class name, such as lightning.pytorch.callbacks.EarlyStopping for EarlyStopping. Each callback has its own hyperparameters and configurations, which means it has its own required fields. The required fields are the same as the arguments of the class specified by _target_. The arguments for each callback class can be found in the PyTorch Lightning documentation and CLArena API documentation.
PyTorch Lightning Documentation (Built-In Callbacks) API Reference (Callbacks) Source Code (Callbacks)
Below is the full list of supported callbacks that can be applied to CL Main experiments. Note that the names in the “Callback” column are the exact class names that you should assign to _target_.
| Callback | Description | Required Config Fields |
|---|---|---|
| SaveModels | Saves the model at the end of each training task. Please refer to Save and Evaluate Model (CLMain) section. | Same as SaveModels class arguments |
| CLPylogger | Provides additional logging messages for during continual learning progress. For example, at the start and end of each training and testing task, log the task ID and other relevant information. | Same as CLPylogger class arguments |
| SaveFirstBatchImages | Saves images and labels of the first batch of training data into files. Applies to all tasks. | Same as SaveFirstBatchImages class arguments |
| CLRichProgressBar | Customised RichProgressBar for continual learning. |
Same as CLRichProgressBar class arguments |
The callbacks that can be applied to CL Main Eval experiment include CLPylogger and CLRichProgressBar.
Under the framework of PyTorch Lightning, we use callbacks to add additional actions and functionalities integrated in different timing of the experiment. This includes before, during, or after training, validating, or testing process.
Each type of callback is designed for one set of actions, such as save sample images, generating logging information, etc. we can apply multiple callbacks to enables different sets of actions at the same time.
This section is only about configuring callbacks other than metric callbacks. To configure metric callbacks, please refer to Configure Metrics (MTL) section.
Callbacks is a sub-config under the experiment index config (MTL). To configure custom callbacks, you need to create a YAML file in callbacks/ folder. At the moment, we only support uniform callbacks setting across all tasks. Below shows examples of the callbacks config.
Example
configs
├── __init__.py
├── entrance.yaml
├── experiment
│ ├── example_mtl_train.yaml
│ └── ...
├── callback
│ ├── mtl_default.yaml
...configs/experiment/example_mtl_train.yaml
defaults:
...
- /callbacks: mtl_default.yaml
...The callbacks config is a list of callback objects:
configs/callbacks/mtl_default.yaml
- _target_: clarena.callbacks.MTLPylogger
- _target_: clarena.callbacks.SaveFirstBatchImages
save_dir: ${output_dir}/samples/
- _target_: clarena.callbacks.SaveModels
save_dir: ${output_dir}/saved_models/Supported Callbacks & Required Config Fields
All Lightning built-in callbacks are supported. In CLArena, we also implemented many callbacks in clarena.callbacks module that you can use for your MTL experiment.
The _target_ field of each callback must be assigned to the corresponding class name, such as lightning.pytorch.callbacks.EarlyStopping for EarlyStopping. Each callback has its own required fields, which are the same as the arguments of the class specified by _target_. The arguments of each callback class can be found in PyTorch Lightning documentation and CLArena API documentation.
PyTorch Lightning Documentation (Built-In Callbacks) API Reference (Callbacks) Source Code (Callbacks)
Below is the full list of supported callbacks that can be applied to MTL experiment. Note that the “Callback” is exactly the class name that the _target_ field is assigned.
| Callback | Description | Required Config Fields |
|---|---|---|
| SaveModels | Saves the model at the end of training. Please refer to Save and Evaluate Model (MTLMain) section. | Same as SaveModels class arguments |
| MTLPylogger | Provides additional logging messages for during multi-task learning progress. | Same as MTLPylogger class arguments |
| SaveFirstBatchImages | Saves images and labels of the first batch of training data into files. | Same as SaveFirstBatchImages class arguments |
Under the framework of PyTorch Lightning, we use callbacks to add additional actions and functionalities integrated in different timing of the experiment. This includes before, during, or after training, validating, or testing process.
Each type of callback is designed for one set of actions, such as save sample images, generating logging information, etc. we can apply multiple callbacks to enables different sets of actions at the same time.
This section is only about configuring callbacks other than metric callbacks. To configure metric callbacks, please refer to Configure Metrics (STL) section.
Callbacks is a sub-config under the experiment index config (STL). To configure custom callbacks, you need to create a YAML file in callbacks/ folder. At the moment, we only support uniform callbacks setting across all tasks. Below shows examples of the callbacks config.
Example
configs
├── __init__.py
├── entrance.yaml
├── experiment
│ ├── example_stl_train.yaml
│ └── ...
├── callback
│ ├── stl_default.yaml
...configs/experiment/example_stl_train.yaml
defaults:
...
- /callbacks: stl_default.yaml
...The callbacks config is a list of callback objects:
configs/callbacks/stl_default.yaml
- _target_: clarena.callbacks.STLPylogger
- _target_: clarena.callbacks.SaveFirstBatchImages
save_dir: ${output_dir}/samples/
- _target_: clarena.callbacks.SaveModels
save_dir: ${output_dir}/saved_models/Supported Callbacks & Required Config Fields
All Lightning built-in callbacks are supported. In CLArena, we also implemented many callbacks in clarena.callbacks module that you can use for your STL experiment.
The _target_ field of each callback must be assigned to the corresponding class name, such as lightning.pytorch.callbacks.EarlyStopping for EarlyStopping. Each callback has its own required fields, which are the same as the arguments of the class specified by _target_. The arguments of each callback class can be found in PyTorch Lightning documentation and CLArena API documentation.
PyTorch Lightning Documentation (Built-In Callbacks) API Reference (Callbacks) Source Code (Callbacks)
Below is the full list of supported callbacks that can be applied to STL experiment. Note that the “Callback” is exactly the class name that the _target_ field is assigned.
| Callback | Description | Required Config Fields |
|---|---|---|
| SaveModels | Saves the model at the end of training. Please refer to Save and Evaluate Model (MTLMain) section. | Same as SaveModels class arguments |
| STLPylogger | Provides additional logging messages for during single-task learning progress. | Same as STLPylogger class arguments |
| SaveFirstBatchImages | Saves images and labels of the first batch of training data into files. | Same as SaveFirstBatchImages class arguments |