Configure Metrics (MTL)

Metrics are used to monitor training and validation process, and evaluate the model and algorithm during testing process.

Under the framework of PyTorch Lightning, callbacks are used to add additional actions and functionalities integrated in different timing of the experiment, which includes before, during, or after training, validating, or testing process. The metrics in our packages are implemented as metric callbacks, which can do:

• Calculate metrics and save their data to files.
• Visualize metrics as plots from the saved data.
• Log additional metrics during training process. (Note the majority of training metrics are handled by Lightning Loggers. See Configure Lightning Loggers section)

The details of the actions can be configured by the metric callbacks. Each group of metrics is organized as one metric callback, for example, MTLAccuracy and MTLLoss correspond to accuracy and loss metrics of multi-task learning. We can apply multiple metrics at the same time.

Metrics is a sub-config under the experiment index config (MTL). To configure custom metrics, you need to create a YAML file in metrics/ folder. At the moment, we only support uniform metrics across all tasks. Below shows examples of the metrics config.

Example

configs
├── __init__.py
├── entrance.yaml
├── experiment
│   ├── example_mtl_train.yaml
│   └── ...
├── metrics
│   ├── mtl_default.yaml
...

configs/experiment/example_mtl_train.yaml

defaults:
  ...
  - /metrics: mtl_default.yaml
  ...

The metrics config is a list of metric callback objects:

configs/metrics/mtl_default.yaml

- _target_: clarena.metrics.MTLAccuracy
  save_dir: ${output_dir}/results/
  test_acc_csv_name: acc.csv
  test_ave_acc_plot_name: ave_acc.png
- _target_: clarena.metrics.MTLLoss
  save_dir: ${output_dir}/results/
  test_loss_cls_csv_name: loss_cls.csv
  test_ave_loss_cls_plot_name: ave_loss_cls.png

Supported Metrics & Required Config Fields

In CLArena, we implemented many metric callbacks in clarena.metrics module that you can use for MTL experiment.

The _target_ field of each callback must be assigned to the corresponding class name, such as clarena.metrics.MTLAccuracy for MTLAccuracy. Each metric callback has its own required fields, which are the same as the arguments of the class specified by _target_. The arguments of each metric callback class can be found in API documentation.

API Reference (Metrics) Source Code (Metrics)

Below is the full list of supported metric callbacks. These callbacks can only be applied to MTL experiment. Note that the “Metric Callback” is exactly the class name that the _target_ field is assigned.

General

These metrics can be generally used unless noted otherwise.

Callback	Description	Required Config Fields
MTLAccuracy	Provides all actions that are related to MTL accuracy metric, which include: Defining, initializing and recording accuracy metric. Logging training and validation accuracy metric to Lightning loggers in real time. Saving test accuracy metric to files. Visualizing test accuracy metric as plots. The callback is able to produce the following outputs: CSV files for test accuracy of all tasks and average accuracy. Bar charts for test accuracy of all tasks.	Same as MTLAccuracy class arguments
MTLLoss	Provides all actions that are related to MTL loss metrics, which include: Defining, initializing and recording loss metrics. Logging training and validation loss metrics to Lightning loggers in real time. Saving test loss metrics to files. Visualizing test loss metrics as plots. The callback is able to produce the following outputs: CSV files for test classification loss of all tasks and average classification loss. Bar charts for test classification loss of all tasks.	Same as MTLLoss class arguments