Output Results (CL)

This page summarizes the output results of continual learning experiment and evaluation pipelines. Their existence, file or folder names, and formats can be customized.

Output Results of Main Experiment and Evaluation

The following output results are produced after running continual learning main experiment or continual learning main evaluation.

Folder or File	Description	Customization
<output_dir>/	Contains all output results	<output_dir>: Field output_dir in the index config
<acc_save_dir>/	Contains all data and figures of test accuracy metrics	To include or exclude these outputs, enable or disable CLAccuracy metric callback (see Configure Metrics) <acc_save_dir>: Field save_dir in CLAccuracy. It is recommended to be set as ${output_dir}/results/ to make sure outputs are under <output_dir>/
<acc_save_dir>/acc.csv	The original data of test accuracy metrics, stored as CSV format. It contains the low triangular accuracy matrix and average accuracy. Please refer to my continual learning beginners’ guide	acc.csv: Field test_acc_csv_name in CLAccuracy. Default is acc.csv
<acc_save_dir>/acc_matrix.png	A figure of the low triangular accuracy matrix (see example here)	acc_matrix.png: Field test_acc_matrix_plot_name in CLAccuracy To exclude this output, set this field to null or remove this field
<acc_save_dir>/ave_acc.png	A curve plot of the average accuracy (see example here)	ave_acc.png: Field test_ave_acc_plot_name in CLAccuracy To exclude this output, set this field to null or remove this field
<loss_cls_save_dir>/	Contains all data and figures of test classification loss metrics	To include or exclude these outputs, enable or disable CLLoss metric callback (see Configure Metrics) <loss_cls_save_dir>: Field save_dir in CLLoss. It is recommended to be set as ${output_dir}/results/ to make sure outputs are under <output_dir>/
<loss_cls_save_dir>/loss_cls.csv	The original data of test classification loss metrics, stored as CSV format. It contains the low triangular accuracy matrix and average accuracy. Please refer to my continual learning beginners’ guide	loss_cls.csv: Field test_loss_cls_csv_name in CLLoss. Default is loss_cls.csv
<loss_cls_save_dir>/loss_cls_matrix.png	A figure of the low triangular classification loss matrix	loss_cls_matrix.png: Field test_loss_cls_csv_name in CLLoss To exclude this output, set this field to null or remove this field
<loss_cls_save_dir>/ave_loss_cls.png	A curve plot of the average classification loss	ave_loss_cls.png: Field test_loss_cls_csv_name in CLLoss To exclude this output, set this field to null or remove this field
<output_dir>/lightning_logs/csv/	Training and validation metrics logged to Lightning CSV logger in real time. Please refer to my article about CL metrics to learn about training and validation metrics in continual learning	To include or exclude these outputs, enable or disable Lightning CSV logger (see Configure Lightning Loggers)
<output_dir>/lightning_logs/tensorboard	Output files for Lightning TensorBoard logger	To include or exclude these outputs, enable or disable Lightning TensorBoard logger (see Configure Lightning Loggers)
<output_dir>/lightning_logs/wandb	Output files for Lightning Weights & Biases logger	To include or exclude these outputs, enable or disable Lightning Weights & Biases logger (see Configure Lightning Loggers)
<samples_save_dir>/	Contains input samples of first batch of each task	To include or exclude these outputs, enable or disable SaveFirstBatchImages callback (see Configure Callbacks) <samples_save_dir>: Field save_dir in SaveFirstBatchImages. It is recommended to be set as ${output_dir}/samples/ to make sure outputs are under <output_dir>/
<saved_models_dir>/	Contains saved model	To include or exclude these outputs, enable or disable SaveModels (see Configure Callbacks) <saved_models_dir>: Field save_dir in SaveModels. It is recommended to be set as ${output_dir}/saved_model/ to make sure outputs are under <output_dir>/
<profile_dir>/	Contains profiling results	To include or exclude these outputs, enable or disable profiler in Trainer (see Configure Trainer(s))
config_tree.log	Full copy of the experiment config in tree format	To include or exclude this output, set save as true or false in config tree config (see Other Configs) config_tree.log: Field save_path in config tree config. It is recommended to be set as ${output_dir}/ to make sure outputs are under <output_dir>/
console.log	Full logs of the console log in text format	To include or exclude this output, include or exclude the config field below in Hydra config (see Other Configs) console.log: the filename config in the Hydra config. It is recommended to be set as ${hydra.runtime.output_dir}/ to make sure outputs are under <output_dir>/ job_logging: handlers: file: filename: ${hydra.runtime.output_dir}/console.log

Output Results of Full Experiment and Evaluation

The following output results are produced after running continual learning full experiment or continual learning full evaluation, in addition to output results of main experiment and evaluation.

Folder or File	Description	Customization
<output_dir>/refjoint/	Contains all output results of the reference joint learning experiment	Cannot be customized
<output_dir>/refindependent/	Contains all output results of the reference independent learning experiment	Cannot be customized
<output_dir>/refrandom/	Contains all output results of the reference random learning experiment	Cannot be customized
<bwt_save_dir>/	Contains all data and figures of BWT metrics	<bwt_save_dir>: Field bwt_save_dir in the index config of full evaluation. Cannot be customized in full experiment. It is recommended to be set as ${output_dir}/results/ to make sure outputs are under <output_dir>/
<bwt_save_dir>/bwt.csv	The original data of BWT metrics, stored as CSV format. It contains BWT values after training each task. Please refer to my article about CL metrics.	bwt.csv: Field bwt_csv_name in the index config of full evaluation. Cannot be customized in full experiment. Default is bwt.csv
<bwt_save_dir>/bwt_plot.png	A curve plot of BWT.	bwt_plot.png: Field bwt_plot_name in the index config of full evaluation. Cannot be customized in full experiment To exclude this output, set this field to null or remove this field
<fwt_save_dir>/	Contains all data and figures of FWT metrics.	<fwt_save_dir>: Field fwt_save_dir in the index config of full evaluation. Cannot be customized in full experiment. It is recommended to be set as ${output_dir}/results/ to make sure outputs are under <output_dir>/
<fwt_save_dir>/fwt.csv	The original data of FWT metrics, stored as CSV format. It contains FWT values after training each task. Please refer to my article about CL metrics.	fwt.csv: Field fwt_csv_name in the index config of full evaluation. Cannot be customized in full experiment. Default is fwt.csv
<fwt_save_dir>/fwt_plot.png	A curve plot of FWT.	fwt_plot.png: Field fwt_plot_name in the index config of full evaluation. Cannot be customized in full experiment To exclude this output, set this field to null or remove this field
<fr_dir>/	Contains all data and figures of FR metrics.	<fr_save_dir>: Field fr_save_dir in the index config of full evaluation. Cannot be customized in full experiment. It is recommended to be set as ${output_dir}/results/ to make sure outputs are under <output_dir>/
<fr_dir>/fr.csv	The original data of FR metrics, stored as CSV format. Currently, it contains the FR value after training last task only, because joint learning of every combination of seen tasks costs too much. Please refer to my article about CL metrics.	fr.csv: Field fr_csv_name in the index config of full evaluation. Cannot be customized in full experiment. Default is fr.csv

Output Results of Specific Algorithms

Some algorithms can produce data and visualization results of their mechanism in addition to results above.

Hard Attention to the Task (HAT)

Hard Attention to the Task (HAT) is an architecture-based continual learning algorithm. It has some extensions, including AdaHAT, FG-AdaHAT. We support saving and visualizing masks and adjustment rate in the extensions. Please refer to Configure CL Algorithm.

Folder or File	Description	Customization
<masks_save_dir>/	Contains HAT masks	<masks_save_dir>: Field save_dir in metric callback HATMasks. It is recommended to be set as ${output_dir}/masks/ to make sure outputs are under <output_dir>/
<masks_save_dir>/test_masks/	Contains test masks of each task	test_masks/: Field test_masks_dir_name metric callback HATMasks To exclude this output, set this field to null or remove this field
<masks_save_dir>/test_cumulative_masks/	Contains test cumulative masks of each task	test_cumulative_masks/: Field test_cumulative_masks_dir_name metric callback HATMasks To exclude this output, set this field to null or remove this field
<masks_save_dir>/training_masks/	Contains training masks of every several steps of each task	training_masks/: Field training_masks_dir_name metric callback HATMasks To exclude this output, set this field to null or remove this field The frequency of logging training masks can be customized by plot_training_mask_every_n_steps in HATMasks
<adjustment_rate_save_dir>/	Contains adjustment rate plot every several steps	<masks_save_dir>: Field save_dir in metric callback HATAdjustmentRate. It is recommended to be set as ${output_dir}/adjustment_rates/ to make sure outputs are under <output_dir>/ The frequency of logging training masks can be customized by plot_adjustment_rate_every_n_steps in HATMasks.