Configure Backbone Network

The backbone network acts as the feature extractor before the output heads. Typically, the backbone is shared and fixed, which is a neural network that truncates the head and output a feature. Hoewever, in some CL approaches (particularly in architecture-based approaches), the backbone is dynamic which can expand, incorporate additional mechanism like masks, or even assign different individual networks uniquely to each task. If you are not familiar with the network architecture involved in continual learning, feel free to get some knowledge from my CL beginner’s guide about backbone network and architecture-based approaches.

Configure Backbone Network

To configure the backbone neural network for your experiment, link the /backbone field in the experiment index config to a YAML file in backbone/ subfolder of your configs. That YAML file should use _target_ field to link to a CL dataset class (as shown in source code: clarena/backbones/) and specify its arguments in the following field. Here is an example:

./clarena/example_configs
├── __init__.py
├── entrance.yaml
├── experiment
│   ├── example.yaml
│   └── ...
├── backbone
│   └── mlp.yaml
...

example_configs/experiment/example.yaml

defaults:
  ...
  - /backbone: mlp.yaml
  ...

example_configs/backbone/mlp.yaml

_target_: clarena.backbones.MLP
input_dim: 784
hidden_dims: [256, 100]
output_dim: 64
batch_normalisation: True

Supported Backbone List

In this package we implemented many CL backbone classes in clarena.backbones module that you can use for your experiment. Below is the full list. Please refer to the API reference of each class to learn its required arguments.

API Reference (Backbones) Source Code

Backbone	Description
MLP	Fully-connected network, all linear layers. The hidden dimension and number of layers are customisable.
ResNet	Residual network, a very popular deep learning architecture. We provide ResNet-18, 34, 50, 101, 152.

We have also implemented HAT masked version for each backbone class. HAT (Hard Attention to the Task, 2018) is an architecture-based continual learning approach that uses learnable hard attention masks to select the task-specific parameters.

Backbone	Description
HATMaskMLP	HAT masked version of MLP.
HATMaskResNet	HAT masked version of ResNet. We provide HATMaskResNet-18, 34, 50, 101, 152.