Configure Backbone Network
The backbone network refers to the feature extractor that precedes the output heads, i.e. a neural network whose head is truncated so it outputs a feature vector. In continual learning, the backbone is typically shared among tasks. However, in some CL approaches (particularly architecture-based approaches), the backbone is dynamic: it can expand, incorporate additional mechanisms like masks, or even assign different networks to each task. If you are not familiar with continual learning backbones, feel free to learn more in my beginnersβ guide: backbone network and architecture-based approaches.
In multi-task learning, the backbone is always shared among tasks. In single-task learning, the backbone is simply a feature extractor.
The backbone is a sub-config under the index config of:
- Continual learning main experiment
- Continual learning full experiment and the reference experiments
- Continual unlearning main experiment
- Continual unlearning full experiment and the reference experiments
- Multi-task learning experiment
- Single-task learning experiment
To configure a custom backbone, create a YAML file in the backbone/
folder. Below is an example of the backbone config.
Example
configs
βββ __init__.py
βββ entrance.yaml
βββ index
β βββ example_cl_main_expr.yaml
β βββ ...
βββ backbone
β βββ clmlp.yaml
...
example_configs/index/example_cl_main_expr.yaml
defaults:
...
- /backbone: clmlp.yaml
...
example_configs/backbone/clmlp.yaml
_target_: clarena.backbones.CLMLP
input_dim: 784
hidden_dims: [256, 100]
output_dim: 64
batch_normalization: true
Supported Backbones & Required Config Fields
In CLArena, we have implemented many backbone networks as Python classes in the clarena.backbones
module that you can use for your experiments.
To choose a backbone, assign the _target_
field to the class name of the backbone. For example, to use CLMLP
, set the _target_
field to clarena.backbones.CLMLP
. Each backbone has its own hyperparameters and configurations, which means it has its own required fields. The required fields are the same as the arguments of the class specified by _target_
. The arguments for each backbone class can be found in the API documentation.
Below is the full list of supported backbone networks. Note that the names in the βBackboneβ column are the exact class names that you should assign to _target_
.
Multi-Task and Single-Task Learning Backbones
These backbones are direct implementations of neural network backbones, which can be generally used in multi-task and single-task learning unless noted otherwise.
Backbone | Description | Required Config Fields |
---|---|---|
MLP | Multi-layer Perceptron (MLP). MLP is a fully-connected network with all linear layers. The hidden dimension and number of layers are customizable | Same as MLP class arguments |
ResNet18, ResNet34, ResNet52, ResNet101, ResNet152 | Different versions of ResNet. ResNet is a convolutional network with residual connections | Same as ResNet18 class, ResNet34 class, ResNet52 class, ResNet101 class, ResNet152 class arguments |
Continual Learning Backbones
These backbones are implemented with addition features for continual learning, which can be generally used in continual learning and unlearning, unless noted otherwise.
Backbone | Description | Required Config Fields |
---|---|---|
CLMLP | Multi-layer Perceptron (MLP) for continual learning | Same as CLMLP class arguments |
CLResNet18, CLResNet34, CLResNet52, CLResNet101, CLResNet152 | ResNets for continual learning | Same as CLResNet18 class, CLResNet34 class, CLResNet52 class, CLResNet101 class, CLResNet152 class arguments |
For architecture-based continual learning, the backbone is specialized for the algorithm by introducing new mechanisms like masks. We provide specialized backbones for several architecture-based CL algorithms.
HAT Backbones
These backbones should be used with the CL algorithm HAT and its extensions AdaHAT, FGAdaHAT. Please refer to Configure CL Algorithm.
Specialized Backbone | Description | Required Config Fields |
---|---|---|
HATMaskMLP | Multi-layer Perceptron (MLP) for HAT | Same as HATMaskMLP class arguments |
HATMaskResNet18, HATMaskResNet34, HATMaskResNet52, HATMaskResNet101, HATMaskResNet152 | ResNets for HAT | Same as HATMaskResNet18 class, HATMaskResNet34 class, HATMaskResNet52 class, HATMaskResNet101 class, HATMaskResNet152 class arguments |
WSN Backbones
These backbones should be used with the CL algorithm WSN. Please refer to Configure CL Algorithm.
Specialized Backbone | Description | Required Config Fields |
---|---|---|
WSNMaskMLP | Multi-layer Perceptron (MLP) for WSN | Same as WSNMaskMLP class arguments |