Configure Optimizer (STL)
Optimizer is the component that manages the learning process by updating model parameters based on the computed gradients.
Optimizer is a sub-config under the experiment index config (STL). To configure a custom optimizer, you need to create a YAML file in optimizer/ folder. Below shows an example of the optimizer config.
Example
configs
├── __init__.py
├── entrance.yaml
├── experiment
│ ├── example_stl_train.yaml
│ └── ...
├── optimizer
│ └── sgd.yaml
...configs/experiment/example_stl_train.yaml
defaults:
...
- /optimizer: sgd.yaml
...configs/optimizer/sgd.yaml
_target_: torch.optim.SGD
_partial_: true # partially instantiate optimizer without 'params' argument. Make sure this is included in any case!
lr: 0.01
weight_decay: 0.0Supported Optimizers & Required Config Fields
In CLArena, we didn’t implement our own optimizers, but rather use the built-in optimizers from PyTorch in torch.optimmodule.
To choose an optimizer, assign the _target_ field to the class name of the optimizer. For example, to use SGD, set _target_ field to torch.optim.SGD. Meanwhile, include _partial: true as well (see below). Each optimizer has its own hyperparameters and configurations, which means it has its own required fields. The required fields are the same as the arguments of the class specified by _target_. The arguments of each optimizer class can be found in PyTorch documentation.
Make sure to include field _partial_: true to enable partial instantiation. PyTorch optimizer need the model parameters as an argument to be fully instantiated, but we are now in the phase of configuration and certainly don’t have that argument, so the optimizer can be only partially instantiated.