Configure Learning Rate Scheduler (MTL)

Modified

August 16, 2025

Learning rate scheduler is a component that adjusts the learning rate during training, which can help improve convergence and performance.

Learning rate scheduler is a sub-config under the experiment index config. To configure a custom learning rate scheduler, you need to create a YAML file in lr_scheduler/ folder. The learning rate scheduler is optional, which means you can apply no learning rate scheduler. Below shows an example of the learning rate scheduler config.

Example

configs
├── __init__.py
├── entrance.yaml
├── experiment
│   ├── example_mtl_train.yaml
│   └── ...
├── lr_scheduler
│   └── reduce_lr_on_plateau.yaml
...

configs/experiment/example_mtl_train.yaml

defaults:
  ...
  - /lr_scheduler: reduce_lr_on_plateau.yaml
  ...

configs/lr_scheduler/reduce_lr_on_plateau.yaml

_target_: torch.optim.lr_scheduler.ReduceLROnPlateau
_partial_: true # partially instantiate learning rate scheduler without 'optimizer' argument. Make sure this is included in any case!
mode: min
factor: 0.1
patience: 10

No Learning Rate Scheduler

If you don’t want to use a learning rate scheduler, you can set the field /lr_scheduler in the experiment index config to null, or simply remove this field.

defaults:
  ...
  - /lr_scheduler: null # or simply remove this field
  ...

Supported Learning Rate Schedulers & Required Config Fields

In CLArena, we didn’t implement our own learning rate schedulers, but rather use the built-in learning rate schedulers from PyTorch in torch.optim.lr_schedulermodule.

To choose a learning rate scheduler, assign the _target_ field to the class name of the scheduler. For example, to use ReduceLROnPlateau, set _target_ field to torch.optim.lr_scheduler.ReduceLROnPlateau. Meanwhile, include _partial: true as well (see below). Each learning rate scheduler has its own hyperparameters and configurations, which means it has its own required fields. The required fields are the same as the arguments of the class specified by _target_. The arguments of each learning rate scheduler class can be found in PyTorch documentation.

PyTorch Documentation (Built-In Learning Rate Schedulers)

Warning

Make sure to include field _partial_: True to enable partial instantiation. PyTorch learning rate scheduler need the optimizer as an argument to be fully instantiated, but we are now in the phase of configuration and certainly don’t have that argument, so the learning rate scheduler can be only partially instantiated.