Ranger: RAdam and LookAhead¶
Ranger combines RAdam and Lookahead together in one optimizer. RAdam fixes the adaptive learning rate's large variance during early stages of training to improve convergence and reducing the need for warmup. Lookahead updates model weights like normal, but every k steps interpolates them with a copy of slow moving weights. This moving average of the model weights is less sensitive to suboptimal hyperparameters and reduces the need for hyperparameter tuning.
Ranger was introduced by Less Wright in New Deep Learning Optimizer, Ranger: Synergistic combination of RAdam + Lookahead for the best of both.
Hyperparameters¶
Ranger works best with a flat learning rate followed by a short learning rate decay. Try raising the learning rate 2-3x larger than AdamW.
optimi sets the default \(\beta\)s to (0.9, 0.99)
and default \(\epsilon\) to 1e-6
. These values reflect current best-practices and usually outperform the PyTorch defaults.
optimi’s implementation of Ranger supports both decoupled weight decay decouple_wd=True
and fully decoupled weight decay decouple_lr=True
. Weight decay will likely need to be reduced when using fully decoupled weight decay as the learning rate will not modify the effective weight decay.
Ranger ¶
Ranger optimizer. RAdam with Lookahead.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
params |
Iterable[Tensor] | Iterable[dict]
|
Iterable of parameters to optimize or dicts defining parameter groups |
required |
lr |
float
|
Learning rate |
required |
betas |
tuple[float, float]
|
Coefficients for gradient and squared gradient moving averages (default: (0.9, 0.99)) |
(0.9, 0.99)
|
weight_decay |
float
|
Weight decay coefficient. If |
0
|
eps |
float
|
Added to denominator to improve numerical stability (default: 1e-6) |
1e-06
|
k |
int
|
Lookahead synchronization period (default: 6) |
6
|
alpha |
float
|
Lookahead weight interpolation coefficient (default: 0.5) |
0.5
|
decouple_wd |
bool
|
Apply decoupled weight decay instead of L2 penalty (default: True) |
True
|
decouple_lr |
bool
|
Apply fully decoupled weight decay instead of L2 penalty (default: False) |
False
|
max_lr |
float | None
|
Maximum scheduled learning rate. Set if |
None
|
kahan_sum |
bool | None
|
Enables Kahan summation for more accurate parameter updates when training in low precision (float16 or bfloat16). If unspecified, automatically applies for low precision parameters (default: None) |
None
|
foreach |
bool | None
|
Enables the foreach implementation. If unspecified, tries to use foreach over for-loop implementation since it is significantly faster (default: None) |
None
|
gradient_release |
bool
|
Fuses optimizer step and zero_grad as part of the parameter's backward
pass. Requires model hooks created with |
False
|
Algorithm¶
Ranger: RAdam and LookAhead.
optimi’s Ranger also supports L2 regularization and fully decoupled weight decay, which are not shown.