Keras reduces the learning rate during neural network training using callbacks, which are utilities that adjust hyperparameters dynamically based on predefined rules or performance metrics. Two primary methods are the LearningRateScheduler and ReduceLROnPlateau callbacks. These tools allow developers to implement adaptive learning rate strategies without manual intervention, improving model convergence and training efficiency. The process is integrated into the training loop, ensuring adjustments occur automatically at specified intervals or when certain conditions are met.
The LearningRateScheduler callback lets developers define a custom function or a predefined schedule to adjust the learning rate at the start of each epoch. For example, a common strategy is step decay, where the learning rate is reduced by a fixed factor after a set number of epochs. A developer might write a function like def lr_step_decay(epoch): return initial_lr * 0.1 ** (epoch // 10), which cuts the learning rate by 90% every 10 epochs. This callback is straightforward to implement by passing the function to LearningRateScheduler and adding it to the callbacks list in model.fit(). It provides explicit control over the learning rate trajectory, making it ideal for scenarios where a predefined decay pattern is known to work well.
The ReduceLROnPlateau callback adapts the learning rate based on model performance during training. Instead of following a fixed schedule, it monitors a metric like validation loss and reduces the learning rate when improvements stall. For instance, if the validation loss hasn’t decreased for 5 epochs (patience=5), the callback multiplies the current learning rate by a factor (e.g., 0.5) until a minimum value (min_lr) is reached. This approach is useful when training dynamics are unpredictable, as it responds to actual model behavior rather than a fixed timeline. Developers configure it by specifying the monitored metric, reduction factor, patience, and bounds, ensuring the model doesn’t get stuck in suboptimal states due to an overly high or low learning rate. Both methods are flexible, require minimal code, and can significantly enhance training outcomes.