How are learning rates managed in federated learning?

In federated learning, learning rates are managed through a combination of server-side and client-side strategies to account for decentralized data and device heterogeneity. The server typically controls the global learning rate used during model aggregation, while clients may adjust their local learning rates during training. For example, in Federated Averaging (FedAvg), the server applies a fixed learning rate when combining client updates, while individual clients can use their own rates during local Stochastic Gradient Descent (SGD). Adaptive optimization methods like FedAdam extend this by adjusting the server’s learning rate dynamically based on historical update patterns, similar to centralized Adam optimization but tailored for federated settings.

One common approach is FedAvg, where the server aggregates model updates by averaging them, often using a learning rate of 1.0 (effectively direct averaging). Clients perform local training using SGD with a fixed or configurable learning rate. For instance, a client with more data might use a smaller rate to avoid overshooting minima, while a client with sparse data might use a larger rate. FedProx introduces a proximal term in the loss function to limit divergence between local and global models, indirectly influencing the effective learning rate by penalizing large updates. Adaptive server-side methods like FedAdam track gradient statistics across rounds to adjust the global rate, improving convergence in non-IID scenarios. For example, the server might scale updates inversely to their variance, reducing the impact of unstable client contributions.

Key challenges include handling non-IID data and varying client capabilities. Clients with skewed data distributions may require personalized local rates to avoid biased updates. Some frameworks decay the server’s global rate over rounds, mimicking centralized learning rate schedules. Others allow clients to set rates based on their dataset size—e.g., scaling by sample count. Communication constraints often favor simpler server-side strategies, as transmitting optimizer states (like Adam’s momentum terms) adds overhead. Practical implementations balance these factors: using fixed client rates for simplicity, adaptive server rates for robustness, and occasional client-specific tuning for heterogeneous environments. For example, a mobile device with limited compute might reduce its local rate to prevent noisy updates, while the server uses FedAdam to adaptively dampen inconsistent contributions.