What algorithms are commonly used in federated learning?

Federated learning relies on algorithms designed to train machine learning models across decentralized devices or servers without sharing raw data. Three commonly used algorithms are Federated Averaging (FedAvg), Federated Stochastic Variance Reduced Gradient (FedSVRG), and Secure Aggregation protocols. These approaches address challenges like communication efficiency, data heterogeneity, and privacy.

Federated Averaging (FedAvg) is the foundational algorithm in federated learning. It works by having clients (e.g., mobile devices or servers) perform local training on their data using stochastic gradient descent (SGD) and sending model updates (e.g., weight changes) to a central server. The server averages these updates to create a global model, which is then redistributed to clients for further training. FedAvg reduces communication overhead by allowing multiple local training epochs before synchronization. For example, in mobile keyboard prediction, devices train locally on user typing data, and only model updates—not the data itself—are shared. However, FedAvg struggles with non-IID (non-identically distributed) data, where local datasets vary significantly between clients.

Privacy-focused algorithms like Secure Aggregation and Federated Learning with Differential Privacy (DP-Fed) enhance data security. Secure Aggregation uses cryptographic techniques to ensure the server cannot reconstruct individual client updates, protecting against data leakage. For instance, Google’s TensorFlow Federated implements this to aggregate encrypted model updates. DP-Fed adds noise to client updates during training, ensuring differential privacy guarantees. This is critical in healthcare applications, where patient data must remain confidential. However, these methods often trade off privacy for model accuracy, as excessive noise can degrade performance.

Optimization algorithms address challenges like system heterogeneity (e.g., varying device capabilities) and convergence speed. FedProx modifies FedAvg by introducing a proximal term in the local objective function, which limits how far local models can deviate from the global model. This improves stability when devices have uneven computational resources or data sizes. FedAdam adapts the Adam optimizer for federated settings, using adaptive learning rates during server-side aggregation to accelerate convergence. For example, in IoT networks with devices of varying compute power, FedAdam can dynamically adjust updates from slower or faster nodes. Another approach, Federated Multi-Task Learning, trains personalized models for each client while leveraging shared global patterns, useful in scenarios like personalized recommendations.

These algorithms are often implemented in frameworks like PySyft or Flower, allowing developers to experiment with trade-offs between communication, privacy, and accuracy. Choosing the right algorithm depends on the specific constraints of the application, such as data distribution, device capabilities, and privacy requirements.