What are the computational overheads of federated learning?

Federated learning introduces computational overheads primarily related to on-device processing, communication costs, and server-side coordination. In this approach, each client device (like a smartphone or IoT sensor) trains a local machine learning model using its own data, which consumes local compute resources. For example, training even a moderately sized neural network on a mobile device can strain its CPU/GPU, memory, and battery life. Devices with limited hardware capabilities may struggle to complete training within reasonable time frames, especially if models are complex (e.g., ResNet-style architectures). Additionally, frequent model updates—such as transmitting thousands of gradient values—increase network usage, which can be costly for devices on metered connections or slow networks.

Server-side operations also contribute to overhead. The central server must aggregate updates from potentially millions of devices, which involves computational work proportional to the number of participants. Aggregation algorithms like Federated Averaging (FedAvg) require combining model parameters from diverse clients, which can become resource-intensive as model size scales. For instance, a server handling a 100MB model for 10,000 clients would process 1TB of data per round, demanding significant bandwidth and processing power. Additionally, the server must manage client selection, handle dropped or delayed participants, and enforce security measures like encryption or differential privacy. These steps add layers of computation, such as encrypting model updates or adding noise to gradients, which slow down the overall process.

Finally, federated learning introduces systemic inefficiencies. Client heterogeneity—variations in hardware, data distribution, and network stability—often forces the system to wait for slower devices, a problem known as the “straggler effect.” For example, a single outdated smartphone training a model on non-IID (non-uniform) data could delay synchronization across all participants. Techniques to mitigate this, such as partial client participation or asynchronous updates, may reduce training quality or require additional compute to reconcile inconsistent model versions. Moreover, repeated training rounds needed to achieve convergence (due to decentralized data) amplify these costs over time. A real-world example is a federated recommendation system requiring hundreds of rounds to stabilize, multiplying baseline compute and communication demands. These factors make optimizing resource usage a critical challenge in federated learning deployments.