What techniques are used to reduce communication overhead in federated learning?

To reduce communication overhead in federated learning, three primary techniques are commonly used: model compression, structured updates with selective parameter sharing, and efficient aggregation methods combined with client sampling. These approaches address the challenge of transmitting large model updates across distributed devices while maintaining training efficiency.

Model compression reduces the size of data sent between clients and the server. Techniques like quantization (using fewer bits to represent numerical values) and pruning (removing less critical model parameters) shrink update sizes. For example, converting 32-bit floating-point numbers to 8-bit integers cuts data volume by 75%. Sparsification—sending only updates exceeding a threshold—further minimizes transfers. Google’s Federated Learning of Cohorts (FLoC) uses such methods to limit bandwidth usage on edge devices. These optimizations ensure fewer bytes are transmitted per communication round without significantly harming model accuracy.

Structured updates and selective parameter sharing focus on reducing how often and what parts of a model are communicated. Instead of sending full model updates, clients might transmit only parameters that changed meaningfully since the last round (e.g., via gradient masking). Alternatively, periodic averaging—updating the server every few local training steps—reduces communication frequency. For instance, a client might perform five local epochs before sending an update, cutting total rounds by 80%. This balances local computation and communication, especially useful in networks with latency constraints or data caps.

Efficient aggregation and client sampling lower the number of participants per round and streamline server-side processing. Sampling a subset of clients (e.g., 10% of devices) each round reduces overall data transfer. Federated Averaging (FedAvg) aggregates updates by computing weighted averages, avoiding redundant data transmission. Delta encoding—sending only changes from the previous model version—can also minimize payloads. For example, a smartphone might send only the differences in weights after local training, rather than the entire model. Combined with compression and structured updates, these methods enable scalable federated learning even in low-bandwidth environments.