How is model accuracy evaluated in federated learning?

In federated learning, model accuracy is evaluated by aggregating performance metrics from decentralized clients while preserving data privacy. Each client computes accuracy locally using its own test data, and these results are combined by a central server to estimate global model performance. For example, after a training round, the server sends the updated model to all clients, who then test it on their local datasets (e.g., a reserved 20% of their data). The server calculates metrics like average accuracy or F1-score across clients to assess overall effectiveness. This approach avoids centralizing raw data but requires careful handling of variations in data distribution and client participation.

A key challenge is managing non-IID (non-independent and identically distributed) data across clients. For instance, in a federated healthcare scenario, one hospital might specialize in cancer data while another focuses on cardiovascular cases. If each evaluates the model only on their local test sets, the aggregated accuracy might mask poor performance on underrepresented conditions. To address this, some frameworks use stratified sampling or assign weights to client metrics based on dataset size. For example, a client with 10,000 samples could contribute more to the global accuracy score than one with 100 samples. Additionally, techniques like federated evaluation allow clients to compute metrics on a shared validation set stored securely on the server, though this requires careful privacy safeguards.

Practical implementation details matter significantly. Communication efficiency is prioritized—clients might send only summary statistics (e.g., confusion matrices) instead of raw predictions to reduce overhead. Privacy-preserving methods like differential privacy can be applied to aggregated metrics to prevent leakage about individual test samples. Tools such as TensorFlow Federated or PySyft provide built-in functions for federated evaluation, automating tasks like metric aggregation. For example, using TensorFlow Federated, developers can define a tff.learning.build_federated_evaluation function that iterates over client datasets, computes metrics, and averages results. However, developers must still validate that local test sets are representative and that clients participate consistently to avoid skewed results.