What is the role of embeddings in federated learning?

Embeddings play a key role in federated learning by enabling models to learn from distributed data while preserving privacy. In federated learning, data remains on local devices (e.g., smartphones or edge servers), and only model updates—not raw data—are shared with a central server. Embeddings, which are compact numerical representations of data (like text, images, or user behavior), allow models to capture patterns without exposing sensitive details. For example, a keyboard app could use embeddings to represent typing patterns locally on a device. The server aggregates these embeddings to improve the global model, ensuring raw keystrokes are never transmitted.

A practical example is training a natural language processing (NLP) model across multiple devices. Each device might generate embeddings for text inputs (e.g., converting words to vectors) and send these embeddings—not the original text—to the server. The server then averages or combines the embeddings to update the global model. Similarly, in healthcare, a federated model could use embeddings of medical images (like MRI scans) to train a diagnostic tool without sharing patient data. This approach reduces privacy risks while maintaining the utility of the data for training.

However, embeddings in federated learning require careful design. Since embeddings can still leak information (e.g., via inversion attacks), techniques like differential privacy or secure aggregation are often applied to mask individual contributions. Additionally, embedding dimensions must be standardized across devices to ensure compatibility during aggregation. Frameworks like TensorFlow Federated or PyTorch’s FL tools handle these challenges by providing built-in methods for embedding alignment and privacy preservation. By balancing efficiency, privacy, and model performance, embeddings make federated learning feasible for real-world applications where data cannot be centralized.