What are embeddings in deep learning?

In deep learning, embeddings are a technique to represent discrete, high-dimensional data as continuous, lower-dimensional vectors. They map items like words, user IDs, or categories into a dense numerical format that captures relationships between the items. For example, the word “cat” might be represented as a 300-dimensional vector where similar words like “kitten” have vectors closer in this space. This compressed representation helps models process complex data more efficiently than raw one-hot encodings.

Embeddings are widely used in natural language processing (NLP) and recommendation systems. In NLP, word embeddings (e.g., Word2Vec, GloVe) convert words into vectors that reflect semantic meaning. For instance, the vectors for “king” and “queen” might be close in direction but differ along a “gender” axis. Similarly, recommendation systems use embeddings to represent users and items (e.g., movies or products). A user embedding might encode preferences like “likes action movies,” while a movie embedding could capture traits like “high-budget” or “sci-fi.” By computing similarity between these embeddings, the model can predict user-item interactions.

Technically, embeddings are learned during training. A neural network includes an embedding layer that starts with random vectors and adjusts them via backpropagation to minimize prediction errors. For example, in PyTorch, nn.Embedding(num_items, embedding_dim) creates a lookup table where each item ID maps to a trainable vector. The embedding dimension (e.g., 32, 64) is a hyperparameter: smaller sizes may lose information but reduce computational cost, while larger ones capture nuance but require more data. This approach transforms sparse, categorical inputs into dense representations that models can process effectively, improving both performance and efficiency.