What are hash-based embeddings?

Hash-based embeddings are a technique used to convert categorical data, such as words or IDs, into numerical vectors for machine learning models. Instead of assigning a unique embedding vector to every possible category (which requires storing a large lookup table), hash-based embeddings use a hash function to map categories directly to indices in a fixed-size embedding table. This approach reduces memory usage and handles unseen categories (out-of-vocabulary tokens) gracefully, as the hash function can map any input to a valid index. For example, the word “apple” might be hashed to index 42, while “orange” could be hashed to index 15, both within a predefined table size like 1000. This method is often called the “hashing trick.”

The process works by first defining an embedding table with a fixed number of slots (e.g., 1000). When a category (like a word) is encountered, a hash function (such as SHA-1 or a simpler non-cryptographic hash) computes a numerical value for it. This value is then mapped to an index within the table using modulo arithmetic. For instance, hashing “apple” might produce 1042, which modulo 1000 becomes index 42. The model then retrieves or updates the embedding vector at that index during training. A key trade-off is that multiple categories can collide to the same index, leading to shared embeddings. However, in practice, models often handle collisions effectively, especially when the embedding dimension is sufficiently high, allowing the network to learn robust representations despite overlaps.

Hash-based embeddings are particularly useful in scenarios with large or dynamic vocabularies, such as processing user-generated text or handling frequent new categories. For example, in a spam detection system where new email senders or domains constantly appear, hash-based embeddings avoid the need to rebuild the embedding table for unseen entries. Frameworks like TensorFlow and PyTorch support this via tools like tf.feature_column.hashed_column or custom hash layers. However, the fixed table size requires balancing: too small a table increases collision risks, while a very large table negates memory savings. Developers often experiment with table sizes and hash functions to optimize for their specific task, prioritizing efficiency when exact uniqueness isn’t critical.