Are Sentence Transformer embeddings context-dependent for words, and how do they handle words with multiple meanings (polysemy)?

Sentence Transformer embeddings are context-dependent for words, though their approach differs from token-level models like BERT. Unlike static embeddings (e.g., Word2Vec), which assign a fixed vector to each word regardless of context, Sentence Transformers generate embeddings by processing entire sentences through transformer-based architectures. These models use attention mechanisms to weigh the importance of surrounding words, allowing the meaning of a word to adapt based on its context. For example, the word “bank” in “river bank” versus “bank account” will influence the sentence embedding differently because the model analyzes how “bank” interacts with adjacent terms. However, since Sentence Transformers output sentence-level embeddings (not individual word vectors), the context-awareness is reflected in the aggregated representation of the entire input.

To handle polysemy (words with multiple meanings), Sentence Transformers rely on their ability to capture contextual relationships within a sentence. When a word like “bat” (which could mean an animal or a sports tool) appears, the model examines the sentence structure and surrounding words to infer the correct meaning. For instance, in “The bat flew out of the cave,” the presence of “flew” and “cave” signals the animal meaning, while “He swung the bat hard” associates “bat” with baseball. The transformer’s attention heads assign higher weights to relevant context words, adjusting the sentence embedding to reflect the intended meaning. This contrasts with static embeddings, which would represent “bat” identically in both cases, leading to ambiguous representations.

The effectiveness of Sentence Transformers in handling polysemy stems from their training process. They are often fine-tuned with objectives like contrastive learning, where the model learns to distinguish between similar and dissimilar sentences. For example, during training, pairs like ("She opened a bank account", “The fish swam near the river bank”) are pushed apart in embedding space, teaching the model to differentiate between “bank” meanings. Additionally, techniques like mean pooling of token embeddings (common in Sentence Transformers) aggregate contextually enriched representations of all words, ensuring polysemous terms contribute appropriately to the final sentence vector. This makes the embeddings robust to word ambiguity while retaining semantic coherence.