How do Sentence Transformers create fixed-length sentence embeddings from transformer models like BERT or RoBERTa?

Sentence Transformers generate fixed-length sentence embeddings from models like BERT or RoBERTa by combining transformer-based token embeddings with pooling techniques and supervised training. These models address a key limitation of vanilla BERT or RoBERTa, which produce variable-length token-level outputs. To create a single fixed-dimensional vector for an entire sentence, Sentence Transformers apply pooling operations—such as averaging token embeddings—to aggregate information across all tokens. This process is enhanced by fine-tuning the model on sentence-pair tasks, which trains it to produce embeddings that capture semantic meaning effectively.

The technical workflow involves three steps. First, the input sentence passes through the transformer model (e.g., BERT), generating contextual embeddings for each token. These embeddings are influenced by the entire sentence context, thanks to the transformer’s self-attention mechanism. Next, a pooling layer condenses these token embeddings into a fixed-size vector. The most common method is mean pooling, where the embeddings of all tokens (excluding padding) are averaged. For example, for the sentence “Machine learning is fascinating,” the model averages the embeddings of all five tokens. Alternatively, max pooling or using the [CLS] token’s embedding (with additional fine-tuning) are options, though mean pooling is often preferred for its simplicity and effectiveness.

The critical innovation is the fine-tuning process. Unlike pretrained transformers optimized for token-level tasks, Sentence Transformers are trained on sentence pairs using contrastive or cosine similarity loss. For instance, during training, the model might receive pairs like ("How are you?", “How’s it going?”) labeled as similar and ("How are you?", “Weather forecast today”) as dissimilar. This teaches the model to adjust its embeddings so that semantically similar sentences cluster closer in vector space. The combination of transformer architecture, pooling, and task-specific training ensures the final embeddings retain semantic information while being fixed-length, enabling efficient comparison via cosine similarity or other metrics. This approach has proven effective for tasks like semantic search, clustering, and retrieval without requiring complex post-processing.