A typical Sentence Transformer model, such as Sentence-BERT (SBERT), is designed to generate dense vector representations (embeddings) for sentences or short texts. The architecture builds on pre-trained transformer models like BERT but modifies them to produce sentence-level embeddings efficiently. Instead of using the raw output of BERT, which processes text at the token level, SBERT adds a pooling layer after the transformer to aggregate token embeddings into a fixed-size sentence representation. This pooling step often uses mean or max pooling, or leverages the output of the [CLS] token, though mean pooling is commonly preferred for its simplicity and effectiveness. The model is then fine-tuned using contrastive or triplet loss objectives to ensure semantically similar sentences have closer embeddings.
The core innovation in Sentence-BERT lies in its use of a siamese or triplet network structure during training. For example, in a siamese setup, two input sentences are processed through the same BERT-based encoder, and their pooled embeddings are compared using a cosine similarity loss or other distance metrics. If training with triplets (anchor, positive, negative), the model learns to minimize the distance between the anchor and positive examples while maximizing it between the anchor and negative examples. This approach is particularly effective for tasks like semantic textual similarity (STS), where the goal is to rank sentences by how closely their meanings align. For instance, SBERT might be trained on datasets like the Stanford Natural Language Inference (SNLI) corpus, which provides labeled pairs of sentences with relationships like entailment, contradiction, or neutrality.
In practice, developers can adapt SBERT for specific use cases by adjusting the training objective or pooling strategy. For example, if the goal is information retrieval, the model might be fine-tuned with a contrastive loss using pairs of queries and relevant documents. The final embeddings are typically 768-dimensional (matching BERT-base) and can be used directly for tasks like clustering (with k-means) or classification (by adding a lightweight classifier on top). Unlike vanilla BERT, which requires expensive pairwise computations for similarity tasks, SBERT precomputes sentence embeddings once, enabling efficient large-scale comparisons. This architecture balances performance and practicality, making it widely used in applications like semantic search, duplicate detection, and recommendation systems.