Why was the Sentence-BERT approach needed, even with powerful language models like BERT already available?

Sentence-BERT (SBERT) was developed to address a key limitation of models like BERT: efficiently creating meaningful sentence embeddings for comparison tasks. While BERT excels at understanding context in tasks like question answering or text classification, it struggles to produce standalone sentence representations that can be directly compared using similarity measures like cosine distance. For example, BERT processes input pairs together (e.g., two sentences for similarity checks), which forces developers to run the model multiple times for pairwise comparisons. This becomes impractical in scenarios like clustering thousands of documents or real-time recommendation systems, where latency and computational costs matter.

The second issue lies in how BERT generates embeddings. BERT outputs contextualized token vectors, but deriving a single sentence vector often involves workarounds like averaging token embeddings or using the [CLS] token’s output. These methods don’t consistently capture semantic meaning. For instance, averaging tokens might dilute the importance of keywords, while the [CLS] token isn’t inherently optimized for semantic tasks. SBERT solves this by fine-tuning BERT with a siamese network structure, where two identical BERT models process sentences independently. During training, it uses objectives like triplet loss or contrastive loss to ensure similar sentences (e.g., “a dog plays” and “a puppy runs”) are closer in the embedding space, while dissimilar ones are pushed apart. This creates embeddings tailored for direct comparison.

Finally, SBERT enables practical scalability. Traditional BERT would require comparing every sentence pair by feeding them through the model together, which is computationally expensive for large datasets. With SBERT, embeddings can be precomputed once and stored, allowing similarity searches using simple vector operations. For example, a search engine could precompute embeddings for millions of articles and quickly find matches for a user query by comparing its precomputed embedding against the database. This reduces inference time from hours to milliseconds in large-scale applications. By addressing efficiency, embedding quality, and scalability, SBERT became a critical tool despite BERT’s foundational capabilities.