Sentence Transformers, which generate dense vector representations for text, are typically trained on datasets that emphasize semantic relationships between sentences. The most common datasets include the Stanford Natural Language Inference (SNLI) corpus, Semantic Textual Similarity (STS) benchmarks, and their extensions like Multi-Genre NLI (MNLI). These datasets provide labeled pairs of sentences, enabling models to learn how to map semantically similar sentences closer in the embedding space while pushing dissimilar ones apart. For example, SNLI contains 570,000 sentence pairs labeled as entailment, contradiction, or neutral, which helps models understand logical relationships. STS datasets (e.g., STS Benchmark, STS 2012-2016) offer sentence pairs with similarity scores (0-5), allowing models to optimize for continuous similarity matching. Additional datasets like AllNLI (a combination of SNLI and MNLI) or Quora Question Pairs (400,000 potential duplicate questions) are also widely used to improve generalization across diverse text types.
Training often involves a combination of these datasets through multi-task learning. For instance, models might first learn from NLI data (SNLI, MNLI) using a classification-based objective (e.g., cross-entropy loss) to distinguish entailment, contradiction, or neutrality. This helps the model capture high-level semantic relationships. Next, STS data is used to fine-tune the embeddings via regression-based objectives (e.g., mean squared error), aligning the cosine similarity of embeddings with human-annotated similarity scores. Some pipelines also incorporate contrastive learning, where triplets of sentences (anchor, positive, negative) are constructed from NLI labels, and the model learns to minimize the distance between the anchor and positive while maximizing it from the negative. This hybrid approach—combining categorical and continuous labels—ensures embeddings are versatile enough for tasks like clustering, retrieval, or semantic search.
Beyond these core datasets, domain-specific or larger corpora are sometimes added for broader coverage. For example, Wikipedia or MS MARCO (a large-scale search dataset) might be used to expose the model to informal queries or factual content. However, SNLI, STS, and MNLI remain foundational for general-purpose embeddings because they provide clear, human-annotated signals for semantic relationships. When training a Sentence Transformer, developers often start with these datasets and optionally augment them with task-specific data. For example, the popular all-mpnet-base-v2 model was trained on SNLI, MNLI, STS, and other NLI data, demonstrating their effectiveness in producing robust embeddings. The key is balancing diversity (multiple genres, text lengths) and annotation quality to ensure the model generalizes well across use cases.