What is a dense vector in IR?

A dense vector in information retrieval (IR) is a numerical representation of data—such as text, images, or other entities—encoded as a fixed-length array of floating-point numbers. Unlike sparse vectors, which use high-dimensional spaces with mostly zero values (e.g., one-hot encoding or TF-IDF), dense vectors are compact and capture semantic relationships in a lower-dimensional space. Each dimension in a dense vector represents a learned feature, enabling the model to encode nuanced patterns like contextual meaning or similarity between items. For example, the word “car” might be represented as [0.24, -0.53, 0.82, …] in a 300-dimensional space, where each value reflects attributes like transportation, speed, or mechanical components.

Dense vectors are typically generated using machine learning models like neural networks. In text-based IR, models such as BERT, Word2Vec, or sentence transformers convert words, phrases, or entire documents into dense embeddings. These models learn to position semantically similar items closer in the vector space. For instance, the vectors for “dog” and “puppy” will have a small cosine distance, while “dog” and “airplane” will be farther apart. During retrieval, systems compare query and document vectors using similarity metrics (e.g., cosine similarity) to rank results. This approach improves over keyword-based methods by understanding context: a search for “cold weather clothing” could retrieve documents mentioning “parkas” or “insulated jackets” even if those exact terms aren’t in the query.

A practical example of dense vectors in IR is modern search engines or recommendation systems. Platforms like Spotify use dense vectors to represent songs based on audio features and user interactions, allowing them to recommend tracks with similar patterns. In e-commerce, product descriptions might be encoded as dense vectors to surface items related to a user’s search, even with varied phrasing. Developers often implement this using libraries like FAISS or Annoy for efficient vector similarity searches. For instance, a Python script using Sentence-BERT might embed a query like “best action movies” and retrieve films tagged with “thriller” or “adventure” based on vector proximity, bypassing rigid keyword dependencies. This flexibility makes dense vectors a core component of modern IR systems.