What is the difference between feature vectors and embeddings?

Feature vectors and embeddings are both numerical representations of data, but they differ in how they are created and what they capture. A feature vector is a collection of explicit, hand-engineered features that describe a data point. For example, in image processing, a feature vector might include pixel values, histogram statistics, or edge detection outputs. These features are chosen based on domain knowledge and are designed to highlight specific aspects of the data. In contrast, an embedding is a learned, dense vector representation that maps data into a lower-dimensional space. Embeddings are typically generated by neural networks (e.g., Word2Vec, BERT, or CNNs) and aim to capture latent patterns or relationships in the data automatically.

The primary distinction lies in their creation process. Feature vectors rely on manual feature engineering, where developers explicitly define which characteristics of the data are relevant. For instance, in natural language processing (NLP), a feature vector for a sentence might include term frequencies or part-of-speech tags. Embeddings, however, are derived through training: a model learns to represent data by optimizing for a task (e.g., predicting neighboring words in Word2Vec or classifying images in a CNN). This means embeddings encode information about relationships (e.g., “king” and “queen” being close in a word embedding space) that might not be obvious in handcrafted features.

Another key difference is their structure and usage. Feature vectors are often high-dimensional and sparse (e.g., one-hot encodings in text data), while embeddings are dense and compact. For example, a one-hot encoded word vector with 10,000 dimensions might be compressed into a 300-dimensional embedding. Embeddings also generalize better across tasks because they capture abstract patterns, whereas feature vectors are tied to specific domain assumptions. A practical example is image classification: using Histogram of Oriented Gradients (HOG) as a feature vector works for basic tasks, but a ResNet-generated embedding can adapt to more complex visual patterns. Developers often use embeddings as inputs to downstream models, while feature vectors are more common in traditional machine learning pipelines.