What types of data can be used for self-supervised learning?

Self-supervised learning (SSL) works with data that contains inherent structure or relationships, allowing models to generate supervision signals without manual labeling. Common data types include text, images, video, audio, time-series sequences, and graph-structured data. The key requirement is that the data must enable the creation of pretext tasks—automatically generated challenges that teach the model meaningful patterns. For example, text can be used to predict missing words, while images might be manipulated to train a model to reconstruct obscured regions. These tasks rely on the natural coherence of the data itself.

Text is a natural fit for SSL because of its sequential and contextual nature. Models like BERT use masked language modeling, where random words in a sentence are hidden, and the model learns to predict them based on surrounding context. Another example is next-sentence prediction, where the model determines if two text segments logically follow each other. For images, common pretext tasks include predicting the rotation angle of an image or solving jigsaw puzzles by rearranging shuffled patches. Video data adds a temporal dimension: models can learn by predicting the order of shuffled frames or estimating the time gap between clips. Audio data, such as speech recordings, can be used to train models to reconstruct masked audio segments or align speech with corresponding text transcripts.

Time-series data, like sensor readings or financial records, often contains patterns that SSL can exploit. For instance, a model might predict future values in a sequence based on past observations or fill in missing data points. Graph-structured data, such as social networks or molecular structures, enables SSL through tasks like node embedding (predicting connections between nodes) or graph-level property prediction. The key consideration for developers is to identify inherent relationships or transformations in their data. For example, video frames have temporal continuity, text has word dependencies, and graphs have node-edge relationships. Preprocessing steps—like extracting frames from video or tokenizing text—are often needed to structure the data for SSL. By leveraging these natural patterns, developers can train robust models even when labeled data is scarce.