Video frames or footage are represented as vectors by converting pixel data into numerical arrays that machines can process. Each frame in a video is essentially an image, which is a grid of pixels. For a grayscale image, each pixel is a single value (e.g., 0-255 for intensity). For color, pixels are represented as RGB tuples (three values per pixel). To turn a frame into a vector, the pixel values are flattened into a 1D array. For example, a 128x128 RGB frame becomes a vector with 128x128x3 = 49,152 dimensions. This raw pixel vector captures spatial information but is often high-dimensional and computationally heavy.
To handle temporal information across multiple frames (like motion), videos are often represented as sequences of vectors. One common approach is stacking frame vectors over time. For instance, a 10-frame video clip might be a 10x49,152 matrix. Alternatively, techniques like optical flow vectors encode motion between frames. Optical flow calculates the direction and speed of pixel movement between consecutive frames, producing a 2D vector field per frame pair. These motion vectors can be combined with raw pixel data to capture both appearance and dynamics. In practice, libraries like OpenCV simplify extracting these features, and frameworks like PyTorch or TensorFlow handle them as tensors (multi-dimensional arrays) for training models.
Real-world applications often use compressed or learned representations. Instead of raw pixels, models like CNNs (Convolutional Neural Networks) extract features. For example, a pretrained ResNet model might reduce a frame to a 2048-dimensional vector, capturing high-level patterns. For temporal modeling, architectures like 3D CNNs (processing video cubes) or RNNs (processing frame sequences) combine spatial and temporal data. A practical example is video classification: a model might take 16-frame chunks, convert each to feature vectors, then aggregate them to predict actions. This approach balances computational efficiency and information retention, enabling tasks like gesture recognition or anomaly detection in videos.