How does the visual backbone (e.g., CNNs, ViTs) interact with language models in VLMs?

In vision-language models (VLMs), the visual backbone (like CNNs or Vision Transformers) and the language model (like BERT or GPT) work together by converting visual data into a form that aligns with text representations. The visual backbone processes raw images into feature vectors, which are then mapped to a shared embedding space that the language model can interpret. For example, a CNN might extract spatial features from an image, which are then flattened and projected into a sequence of vectors matching the language model’s input dimensions. The language model treats these vectors as additional tokens, similar to text, enabling joint processing of visual and textual data through mechanisms like cross-attention or concatenation. This allows the model to generate text that references visual content or answer questions about images.

A concrete example is CLIP, which uses a ViT or CNN to encode images and a transformer to encode text. Both encoders output embeddings in a shared space, allowing similarity comparisons between images and text. In models like Flamingo or LLaVA, the visual backbone generates a grid of features, which are fed into the language model alongside text tokens. The language model uses cross-attention layers to “attend” to these visual features when generating each word. For instance, when answering “What color is the car?” about an image, the visual backbone identifies car-related features, and the language model links them to the word “color” in the question. Training often involves tasks like image captioning, where the model learns to align visual features (e.g., object shapes) with textual descriptions (e.g., “a red car”).

Developers implementing VLMs must address challenges like aligning visual and text feature dimensions. A common approach is adding a linear projection layer to map visual features to the language model’s embedding size. For efficiency, some architectures freeze the visual backbone during training to reduce compute costs, focusing updates on the language model and projection layers. Another consideration is handling variable input sizes: images are often resized or split into patches (as in ViTs) to create fixed-length sequences. Tools like Hugging Face’s transformers library provide APIs to combine pretrained vision and language models, simplifying experimentation. For example, using a pretrained ResNet and GPT-2, a developer can stack a projection layer on top of ResNet’s output and feed the result into GPT-2’s input embeddings to build a basic VLM.