What are decoder-only models vs. encoder-decoder models?

Decoder-only models and encoder-decoder models are two common architectures for sequence-based tasks in machine learning. The key difference lies in how they process input and generate output. Encoder-decoder models use separate components to first understand the input (encoder) and then produce the output (decoder). Decoder-only models, on the other hand, combine these steps into a single component that generates output directly, often using the input as part of the generation process itself.

Encoder-Decoder Models Encoder-decoder architectures are designed for tasks where the input and output are distinct sequences, such as translation or summarization. The encoder processes the input (e.g., a sentence in French) into a compressed representation (often called a context vector). The decoder then uses this representation to generate the output sequence (e.g., the English translation). For example, the original Transformer model uses an encoder with self-attention to analyze the input and a decoder with both self-attention and cross-attention (to the encoder’s output) for generation. Models like BART and T5 follow this design, making them effective for tasks requiring a deep understanding of the input before generating a structured output. The separation allows the model to handle complex mappings between input and output, but it also increases computational overhead due to the dual components.

Decoder-Only Models Decoder-only models simplify the architecture by removing the encoder. These models, such as GPT-3 or LLaMA, generate output autoregressively—predicting one token at a time based on previous tokens. They treat the input as part of the output generation process. For instance, when given a prompt like “Translate to French: ‘Hello’,” the model generates the translation token by token without a dedicated encoding phase. This approach relies heavily on masked self-attention, which ensures each token only attends to previous tokens in the sequence. While efficient for text generation (e.g., chatbots or code completion), decoder-only models may struggle with tasks requiring bidirectional understanding of the input, as they process information sequentially rather than holistically.

When to Use Each Encoder-decoder models excel when input and output differ significantly in structure or meaning, such as translating between languages or answering questions from a context. Their two-step process ensures the model fully “comprehends” the input before generating. Decoder-only models are better suited for tasks where the input and output are closely aligned, like text continuation or instruction following, as they streamline generation by merging understanding and production. For example, fine-tuning GPT-3 for code generation works well because the input (a comment describing code) and output (the code itself) are tightly coupled. Choosing between the two often depends on task complexity and computational constraints: encoder-decoder offers precision for complex mappings, while decoder-only prioritizes efficiency for generation-heavy workflows.