What are skip connections or residual connections?

Skip connections, also called residual connections, are a neural network design technique that helps train deeper models by allowing gradients to flow more effectively during backpropagation. They work by creating a shortcut path that skips one or more layers, adding the input of a layer directly to its output. This addition operation ensures that even if the main layer(s) learn nothing useful, the network can still pass the original input forward, preventing degradation in performance as the network grows deeper. For example, in a convolutional neural network (CNN), a skip connection might take the input of a convolutional block, pass it through the block, and then add it to the block’s output before applying a non-linear activation.

A key example of skip connections in practice is the ResNet architecture, which popularized their use. In ResNet, each residual block contains two or three convolutional layers, followed by batch normalization and ReLU activations. The input to the block is added to the output of these layers, creating a “residual” path. If the layers in the block learn to make small adjustments (residuals) to the input, the network can approximate complex functions more efficiently. For instance, ResNet-34 uses 34 layers with skip connections every few layers, while deeper variants like ResNet-101 stack more blocks. Without skip connections, training such deep networks often leads to vanishing gradients, where updates to early layers become negligible during backpropagation. Skip connections mitigate this by providing a direct path for gradients to flow backward.

The benefits of skip connections extend beyond CNNs. They are widely used in architectures like U-Net for image segmentation, where they connect encoder and decoder layers to preserve spatial details. Skip connections also simplify optimization: even if some layers are underperforming, the network can still rely on the identity mapping from the skip path. This makes models more robust and easier to train. For example, in transformer-based models, skip connections are often applied around attention and feed-forward layers to stabilize training. By combining these shortcuts with techniques like batch normalization, developers can build deeper, more accurate models without sacrificing training stability.