What is the exploding gradient problem?

The exploding gradient problem occurs when the gradients computed during neural network training become excessively large, causing unstable updates to the model’s weights. This issue is common in deep networks and recurrent neural networks (RNNs), where repeated multiplication of gradients during backpropagation can amplify small numerical values into uncontrollably large ones. For example, in a network with many layers, if the weights are initialized to values greater than 1, the chain rule used in backpropagation multiplies these values across layers, leading to gradients that “explode” exponentially. This makes the model’s parameters oscillate wildly or become numerically unstable, preventing effective learning.

The immediate effect of exploding gradients is that the model’s weights receive updates so large that they overshoot reasonable values, often resulting in numerical overflow (e.g., NaN values in training logs). For instance, in an RNN processing a long sequence, the gradients for early time steps might be multiplied by the same weight matrix repeatedly, causing them to grow exponentially. This destabilizes training, as the optimizer cannot converge to a meaningful solution. In practice, you might observe the loss suddenly spiking or becoming undefined, even if the network architecture and data seem correct. This problem is distinct from its counterpart, the vanishing gradient issue, where gradients become too small—exploding gradients are about magnitude exceeding practical limits.

To mitigate exploding gradients, developers use techniques like gradient clipping, which caps gradient magnitudes during backpropagation. For example, in PyTorch, you can apply torch.nn.utils.clip_grad_norm_ to limit the maximum norm of gradients. Another approach is careful weight initialization—using methods like Xavier or He initialization to set initial weights in a range that avoids extreme values. Architectural choices also matter: switching from activation functions like tanh (which can produce larger derivatives) to ReLU (with a derivative of 1 for positive inputs) reduces the risk. For RNNs, architectures like LSTMs or GRUs include gating mechanisms that regulate gradient flow. Additionally, batch normalization layers can stabilize intermediate activations, indirectly controlling gradient scales. These strategies, combined with monitoring gradient norms during training, help maintain stable learning dynamics in deep networks.