What is catastrophic forgetting in RL?

Catastrophic forgetting in reinforcement learning (RL) occurs when an agent loses the ability to perform previously learned tasks after being trained on new ones. This happens because the neural network’s parameters, which encode the original knowledge, are overwritten during training on new data. Unlike humans, who can retain skills while learning new ones, RL models often struggle to balance old and new information, leading to a rapid decline in performance on prior tasks. The problem is particularly common in sequential learning scenarios, where an agent faces a series of tasks without access to past training data.

The root cause lies in how neural networks update their parameters. When an RL agent learns a new task, gradient-based optimization adjusts the network’s weights to minimize errors for the current task. However, these adjustments aren’t constrained to protect knowledge from earlier tasks. For example, imagine an agent trained to navigate a maze. If it’s later trained to avoid moving obstacles in the same maze, the updates to its policy network might erase the original navigation strategy, making it unable to reach the goal even if obstacle avoidance is mastered. This effect is amplified in RL because the agent’s own actions influence the data it learns from, creating a feedback loop where outdated strategies are discarded entirely.

To mitigate catastrophic forgetting, developers use techniques like experience replay (storing past data and retraining on it alongside new tasks) or elastic weight consolidation (identifying and protecting critical network weights for old tasks). For instance, in a robot learning to grasp objects, experience replay might involve periodically revisiting earlier grasping scenarios to reinforce those skills. Another approach is modular architecture design, where separate networks handle different tasks. While these methods reduce forgetting, they often come with trade-offs in computational cost or flexibility. Balancing stability (retaining old knowledge) and plasticity (learning new tasks) remains a key challenge in building robust RL systems.