What is Prioritized Experience Replay (PER)?

Prioritized Experience Replay (PER) is a technique used in reinforcement learning to improve how an agent learns from past experiences. In standard experience replay, an agent stores interactions (like state, action, reward, and next state) in a memory buffer and randomly samples them during training. PER modifies this by assigning a priority to each experience, ensuring that more “important” experiences are sampled more frequently. Priorities are typically based on the temporal difference (TD) error—the difference between the predicted and target Q-values. Experiences with larger TD errors are considered more valuable because they represent situations where the agent’s predictions were less accurate, indicating opportunities for faster learning. This prioritization helps the agent focus on less predictable or more impactful events, accelerating training.

Implementing PER requires balancing efficiency and computational overhead. Priorities are often managed using a binary heap or a sum-tree data structure to quickly retrieve high-priority experiences. Two common prioritization strategies are proportional (priority directly scales with TD error) and rank-based (priority depends on the error’s rank in the buffer). However, prioritizing certain experiences introduces bias, as frequently sampled data points can dominate training. To mitigate this, PER uses importance sampling (IS) weights during gradient updates. These weights adjust the impact of each sample based on how likely it was to be selected, ensuring that the agent’s updates remain unbiased despite uneven sampling. For example, a rarely sampled experience with low priority would receive a higher IS weight to compensate.

A practical example of PER’s effectiveness is in training game-playing agents, such as those for Atari games. In games like Pong or Breakout, critical moments (e.g., losing a life or scoring a point) occur infrequently but have high impact. By prioritizing these events, PER helps the agent learn to avoid mistakes or replicate successful strategies faster than random sampling. However, PER increases computational complexity due to maintaining priority values and recalculating IS weights. Developers must also tune hyperparameters like the prioritization exponent (which controls how aggressively high-priority samples are favored) and the IS weight adjustment rate. Despite these trade-offs, PER is widely adopted in frameworks like DeepMind’s Rainbow DQN, demonstrating measurable improvements in sample efficiency and final performance compared to standard experience replay.