What is the difference between on-policy and off-policy learning?

On-policy and off-policy learning are two approaches in reinforcement learning that differ in how they use data to update a policy. On-policy learning requires that the data used to train the policy is generated by the same policy currently being improved. In other words, the agent learns exclusively from actions it takes while following its current strategy. Off-policy learning, by contrast, allows the agent to learn from data generated by a different policy, such as an older version of the policy or a completely separate exploration strategy. This distinction impacts how algorithms collect data, update policies, and balance exploration with exploitation.

A classic example of on-policy learning is the SARSA algorithm. SARSA updates its action-value estimates based on the current policy’s next action. For instance, if an agent is in state S, takes action A, moves to state S’, and then selects action A’ (using its current policy), SARSA uses this sequence (S, A, R, S’, A’) to update its Q-values. Since the next action A’ is chosen by the same policy being trained, the algorithm stays on-policy. In contrast, Q-learning is off-policy because it updates Q-values using the maximum estimated future reward for the next state (max Q(S’, a)), regardless of which action the current policy would actually take. This allows Q-learning to learn optimal policies even when the agent’s exploration strategy (e.g., ε-greedy) doesn’t always choose the best action.

The choice between on-policy and off-policy methods depends on the problem’s requirements. On-policy methods like Advantage Actor-Critic (A2C) are often simpler to implement and more stable because they directly optimize the policy generating the data. However, they can be less sample-efficient, as they discard data after each policy update. Off-policy methods like Deep Q-Networks (DQN) with experience replay are more flexible and sample-efficient, as they reuse past experiences. For example, DQN stores transitions in a buffer and randomly samples them to break correlations in the data, enabling better generalization. However, off-policy algorithms can be more complex, requiring techniques like importance sampling to correct for mismatches between the data-generating policy and the target policy. Developers might choose on-policy methods for stability in dynamic environments and off-policy methods when data efficiency or reuse is critical.