What is the advantage function in RL?

The advantage function in reinforcement learning (RL) quantifies how much better a specific action is compared to the average action in a given state. It is defined as the difference between the action-value function ( Q(s, a) ) (the expected return from taking action ( a ) in state ( s )) and the state-value function ( V(s) ) (the average expected return from state ( s )): ( A(s, a) = Q(s, a) - V(s) ). This subtraction isolates the incremental benefit of choosing action ( a ) over the “default” behavior represented by ( V(s) ). For example, if ( Q(s, a) = 10 ) and ( V(s) = 7 ), the advantage ( A(s, a) = 3 ) signals that action ( a ) is 3 units better than the average action in ( s ).

The primary benefit of the advantage function lies in reducing variance during policy updates. In policy gradient methods, updates depend on the estimated return of actions. Using raw returns (e.g., ( Q(s, a) )) can lead to high variance because returns fluctuate based on environmental randomness. By subtracting ( V(s) ), the advantage function centers the scale of updates around zero, emphasizing actions that outperform the baseline ( V(s) ). For instance, in a game where an agent navigates a maze, ( V(s) ) might estimate the average time to exit from a hallway, while ( A(s, a) ) would highlight whether turning left or right shortens that time. This centering stabilizes training, allowing the policy to focus on meaningful action differences rather than absolute rewards.

Practical implementations often estimate the advantage function using temporal difference (TD) errors or generalized advantage estimation (GAE). For example, in the A3C algorithm, a neural network predicts ( V(s) ), and the advantage is computed as the discounted sum of rewards minus ( V(s) ). GAE combines multi-step TD errors to balance bias and variance. In a robot control task, if moving a joint yields a higher reward than predicted by ( V(s) ), the advantage for that action becomes positive, reinforcing the policy to choose it more often. By decoupling action-specific benefits from state values, the advantage function enables clearer credit assignment, making it a cornerstone of modern RL algorithms like PPO and TRPO.