What is the difference between policy gradients and Q-learning?

Policy gradients and Q-learning are two core approaches in reinforcement learning, differing primarily in what they optimize. Policy gradients directly adjust the parameters of a policy (a function mapping states to actions) to maximize expected rewards. For example, a neural network might output probabilities for taking actions in a game, and the algorithm tweaks these probabilities based on whether past actions led to high rewards. In contrast, Q-learning focuses on learning a value function (the Q-function) that estimates the long-term reward of taking a specific action in a given state. The policy is then derived indirectly by selecting actions with the highest Q-values. For instance, in a grid-world navigation task, Q-learning would assign a value to each possible move in every cell, guiding the agent toward the optimal path.

The way each method handles exploration and updates also differs. Q-learning is “off-policy,” meaning it can learn from historical or random actions stored in a replay buffer. It updates Q-values using the Bellman equation, which combines immediate rewards with discounted future rewards. For example, if moving right in a grid cell gives a reward of +1, the Q-value for that action is updated to reflect both the +1 and the best possible future rewards from the next state. Policy gradients, however, are typically “on-policy,” requiring fresh data from the current policy. They compute gradients of the expected reward with respect to the policy parameters, nudging actions toward those that yielded higher rewards. For example, if a robot arm’s movement led to success, the policy gradient method increases the likelihood of similar movements in the future.

Use cases often dictate which method to choose. Q-learning works well for discrete action spaces (e.g., choosing between 4 directions in a grid) but struggles with continuous actions (e.g., adjusting a motor’s torque). Policy gradients excel in continuous or high-dimensional action spaces, like training a character to walk in a simulation. Q-learning can be more sample-efficient due to replay buffers, while policy gradients often require more interactions but are more flexible. For example, Q-learning might train a game-playing agent faster with limited data, whereas policy gradients could better handle complex robotics tasks where actions are fine-grained and continuous. Both methods have trade-offs, and hybrid approaches (like Actor-Critic) sometimes combine their strengths.