What is an intrinsic reward in RL?

An intrinsic reward in reinforcement learning (RL) is a self-generated signal that encourages an agent to explore or learn behaviors beyond what is explicitly defined by external rewards (extrinsic rewards). Unlike extrinsic rewards, which come from the environment (e.g., points for winning a game), intrinsic rewards are designed to guide the agent’s learning process internally. They help address challenges like sparse rewards—where the agent rarely receives meaningful feedback—by creating intermediate goals that promote exploration and skill acquisition. For example, in a maze-solving task, an extrinsic reward might only be given when reaching the exit, while an intrinsic reward could incentivize the agent to visit new areas along the way.

A common implementation of intrinsic rewards is curiosity-driven exploration. Here, the agent generates rewards based on how novel or unfamiliar a state or action is. One approach uses a prediction model: the agent predicts the outcome of its actions, and the difference between the predicted and actual outcome (prediction error) becomes the intrinsic reward. For instance, in a grid-world environment, if an agent moves into an unexplored region and its model fails to predict the resulting state accurately, the high prediction error would yield a large intrinsic reward. This encourages the agent to revisit similar scenarios until the error diminishes, effectively driving exploration. Another method is count-based exploration, where the agent assigns higher rewards to states it has visited less frequently, using techniques like hash-based counting or neural density models to track state visitation.

Intrinsic rewards are often combined with extrinsic rewards to balance exploration and exploitation. For example, in the game Montezuma’s Revenge—a classic RL challenge with sparse rewards—agents using intrinsic motivation (e.g., curiosity or novelty bonuses) discover key items and rooms faster than those relying solely on extrinsic rewards. Developers can implement intrinsic rewards by adding them to the extrinsic reward signal during training. Libraries like OpenAI’s Gym or PyTorch offer frameworks for experimenting with these concepts. However, designing effective intrinsic rewards requires careful tuning to avoid overfitting to exploration (e.g., an agent endlessly chasing novelty without solving the task). By integrating intrinsic rewards thoughtfully, developers can create agents that learn more efficiently in complex, uncertain environments.