What is intrinsic motivation in reinforcement learning?

Intrinsic motivation in reinforcement learning (RL) refers to techniques that encourage an agent to explore its environment by creating internal rewards based on its own experiences, rather than relying solely on external rewards provided by the environment. Unlike extrinsic motivation, which depends on predefined goals (e.g., earning points in a game), intrinsic motivation drives the agent to seek novelty, learn skills, or reduce uncertainty. For example, an agent might reward itself for visiting unfamiliar states or making unexpected predictions, even if those actions don’t immediately contribute to solving the task. This approach helps agents explore more effectively, especially in environments where external rewards are sparse or delayed.

One common method for implementing intrinsic motivation is curiosity-driven exploration. Here, the agent generates an internal reward based on how surprised it is by the outcomes of its actions. For instance, the Intrinsic Curiosity Module (ICM) uses a prediction model to estimate how well the agent can predict the next state given its current state and action. The difference between the predicted and actual next state—the prediction error—becomes the intrinsic reward. Another example is Random Network Distillation (RND), where the agent learns to predict the output of a randomly initialized neural network. States that are harder to predict (higher error) yield higher rewards, encouraging exploration of less familiar areas.

The primary benefit of intrinsic motivation is improved exploration in complex or sparse-reward environments. For example, in a maze-solving task where the external reward is only given upon reaching the exit, an agent with intrinsic motivation might explore dead-ends more thoroughly, increasing its chances of eventually finding the correct path. Similarly, in robotics, an agent learning to walk could use intrinsic rewards to experiment with different movements, even if no external feedback exists until a stable gait is achieved. However, intrinsic motivation isn’t a silver bullet—some methods can lead to distractions (e.g., agents fixating on unpredictable but irrelevant states). Developers often combine intrinsic and extrinsic rewards to balance exploration and task-specific goals, making the approach adaptable to diverse RL scenarios.