What is an episodic vs. continuous task in RL?

In reinforcement learning (RL), tasks are categorized as episodic or continuous based on how they handle termination and interaction timelines. Episodic tasks have distinct start and end points, called episodes, where the agent’s interactions reset after reaching a terminal state. For example, a chess game ends when checkmate occurs. Continuous tasks, also called non-episodic, lack predefined endpoints—the agent interacts with the environment indefinitely, aiming to maximize long-term rewards without episodic resets. The distinction impacts how agents learn, evaluate performance, and manage rewards.

Episodic tasks are structured around independent trials. Each episode allows the agent to explore actions, receive rewards, and reset to a starting state, enabling clear performance evaluation. For instance, training an agent to play a video game level involves episodes that end when the agent wins, loses, or exceeds a time limit. This structure simplifies learning because the agent can analyze complete trajectories (state-action-reward sequences) after each episode. Algorithms like Monte Carlo methods leverage this by updating policies only after an episode concludes. Episodic frameworks also simplify debugging, as developers can track progress per episode (e.g., average rewards per level). However, they assume environments can be reliably reset, which isn’t always practical in real-world systems.

Continuous tasks require the agent to optimize behavior without resets, making them inherently more complex. For example, a robot maintaining balance must continuously adjust to disturbances without a natural endpoint. Here, the discount factor (gamma) becomes critical to prioritize immediate rewards over distant ones, preventing infinite reward sums. Temporal Difference (TD) methods, like Q-learning, are often used because they update estimates incrementally without waiting for episode completion. Continuous tasks also face challenges like exploration-exploitation trade-offs in perpetually changing environments. Developers must design reward functions carefully to avoid unintended behaviors, as the agent’s actions have unbounded consequences. Real-world applications like autonomous driving or energy management systems often fall into this category, demanding algorithms that handle indefinite interaction and partial observability.