What is Temporal Difference (TD) learning in reinforcement learning?

Temporal Difference (TD) learning is a reinforcement learning method that combines ideas from Monte Carlo sampling and dynamic programming to estimate the value of states or actions. Unlike Monte Carlo methods, which wait until the end of an episode to update value estimates, TD learning updates estimates incrementally after each step. This allows it to learn online, making it more efficient for tasks where episodes are long or infinite. TD learning uses a concept called “bootstrapping,” where current estimates are refined using subsequent estimates, balancing immediate rewards with future predictions.

A core example is the TD(0) algorithm, which updates a state’s value based on the observed reward and the estimated value of the next state. Suppose an agent is navigating a grid-world environment. When moving from state s to s’, TD(0) adjusts the value of s using the formula: V(s) = V(s) + α [R + γV(s') - V(s)]. Here, α (learning rate) controls how aggressively the update is applied, γ (discount factor) reduces future rewards’ importance, R is the immediate reward, and [R + γV(s') - V(s)] is the TD error—the difference between the current estimate and the new target. This error drives the learning process, correcting the value of s without waiting for the episode’s outcome.

TD learning is foundational to algorithms like Q-learning and SARSA. For instance, Q-learning uses TD to update action-value pairs (Q(s,a)) by comparing the best possible future value to the current estimate. A practical use case is training a game AI: the agent might explore a maze, receiving rewards for finding keys and penalties for hitting obstacles. With TD, the AI adjusts its strategy after every move, refining its understanding of which paths are valuable. This approach is computationally efficient and works well in environments where immediate feedback is sparse, making it a staple in real-world applications like robotics control or recommendation systems.