What is Monte Carlo (MC) learning in reinforcement learning?

Monte Carlo (MC) learning is a reinforcement learning technique that estimates value functions (like state or action values) by averaging the returns from complete episodes of interaction with an environment. Unlike methods that update estimates incrementally (e.g., Temporal Difference learning), MC waits until an episode concludes before computing the total reward and updating values. This approach is model-free, meaning it doesn’t require prior knowledge of the environment’s dynamics (e.g., transition probabilities). MC is particularly useful in episodic tasks where experiences naturally break into distinct sequences, such as games with clear endings or tasks with terminal states.

A key advantage of MC is its simplicity and direct use of actual returns, which makes it unbiased—it doesn’t rely on estimates of future rewards. For example, in a game like Blackjack, the agent can play a full hand and use the final outcome (win/loss) to update the value of every state encountered during that episode. However, MC has drawbacks. Since it requires completing an episode before updating, it’s inefficient for long or continuous tasks. Additionally, the reliance on full returns introduces high variance in updates, as the outcomes of episodes can vary widely. This contrasts with methods like TD learning, which trade off some bias for lower variance by bootstrapping (estimating future rewards incrementally).

To illustrate, consider training an agent to play Blackjack. Each episode corresponds to a single hand. The agent’s state might include its current card sum, the dealer’s visible card, and whether it holds a usable ace. Actions are “hit” or “stand,” and the reward (+1, -1, or 0) is only revealed at the end. After each hand, MC updates the value of every state visited by averaging the returns from all episodes where that state occurred. For instance, if the agent won after hitting in a state with a sum of 15 and a dealer’s 6, that state-action pair’s value is adjusted upward. This averaging over many episodes helps the agent learn which states and actions lead to better outcomes, even in stochastic environments where outcomes are uncertain.