๐งฉ Maze Solver โ
Watch three algorithms โ Q-Learning, SARSA, and Monte Carlo โ race through a procedurally generated maze in real time. Switch between them and see what makes each one different.
โถ Open Live Demo โWhat this demo shows โ
The Q-value heatmap. The grid is colour-coded: each cell's brightness represents how valuable the algorithm thinks that cell is. Watch the heatmap fill in from the exit outward โ that's the Bellman equation propagating reward backward through the maze.
The policy arrows. Switch to "Policy view." Each cell shows an arrow indicating the greedy action โ the best move the agent knows. Early training: arrows point randomly. Late training: all arrows form a continuous path to the exit.
Three algorithms, one maze. The key comparison isn't raw performance โ it's how each algorithm learns:
| Algorithm | Updates when? | Follows which policy? | Safer near walls? |
|---|---|---|---|
| Q-Learning | Every step | Greedy (optimal) | No โ optimistic |
| SARSA | Every step | Behavioural (ฮต-greedy) | Yes โ conservative |
| Monte Carlo | End of episode | โ (uses full returns) | No โ high variance early |
Try these experiments โ
Experiment 1 โ Short vs long maze. Set maze size to 5ร5. Watch all three converge quickly. Now switch to 15ร15. Monte Carlo's learning curve flatlines early โ it needs successful episodes to learn, and those are rare at first in a large maze.
Experiment 2 โ Cliff edge. Enable the "cliff" mode (penalty โ10 for falling). Q-Learning will still walk near the edge because it assumes optimal future play. SARSA will stay further back โ it knows the agent will sometimes explore randomly, so it accounts for the risk.
Experiment 3 โ Compare variance. Run Q-Learning and Monte Carlo for 500 episodes each with the same random seed. Plot their episode reward curves. Monte Carlo's curve is bumpier. That's variance: full-episode returns bounce around more than step-by-step TD estimates.
The chapter behind this demo โ
This demo is paired with two chapters:
- Teaching an Agent to Remember โ covers Q-Learning and SARSA in full, including the Bellman equation and on-policy vs off-policy learning
- Waiting for the Ending โ covers Monte Carlo methods, the discounted return formula, and the bias-variance tradeoff
Difficulty: Beginner ยท Algorithms: Q-Learning, SARSA, Monte Carlo ยท Environment: Grid Maze