home » the workshop » Bridge-Crossing RL Sim
Machine learning
NEW!Bridge-Crossing RL Sim
Hierarchical reinforcement learning in a grid world. A manager agent picks sub-goals, a worker agent walks there, and together they learn to fetch a log, build a bridge and get home. A “profit-sharing” reward keeps their goals aligned.
- access:
- public
- year:
- 2026
- built with:
- Python, PyTorch, Django, Web Workers
A Django web app for watching a hierarchical reinforcement learning agent learn in real time. The agent lives in a grid world split by a river. To get home it has to pick up a bridge piece, place it in the river, cross, and walk to the house.
The work is split between two agents. The manager only sees the big picture (do I have the bridge piece, is it placed, have I crossed?) and issues sub-goals like GOTO_LOG or GOTO_RIVER. The worker sees its own position and moves up, down, left and right to reach whatever sub-goal it was handed.
Training two separate value estimators toward one shared goal is fragile. A few clumsy worker moves can drag the manager’s Q-values far from a strategy it had already found. The fix here is profit sharing, in the spirit of potential-based reward shaping: the manager is rewarded when a command actually changes the world (it said GOTO_LOG, and now the agent HAS_LOG), and the worker earns a cut of that profit every step, positive or negative depending on what the manager currently expects. That small signal is enough to tell the worker why a square is worth walking to.
~ highlights ~
- Both agents are Q-learners backed by small neural networks, with periodically synced target networks to keep training stable
- Hindsight Experience Replay lets the worker learn from failed attempts by relabeling them as successes toward wherever it actually ended up
- Dense worker rewards: +100 for each milestone, −1000 for drowning, and a small penalty for every step
- A Web Worker polls the simulation API so training runs without freezing the page, and you can inspect the worker’s live Q-value map