Just publishedA GitHub Actions CI/CD Security Baseline: Pinned Actions, Scoped Secrets, Verified Artifacts, and Useful Logs
UgraByte
Reinforcement learning

Q-Learning Grid World — Watch Reinforcement Learning Train

Train an agent with no map and no instructions, and watch a policy emerge from trial and error.

Loading the interactive version…

What this actually is

Reinforcement learning has no dataset. There are no labelled examples and no structure to uncover — only an agent, an environment, and a stream of consequences. The agent acts, the environment answers with a new situation and occasionally a reward, and the agent must work out which earlier decisions earned a reward that arrived much later.

That is credit assignment, and it is the hard part. Reaching the goal earns one point, but the move that mattered might have been twenty steps earlier. Q-learning keeps a number for every cell-and-direction pair — the expected total future reward — and nudges each toward the reward just received plus the best value from wherever it led. Value seeps backwards from the goal, one step per visit.

What you’re watching

The grid is the environment and the circle is the agent, heading from the outlined cell to the filled one. It is given no map — it does not know where the obstacles are, or that walls exist, and finds out by walking into them.

The tint on each cell is the value learned for standing there. Everything starts blank: until the goal is found once by accident, there is nothing anywhere to be greedy about. After that a faint tint appears beside the goal and creeps outward about one cell per episode — that spreading stain is credit assignment happening.

The arrows are the policy: the move the agent would make if it stopped exploring. They resolve into a clean route well before the agent's own wandering settles down, and that gap is worth sitting with. The first episode takes fifty to two hundred steps; by episode twenty it is usually in the high teens, against an optimum of fourteen.

How to drive it

  1. Tap cells to add or remove obstacles, then press Train and watch the agent wander before it starts to improve.

  2. Compare the step count for this episode against the best it has managed and against the shortest route that exists — the three converge as the policy improves.

  3. Turn the exploration rate up to 1 and keep training: the agent's wandering never improves, but the policy arrows still resolve to the optimal route. Q-learning learns the best route even while refusing to walk it.

  4. Drop the learning rate to its minimum and reset. The same run takes several times as many episodes to reach the same policy, because each experience is now allowed to change the estimates only slightly.

How this relates to real systems

Two things separate this from a working system, and both are size. The Q table here is 36 cells by 4 directions — 144 numbers, few enough to draw. Real problems have state spaces nobody could enumerate: a chess position, a frame of video. Deep Q-learning replaces the table with a network estimating those same values without storing them.

Failure here is also free, which is why RL succeeded first in simulators. A robot arm or a treatment policy cannot learn by breaking things, so applied RL trains in simulation and transfers, or learns from logged behaviour. The exploration tradeoff is real though — it drives recommenders and ad auctions as the multi-armed bandit problem.

Frequently asked questions

What is the Q in Q-learning?
Q is the name of the function being learned: Q(state, action) is the expected total future reward from taking that action in that state and behaving well afterwards. In this demo it is literally a table with one row per cell and one column per direction, which is why the approach is called tabular Q-learning. The whole algorithm is the rule for updating one of those numbers after each move.
Why does the policy look right before the agent starts walking well?
Because they are two different things. Q-learning is off-policy: it learns the value of acting optimally regardless of how the agent actually behaved while gathering the experience. So the arrows can point along the shortest route while the agent, still taking random moves a fraction of the time, keeps wandering off it. Turn the exploration rate to 1 and this becomes obvious — the agent never improves at all, but its learned policy still reaches the optimum.
What does the exploration rate actually do?
It is the probability that the agent ignores everything it knows and moves at random. At 0 it always takes the action it currently rates highest; at 1 it never uses what it has learned. Raising it makes the agent's behaviour steadily worse — median episode length on the default layout runs about 14 steps at 0, 17 at 0.1, 21 at 0.3, 31 at 0.5 and around 190 at 1 — while the learned policy reaches the optimum at every one of those settings.
Shouldn't an exploration rate of zero get the agent stuck?
That is the usual intuition and it does not hold in a grid this small, which is worth knowing rather than glossing over. Because every ordinary step costs a little, any action the agent has already tried has a slightly negative value while every untried action still reads zero — so a purely greedy agent keeps picking actions it has not tried yet, which amounts to a systematic sweep of the grid. On the default layout it reaches the goal by the second episode. In a large or stochastic environment that accidental exploration is nowhere near enough, which is why real systems explore deliberately.
What does the learning rate change?
How much of each new estimate replaces the old one. At 1 the agent completely believes its most recent experience; near 0 it barely updates at all and needs many more episodes to reach the same policy — on the default layout roughly 90 episodes at 0.05 against under 20 at 0.5. A high rate is fine here because the environment is deterministic, so each experience is trustworthy. In a noisy environment a high rate makes the estimates thrash, chasing whatever happened most recently rather than averaging over what usually happens.