Reinforcement learning basics

Learning from consequences, not from an answer key

Every form of learning covered elsewhere in this roadmap, from linear regression through generative models, learns from a dataset prepared in advance — examples already paired with a correct answer, or at minimum a defined target to reconstruct. Reinforcement learning has no such dataset at all. Instead, an agent takes actions inside an environment, observes the resulting state, and receives a reward signal reflecting how good or bad that outcome was. Over many repeated attempts, the agent learns which actions, taken in which situations, tend to lead toward higher cumulative reward — covered formally in discounted return, which accounts for the fact that a reward received later is typically worth somewhat less than the same reward received sooner.

What to watch for

The video covers the full agent-environment-reward loop and the exploration- exploitation tradeoff with genuinely concrete examples, at a well-paced depth that is the most complete treatment available among the candidates considered for this lesson, including a path toward Q-learning as the concrete algorithm this lesson names.

Explore versus exploit: the central tension

At every decision point, an agent faces a real tension. Exploiting means choosing whichever action currently looks best based on experience so far — sensible if that belief is accurate, but it forecloses ever discovering a genuinely better action the agent has not yet sufficiently tried. Exploring means deliberately trying something other than the current best guess, purely to gather more information — valuable for discovering better options, but costly if it means repeatedly passing up a reward the agent already knows how to get. A reasonable strategy balances both, typically exploiting the current best-known action most of the time while still exploring some small fraction of the time, so the agent keeps improving its knowledge of the environment without entirely sacrificing the reward it could otherwise be collecting.

Q-learning: estimating the value of an action

Q-learning gives this general idea a concrete algorithm. It maintains an estimate, called a Q-value, of how good it is to take a specific action in a specific state — formally, the expected cumulative future reward from taking that action there and acting well afterward, covered in the action-value function. Every time the agent takes an action and observes the resulting reward and new state, it updates its estimate for that state-action pair slightly, nudging it toward what was actually just observed — the precise update rule is given in the Q-learning update, and the underlying consistency it is built to satisfy is the Bellman expectation equation. Over enough repeated experience, these estimates converge toward accurate values, at which point the agent can act well simply by choosing whichever action has the highest estimated value in its current state.

A deliberately brief introduction

This lesson introduces the vocabulary and the core loop that every reinforcement learning method builds on, using Q-learning as one concrete, foundational example of how an agent actually turns experience into improved behaviour. The field extends considerably further than this single lesson covers — combining reinforcement learning with the neural networks covered earlier in this roadmap to handle environments with far too many possible states to track individually is itself an entire area of further study, touched on only briefly here as the natural next step for anyone who wants to go beyond this roadmap’s scope.