Q-Learning Explained

The 'Q' in Q-Learning stands for Quality. We want to know the quality of an action $a$ in a state $s$. We store these values in a Q-Table, a grid where rows are states and columns are actions. Initially, the table is full of zeros (the agent knows nothing). As the agent explores, it fills the table with the 'Expected Future Return' for every action, eventually creating a complete map of the best possible moves for any situation.

What makes Q-Learning special is that it is Off-Policy. This means it learns about the Optimal Policy (the best way to win) while following a Behavior Policy (which includes random exploration). The update rule uses the max of the next state's Q-values. It assumes that in the future, it will act perfectly, even if right now it is still exploring. This allows the agent to learn the 'true' best strategy even from a path of mistakes.

If an agent finds a small reward, it might stop looking for a bigger one. This is the 'Local Optima' trap. To avoid this, we use $epsilon$-Greedy Exploration. With a probability of $epsilon$ (usually 0.1), the agent ignores its table and takes a Random Action. With a probability of $1-epsilon$, it takes the best action it knows. Over time, we usually 'decay' $epsilon$, so the agent explores less as it becomes more confident in its knowledge.

Knowing how good a state is isn't enough; you need to know which ACTION is best. Q-Learning is the legendary algorithm that allows an agent to find the optimal strategy through pure trial and error.

In Q-Learning, we maintain a 'Q-Table' that stores the value of every action in every state. We update it by looking at the maximum possible value we could get in the next state.

Q-Learning is 'Off-Policy'. It learns about the optimal policy while the agent is actually exploring randomly. It's like learning to win while playing for fun.

Checkpoint: What does the 'max' operator in the Q-Learning update rule represent?

• →Randomly picking a move
• →Assuming the agent will take the absolute best possible move in the next state

To ensure we find the best path, we use 'Epsilon-Greedy'. With a small probability (ε), we pick a random move. This 'Exploration' prevents the agent from getting stuck in a mediocre strategy.

Q-Learning is the foundation of almost all Deep RL. By replacing the Q-Table with a Neural Network, we get the world-famous Deep Q-Network (DQN).

Checkpoint: If ε (epsilon) is set to 0.1, how often will the agent explore a random action?

• →Never
• →10% of the time

Q-Learning mastered! You've learned to optimize actions. Ready to scale this to massive worlds with Deep Q-Networks?

Run a Real Q-Learning Update. Finish the Q-learning update rule, blending the old estimate with the new observed target.