Rewards & Returns in AI & Artificial Intelligence

The Reward Hypothesis is a central idea in RL: all what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called Reward). Whether you want an AI to win at Go or drive a car, you must reduce that complex goal into a stream of numbers. If the reward function is perfect, maximizing it *is* the same as solving the problem.

Why do we discount the future? Mathematically, the Discount Factor (γ) ensures that the sum of rewards (the Return) doesn't become infinite in tasks that never end. Perceptually, it models the 'Uncertainty' of the future. A reward today is worth more than a reward tomorrow because the environment might change. By setting γ, we control the agent's Horizon. A low γ makes the agent impulsive; a high γ makes it a strategic planner.

AI is incredibly good at finding 'Shortcuts'. Reward Hacking occurs when an agent finds a way to get high rewards without actually performing the intended task. A classic example is an agent designed to play a racing game that discovers it can get infinite points by driving in circles at the start line rather than finishing the race. To prevent this, rewards must be designed to be Sparse (only at the goal) or carefully Shaped to prevent unintended behaviors.

How do you tell an AI what you want? You don't give it instructions; you give it Rewards. 'Reward Engineering' is the process of defining the signals that guide your agent's evolution.

The 'Reward' is immediate feedback. But the agent cares about the 'Return' (G)—the sum of all future rewards. To prioritize short-term wins, we use the Discount Factor (γ).

If γ is 0, the agent is 'Myopic' (short-sighted). It only cares about the next reward. If γ is near 1, it's 'Far-sighted', valuing the distant future as much as the present.

Checkpoint: What happens to the importance of a reward received 10 steps in the future as you decrease the value of γ (Gamma)?

• →It becomes more important
• →It becomes less important

Poorly designed rewards lead to 'Reward Hacking'. An agent might find a shortcut to get points without actually solving the task, like a vacuum robot that just spins in circles to get 'movement points'.

Mastering rewards and returns is the difference between an AI that works and an AI that breaks. It is the language of desire for machines.

Checkpoint: In the equation $G_t = R_{t+1} + γR_{t+2} + ...$, what does the $G_t$ represent?

• →Immediate Reward
• →Cumulative Discounted Return

Rewards and Returns mastered! You've learned to define success. Ready to start solving these MDPs with Dynamic Programming?

Compute a Real Discounted Return. Finish computing the discounted return as a weighted sum of future rewards.