MDP Foundations in AI & Artificial Intelligence

The Markov Property states that 'the future is independent of the past given the present.' In an MDP, the current State must be sufficient to make the optimal decision. If an agent needs to know its previous three positions to decide its next move, the state isn't Markov. We fix this by including the necessary history directly into the current state (e.g., adding velocity to position), ensuring the agent always has the 'context' it needs without needing an infinite memory.

Every Reinforcement Learning problem can be mapped to an MDP Tuple (S, A, P, R, γ). S is the State Space (all possible configurations). A is the Action Space (all possible moves). P is the Transition Function, which defines the probability of moving from one state to another. R is the Reward Function, defining the immediate payoff. Finally, γ (Gamma) is the Discount Factor, which determines how much the agent values future rewards compared to immediate ones.

The real world is rarely 100% predictable. In an MDP, the Transition Function $P(s' | s, a)$ captures this uncertainty. If a robot tries to 'Move Forward,' there might be an 80% chance it succeeds, a 10% chance it slips left, and a 10% chance it slips right. By modeling these Stochastic Dynamics, RL agents learn to be robust to unexpected outcomes, choosing the path that has the highest *expected* reward rather than the most optimistic one.

Behind every RL environment is a mathematical framework called the Markov Decision Process (MDP). It provides the formal language we use to describe states, actions, and the 'laws of physics' of our digital world.

The 'Markov Property' is the core assumption: the future only depends on the present state, not on how we got there. The current state contains all the information needed to make the next decision.

An MDP is defined by a 5-tuple: States (S), Actions (A), Transitions (P), Rewards (R), and the Discount Factor (γ).

Checkpoint: What is the 'Markov Property'?

• →The agent must remember every single past action
• →The future state depends only on the current state and action, not the history

The Transition Function P(s' | s, a) tells us the probability of landing in state s' if we take action 'a' in state 's'. It represents the 'Dynamics' of the environment.

MDPs turn 'Learning' into 'Math'. By solving an MDP, we find the optimal policy that maximizes the expected reward for our agent.

Checkpoint: In the MDP 5-tuple, what does the 'P' represent?

• →Policy
• →Transition Probability Function

MDP foundations mastered! You've learned the math of the environment. Ready to explore how we define 'Success' with Rewards and Returns?

Look Up a Real MDP Transition. Finish looking up the transition probability for a given (state, action, next_state) triple.