Reinforcement Learning: Making Robots Think
Pascual Vila
AI & Robotics Lead // Code Syllabus
"We are no longer programming exactly what the robot should do at every millisecond. Instead, we are programming the rules of the world, a goal, and letting the robot figure out the best way to achieve it."
The Markov Decision Process (MDP)
To apply RL to robotics, we formulate the problem as an MDP. The robot acts as the Agent, existing within a simulated or physical Environment.
At each timestep $t$, the agent receives the environment's current State ($s_t$), takes an Action ($a_t$), and receives a Reward ($r_&123;t + 1&125;$) along with the next state ($s_&123;t + 1&125;$).
The Bellman Equation & Q-Learning
How does the robot know which action to take? It uses a Policy ($\pi$). A common method to find the optimal policy is Q-Learning, which calculates the expected future reward for a given action in a given state.
Here, $\alpha$ is the learning rate, and $\gamma$ is the discount factor, ensuring the robot cares about long-term success rather than just immediate gratification.
View Architecture Tip: Sim2Real+
Domain Randomization: Training an RL agent on a physical robot takes years and risks destroying the hardware. We train in simulators (Isaac Gym, MuJoCo). To ensure the policy works in the real world (Sim2Real), we use Domain Randomization: constantly altering friction, mass, and lighting in the simulator so the neural network learns a robust, generalized policy.
🤖 Intelligence Briefing (FAQ)
What is Exploration vs Exploitation?
Exploration: The agent tries random actions to discover new paths and potentially higher rewards.
Exploitation: The agent uses its current Q-Table (learned knowledge) to perform the best known action.
An epsilon-greedy strategy balances this by slowly shifting from exploring to exploiting over time.
What is Reward Shaping?
If a robot is only rewarded at the very end of a maze (sparse reward), it may never learn. Reward shaping involves giving smaller, intermediate rewards (e.g., getting closer to the goal, standing upright) to guide the learning process.
Why use Deep Reinforcement Learning (DRL)?
A standard Q-Table becomes impossible to manage when there are continuous states (like precise joint angles on a robot arm). DRL replaces the table with a Deep Neural Network (DQN) to approximate the Q-values, allowing the agent to handle infinite state spaces.