
The Mountain Car Problem is a classic challenge in the field of reinforcement learning. It was first introduced by Andrew Moore in 1990 as a simple yet realistic problem to test the capabilities of machine learning algorithms.
The problem involves a car that is stuck between two hills and must learn to drive up the steep hill on the right. The car's position and velocity are the only features that are observable, and the goal is to drive the car up the hill.
The Mountain Car Problem is often used as a benchmark to evaluate the performance of reinforcement learning algorithms.
Take a look at this: Sequence Labeling
What Is the Mountain Car Problem?
The Mountain Car Problem is a classic challenge in the field of reinforcement learning. It's a simple yet tricky game where a car must drive up a mountain, but its engine is not strong enough to make it to the top in one go.
The car's engine is too weak to accelerate all the way uphill from a stop, so the player must "swing" the car back and forth between the valley to gather enough momentum for the climb. This is the only way to succeed in the game.
The goal is to drive the car to the right peak where the yellow flag stands, but it's not just a matter of moving the car in the right direction. The player must use the correct control inputs, such as moving left, no action, or moving right, to navigate the car to the top.
The Mountain Car environment has a dead simple control: move left, no action, or move right. This makes it a great starting point for learning about reinforcement learning and AI.
The code for playing the game manually is straightforward, using j (move left) and k (move right) as control inputs. This code can be used as a starting point for building an AI agent to solve the game.
The Mountain Car Problem can be summarized as follows:
Background and Basics
The Mountain Car problem is a classic example of a reinforcement learning problem. In this environment, an agent, represented by a car, must navigate a one-dimensional mountainous terrain to reach a goal.
The agent's primary goal is to maximize its cumulative reward, which is a numerical value received after taking an action. The reward is defined in a way that the agent is punished for not being in a position we want it to be, and the faster it reaches the goal, the better.
The agent's actions are limited to moving left, moving right, or staying still. The environment is a black-box world, and the agent perceives it through observations of its X position and velocity.
Here are the main concepts in the Mountain Car problem:
- Environment: the Mountain Car game itself, featuring a car starting on a valley and trying to reach a flag on a “hill” at the screen’s top right corner.
- State: the agent’s current position and the current layout of the environment at a moment in time.
- Action: what the agent chooses to do in each state (move left, move right, or stay still).
- Reward: the numerical value that the agent receives after taking an action.
The agent's performance is measured by its total reward, which is the sum of rewards from one episode. If it reaches the goal, it receives a reward of -110, which is a great result. If it doesn't reach the goal, it is punished for 200 steps, resulting in a reward of -200.
Implementation and Visualization
To implement the Mountain Car problem, we need to discretize the observation space into bins. The Gym library provides a method to create an environment, and we can get the stats of the observation space using `env.observation_space`. The observation space has two dimensions: position and velocity, and we can divide it into 30 bins for each dimension.
The MDP has 180 states (18 * 10 = 180 states) and we can use the `value_iteration()` method to find the optimal value function V*. The `theta` argument can be used to stop the value iteration earlier or later. We can also use the `get_optimal_action()` method to get the optimal action to take given a state.
The `solve()` method can be used to let the AI agent play the Mountain Car game using the computed optimal policy. We can set `record` to True to record and save the game as a video.
Python Implementation
The Python implementation of the Mountain Car environment is a crucial part of making the game work. The implementation uses the Gym library to create an environment and the Matplotlib library to visualize the result.
The environment is created using `env = gym.make('MountainCar-v0')`, which returns an instance of the MountainCar-v0 environment. The observation space of the environment is continuous, but we need to discretize it before our agents can learn. We divide the observation space into 30 "bins" for each dimension.

The observation space has two dimensions: the position and velocity of the car. Each "slot" in the Q-table is characterized by a normalized position and velocity. The more "bins" we decide to divide our observation space into, the more information from the original observation space we can retain.
Here's a breakdown of the observation space discretization:
The episode termination criteria are met when either the position of the car is equal to or greater than the goal position (0.5), or the 200 steps limit is reached. This means that the agent will learn to navigate the environment until it reaches the goal or runs out of steps.
The Q-tables are three-dimensional, with a size of 30 x 30 x 3, representing the velocity bins x position bins x action space size. The Q-learning and SARSA agents select actions based on an epsilon-greedy policy, where epsilon decreases by (number of episodes — 2)/(number of episodes) after each episode.
Visual Representation
The Mountain Car problem can be visualized using the OpenAI Gym library, which provides an implementation of the environment.
We don't need to create our own Mountain Car environment, as the OpenAI Gym library already has it implemented.
A random agent in our environment takes random actions, but we can use the library to create a more intelligent agent.
The Mountain Car environment is represented by a green area, which shows the hilly terrain the car is traversing.
The black circle represents the center of the car, and the orange arrow shows the applied control.
Interestingly, the applied control generally has the same sign as the velocity and is 'bang-bang', meaning it wants to be at its maximum possible magnitude.
Solving the Problem
The final step is to let the AI agent trained using value iteration play the game. To do this, we run the policy extraction code, which finds the optimal solution for our AI agent to take given the current state.
The policy extraction code is equivalent to finding the argmax of the Q-table on the axis of action. This is shown in the code snippet, where we use the np.argmax function to find the maximum value in the Q-table for each state.
Here's a summary of the policy extraction process:
This table shows the optimal action for each state, which is determined by finding the maximum value in the Q-table for each state.
Policy Extraction
Policy Extraction is a crucial step in solving the problem of reinforcement learning. It's the process of finding the optimal policy, which is the action to take given the current state.
The optimal policy is the action that maximizes the value function V(s), which is the expected future reward from the current state s. In Q-learning, this is equivalent to finding the argmax of the Q-table on the axis of action.
To find the optimal policy, you can use the following formula: pi(s) = np.argmax(mdp.qa_table, axis=1). This will give you the optimal action to take for each state.
Here's an example of what the optimal policy might look like:
In this example, the optimal policy is to take the left action at state 0, no action at state 1, and the right action at state 2. This is just one possible optimal policy, and there may be others.
The optimal policy is not just a list of actions to take, but also a way of making decisions in the future. By following the optimal policy, the agent can make the best decisions possible to maximize the expected future reward.
In summary, policy extraction is an important step in solving the problem of reinforcement learning. It involves finding the optimal policy, which is the action to take given the current state.
Solving the Minimum-Time Problem with Dymos
The Minimum-Time Problem is a classic challenge in optimization, and Dymos is a powerful tool for solving it. Dymos uses a direct transcription approach to solve the problem, which is a game-changer for many applications.
Dymos can solve the Minimum-Time Problem by transcribing the problem into a nonlinear programming problem, which can be solved using standard optimization algorithms. This approach allows for a high degree of flexibility and accuracy.
The key to solving the Minimum-Time Problem with Dymos is to define the problem correctly, including the objective function, constraints, and initial conditions. This requires careful consideration of the problem's dynamics and the desired outcome.
Dymos can handle complex problems with multiple variables and constraints, making it a valuable tool for many applications. By leveraging the power of Dymos, you can solve the Minimum-Time Problem with confidence and accuracy.
Reinforcement Learning Concepts
Reinforcement learning is a field of machine learning where an agent generates its own training dataset by engaging with the environment to maximize its cumulative reward during an episode of interaction with the environment.
The main concepts in reinforcement learning are crucial to understand. The environment is the simulation containing the agent, in this case, the Mountain Car game itself. The agent's current position and the current layout of the environment at a moment in time is known as the state.
In our Mountain Car game, the agent can move left, move right, or stay still, which is known as the action. The numerical value that the agent receives after taking action is called the reward.
Here are the main concepts summarized:
- Environment: the simulation containing the agent (Mountain Car game)
- State: the agent's current position and the current layout of the environment
- Action: what the agent chooses to do in each state (move left, move right, or stay still)
- Reward: the numerical value that the agent receives after taking action
As the agent interacts with the environment, it must balance exploration and exploitation. Exploration is desirable during early training, allowing the agent to "sample" as many scenarios as possible.
Additional reading: Learning with Errors Problem
Results and Conclusion
The random agent's performance was not good, as it randomly chose an action at each time step and didn't learn, resulting in a constant reward of -200.
The Q-learning agent managed to get a notable number of successful attempts from around episode 5,000 onwards, with most attempts taking around 140 steps to reach the goal position.
The SARSA agent, on the other hand, performed even better, with more successful attempts than the Q-learning agent and an average of fewer than 120 steps to reach the goal position.
Results
The random agent's performance was dismal, with a constant reward of -200, indicating it stopped each episode after reaching the 200 steps limit.
The Q-learning agent started to show improvement around episode 5,000, with a notable number of successful attempts.
It took the Q-learning agent an average of 140 steps to reach the goal position on its successful attempts.
The SARSA agent outperformed the Q-learning agent, achieving more successful attempts and averaging fewer than 120 steps to reach the goal position.
The SARSA agent's superior performance is likely due to its "look-ahead" ability, allowing it to avoid disadvantageous actions not just in the current step but also in future ones.
Conclusion
In the Mountain Car game, SARSA outperformed Q-learning due to its ability to consider the reward of more than one state-action pair.
This suggests that SARSA is a more robust algorithm for this specific task.
The Mountain Car game is a challenging problem that requires careful consideration of multiple factors, and SARSA's ability to handle this complexity is a key factor in its success.
In fact, SARSA worked better than Q-learning in this case, which is an important finding for anyone looking to apply reinforcement learning to similar problems.
The author plans to explore Deep Q-learning and Double Deep Q-learning in the next article, which could potentially lead to even better results.
Featured Images: pexels.com





