Back to Articles Reinforcement Learning • 15 min read

Training a Q-Learning Agent From Scratch: 500 Failure Episodes and One Eureka Moment

Reinforcement Learning (RL) sounds incredible when you read about DeepMind beating world champions at Go or chess. But the first time I actually tried to implement a basic Q-learning algorithm to solve OpenAI Gym's `FrozenLake-v1` grid environment, I spent four hours staring at a terminal outputting minus ones while my digital agent repeatedly walked off a cliff into an icy hole.

For the first 500 episodes, my agent learned absolutely nothing. It was frustrating, perplexing, and deeply humbling. Here is how I debugged my Q-learning implementation and what I learned about the fragile balance of exploration versus exploitation.

---

1. The Bug That Made My Agent Randomly Wander Forever

In Q-learning, the agent uses an $\epsilon$-greedy strategy: with probability $\epsilon$, it takes a completely random action to explore the map; otherwise, it chooses the action with the highest estimated value in its Q-table.

My mistake was simple but fatal: I declared `epsilon = 1.0` at the top of my code, but I forgot to decay it after each training episode. My agent was taking 100% random actions for 10,000 consecutive episodes! Even when it stumbled across the goal flag by luck, it never used that stored Q-value on the next episode because it was still rolling a random die every single step.

Here is the corrected Python implementation I wrote to track Q-table updates and epsilon decay:

import numpy as np
import gym

env = gym.make("FrozenLake-v1", is_slippery=False)
q_table = np.zeros([env.observation_space.n, env.action_space.n])

# Hyperparameters I tuned by trial and error
alpha = 0.8         # Learning rate
gamma = 0.95        # Discount factor (future reward value)
epsilon = 1.0       # Initial exploration rate
max_epsilon = 1.0
min_epsilon = 0.01
decay_rate = 0.005  # Epsilon decay rate

episodes = 2000
for episode in range(episodes):
    state = env.reset()[0]
    done = False
    
    while not done:
        # Epsilon-greedy action selection
        if np.random.uniform(0, 1) < epsilon:
            action = env.action_space.sample() # Explore
        else:
            action = np.argmax(q_table[state, :]) # Exploit
            
        new_state, reward, done, truncated, info = env.step(action)
        
        # Q-learning Bellman update equation
        old_value = q_table[state, action]
        next_max = np.max(q_table[new_state, :])
        
        q_table[state, action] = old_value + alpha * (reward + gamma * next_max - old_value)
        state = new_state
        
    # CRITICAL FIX: Exponentially decay epsilon so the agent starts exploiting learned rewards
    epsilon = min_epsilon + (max_epsilon - min_epsilon) * np.exp(-decay_rate * episode)

print("Training finished! Final learned Q-Table values for State 0:")
print(q_table[0, :])
            

---

2. The Eureka Moment: Episode 650

Once I added the decay rate, I ran the script again and plotted the total reward per episode. The transition was mind-blowing:

Terminal Output Log:
[Episode 0050] Reward: 0.00 | Epsilon: 0.778 | Avg Steps: 6.2 (Random movement, falling into holes)
[Episode 0300] Reward: 0.00 | Epsilon: 0.223 | Avg Steps: 8.4 (Stumbling near goal)
[Episode 0650] Reward: 1.00 | Epsilon: 0.048 | Avg Steps: 6.0 (Eureka! Optimal path discovered)
[Episode 1000] Reward: 1.00 | Epsilon: 0.010 | Avg Steps: 6.0 (100% win rate on deterministic grid)
            

Around episode 600, as epsilon dropped below 10%, the Bellman backup updates propagated backward from the goal state all the way to the starting square. Watching the terminal change from endless zeroes to a unbroken streak of `Reward: 1.00` felt like watching a digital organism learn to walk.

---

Key Takeaways From My RL Experiments

• Reward Shaping is Hard: If you give an agent a tiny reward for surviving each step, it will often learn to run around in circles forever to farm infinite points rather than completing the maze.
• Discount Factor Matters ($\gamma$): Setting $\gamma = 0$ makes the agent completely short-sighted (only caring about immediate rewards), while $\gamma = 0.99$ makes it look dozens of steps into the future.
• Q-Tables Don't Scale: A Q-table works great for 16 discrete states on FrozenLake. But for chess or Atari games with $10^{170}$ states, you need a Deep Q-Network (DQN) that uses a neural network as a function approximator.

RL is completely different from supervised learning—you aren't giving the model answers, you're just giving it a score and letting it discover the rules through trial, error, and math.