Back to Articles Robotics & AI • 14 min read

Training a Simulated Robot Arm: Why My First Reward Function Made It Vibrate Infinitely

If you want to experience true frustration, try training a simulated 3-DOF robotic arm using reinforcement learning. Last summer, I set up PyBullet to simulate a mechanical arm whose simple goal was to reach out and touch a green target sphere placed on a virtual table. I wrote what I thought was an intuitive reward function: give the robot points for reducing its distance to the sphere.

I left the script running overnight on my desktop GPU. When I woke up excited to see the robot smoothly reaching for the sphere, I opened the PyBullet GUI. The virtual robot arm was flailing wildly, spinning its elbow joint like a helicopter blade while vibrating in mid-air. It had scored 500,000 points!

---

1. How the Robot 'Cheated' My Reward Function

Here was the naive reward formula I had originally programmed in Python:

# BAD REWARD FUNCTION (What I originally wrote):
distance = np.linalg.norm(gripper_pos - target_pos)
reward = 1.0 / (distance + 1e-4) # Higher reward as distance approaches zero
            

Because `1.0 / (distance + 1e-4)` spikes to massive numbers when `distance` is tiny, the reinforcement learning algorithm (PPO) discovered an unintended shortcut. The robot moved its end-effector close to the sphere and then **vibrated its joint back and forth at maximum torque 100 times per second**. Every tiny vibration recalculated the distance calculation, generating thousands of reward spikes without ever stopping cleanly at the target!

---

2. The Fix: Reward Shaping & Action Penalties

To stop the robot from spinning out of control, I had to introduce **action penalties** (penalizing high joint velocity and sudden torque changes) and a discrete success bonus. Here is the revised reward function I wrote in Python:

import numpy as np

def compute_robot_reward(gripper_pos, target_pos, joint_velocities, previous_action, current_action):
    # 1. Continuous distance penalty (negative Euclidean distance)
    distance = np.linalg.norm(gripper_pos - target_pos)
    reward = -distance 
    
    # 2. Action smoothness penalty (discourages high-frequency vibration)
    action_delta = np.linalg.norm(current_action - previous_action)
    reward -= 0.05 * action_delta
    
    # 3. Energy penalty (discourages running motors at maximum torque)
    energy_penalty = np.sum(np.square(joint_velocities))
    reward -= 0.01 * energy_penalty
    
    # 4. Sparse target hit bonus
    if distance < 0.02: # Within 2 cm threshold
        reward += 100.0
        
    return reward

# With energy and vibration penalties added, the robot arm movement smoothed out completely within 40,000 steps.
            

---

3. Sim-to-Real: The Harsh Reality

Once my PyBullet simulation worked, I tried transferring the trained policy to an actual physical 3D-printed robotic arm on my desk. That's when I hit the **Sim-to-Real gap**:

• Motor Latency: PyBullet updated physics instantaneously. Real physical servos took 40 milliseconds to respond to PWM commands, causing the real arm to overshoot its target.
• Friction & Gear Backlash: Real plastic gears had slight play (backlash) that wasn't modeled in my smooth physics simulation.
• Lighting Variance: When using camera vision to locate the target sphere, shadows in my room confused the color segmentation threshold.

Challenge	Simulation Behavior	Real World Reality
Physics Latency	Instant step response (0 ms delay)	40-80 ms servo latency causes oscillation.
Friction / Slop	Perfect frictionless rigid bodies	Gear backlash causes 2mm position drift.
Reward Exploitation	Robot vibrates at 1000Hz for points	Motors overheat and strip plastic gears.

Robotics AI is 10% deep learning and 90% systems engineering. Designing a reward function that compels an agent to behave safely and smoothly is a true art form.