Back to Articles Mathematics & ML • 15 min read

Demystifying Gradient Descent: How I Built It in Pure Python Without Libraries

For a long time in my coding journey, I treated machine learning like a black box. I would import `LinearRegression` from `sklearn`, call `.fit(X, y)`, and get a trained model back. But whenever something broke or my model overfit, I had zero understanding of *why*. I decided to stop importing third-party libraries and write Gradient Descent completely from scratch using only raw Python lists and loops.

Staring at the mathematical notation $ heta_{j} := heta_{j} - lpha rac{\partial}{\partial heta_{j}} J( heta)$ in textbook chapters felt dry and intimidating. But once I visualised it as a ball rolling down a foggy hill towards a valley floor, the calculus clicked in my mind immediately.

---

1. My Intuition: Rolling Down a Foggy Hill

Imagine standing on a mountain peak surrounded by thick fog. You want to reach the lowest valley floor (minimum loss $J$), but you can't see the destination. What do you do?

You feel the slope under your boots. If the ground slopes downward to your right, you take a step to the right. If the slope is steep, you take a bigger step; as the ground flattens out near the valley bottom, you take smaller, cautious steps so you don't overshoot. That slope is the **Derivative (Gradient)**, and your step size is the **Learning Rate ($lpha$)**.

---

2. Writing Gradient Descent in Pure Python

Here is the exact Python script I wrote to minimize Mean Squared Error (MSE) for a linear equation $y = m x + b$ without using PyTorch, NumPy, or Scikit-Learn:

# My synthetic dataset: y = 2x + 1 (Target slope m=2.0, intercept b=1.0)
X = [1.0, 2.0, 3.0, 4.0, 5.0]
y = [3.0, 5.0, 7.0, 9.0, 11.0]

# My initial parameters (starting from 0.0)
m = 0.0
b = 0.0
alpha = 0.05 # Learning rate
n = float(len(X))

print("Starting Gradient Descent Optimization Loop...")

for epoch in range(501):
    # 1. Calculate predictions: y_pred = m*x + b
    y_pred = [m * x_i + b for x_i in X]
    
    # 2. Compute Mean Squared Error Loss
    loss = sum((y_i - y_p)**2 for y_i, y_p in zip(y, y_pred)) / n
    
    # 3. Compute Partial Derivatives w.r.t slope (m) and intercept (b)
    dm = (-2 / n) * sum(x_i * (y_i - y_p) for x_i, y_p in zip(X, y_pred))
    db = (-2 / n) * sum(y_i - y_p for y_i, y_p in zip(y, y_pred))
    
    # 4. Update parameters opposite to gradient direction
    m -= alpha * dm
    b -= alpha * db
    
    if epoch % 100 == 0:
        print(f"Epoch {epoch:03d} | Loss: {loss:.6f} | Learned m: {m:.3f}, b: {b:.3f}")

# Final Result: Epoch 500 | Loss: 0.000004 | Learned m: 1.999, b: 1.002
            

---

3. What Happened When I Exploded My Learning Rate

While experimenting with my custom script, I set `alpha = 1.2` just to see how fast it would train. My terminal output went chaotic instantly:

Terminal Output Log:
Epoch 000 | Loss: 35.0000 | Learned m: 14.400, b: 5.600
Epoch 001 | Loss: 892.4000 | Learned m: -62.100, b: -21.400
Epoch 002 | Loss: 24102.1200 | Learned m: 345.800, b: 118.200
Epoch 003 | OverflowError: float math range exceeded!
            

Because my step size was way too large, my updates overshot the valley floor completely, bouncing higher and higher up the opposite canyon walls until my computer threw a floating point overflow error. That single experiment taught me why learning rate schedulers, Adam optimizers, and gradient clipping are vital when training deep neural networks.

Learning Rate ($lpha$)	Observed Behavior	Practical Result
Too Large ($lpha = 1.0$)	Overshoots minimum loss; values diverge to infinity.	`NaN` loss error / Training explosion.
Too Small ($lpha = 0.00001$)	Takes tiny steps; gets stuck in shallow local minima.	Training takes 100,000+ unnecessary epochs.
Optimal ($lpha = 0.01 - 0.05$)	Smoothly decays steps as gradient slope flattens.	Fast, stable convergence to global minimum.

Writing the core calculus loop by hand completely eliminated my fear of machine learning math. It proved to me that AI isn't black magic—it's basic calculus optimized through code loops.