Back to Articles Generative AI • 15 min read

Training a GAN from Scratch: Mode Collapse, Exploding Gradients, and My First Generated Image

Generative Adversarial Networks (GANs) are famous for creating hyper-realistic synthetic human faces and artwork. Intrigued by Ian Goodfellow's original 2014 paper, I decided to build a simple Deep Convolutional GAN (DCGAN) in PyTorch to generate synthetic hand-written digits using the MNIST dataset. I thought it would take a few hours.

Instead, I spent three painful days dealing with training instability, zero gradients, and the notorious **Mode Collapse** problem, where my generator refused to create anything except blurry white blobs that looked like the digit '1'.

---

1. The Nightmarish Mode Collapse

In a GAN, two neural networks play a zero-sum game: the **Generator ($G$)** tries to synthesize realistic images from random noise, while the **Discriminator ($D$)** acts as an inspector trying to tell real images from fake ones.

On my first training run, my Discriminator learned too quickly. It achieved 100% accuracy within 5 epochs, overwhelming the Generator. The Generator got stuck in a local minimum: it realized that outputting a blurry digit '1' scored a tiny 2% chance of tricking the Discriminator, while outputting anything else got 0%. So it collapsed and produced **only digit 1s for thousands of batches**!

---

2. The Code Changes That Fixed It

To stabilize the adversarial game, I made three critical structural changes to my PyTorch training loop:

• Lower Learning Rate for Discriminator: I slowed down $D$'s learning rate relative to $G$ so $G$ had time to adapt.
• LeakyReLU Activation: Replacing standard ReLU with `LeakyReLU(0.2)` in the Discriminator prevented dying neuron gradients.
• Label Smoothing: Replacing hard targets (1.0 for real, 0.0 for fake) with smooth targets (0.9 for real) prevented $D$ from becoming overly confident.

import torch
import torch.nn as nn

# Discriminator with LeakyReLU and Batch Normalization
class Discriminator(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=4, stride=2, padding=1),
            nn.LeakyReLU(0.2, inplace=True), # LeakyReLU prevents gradient death
            
            nn.Conv2d(64, 128, kernel_size=4, stride=2, padding=1),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.2, inplace=True),
            
            nn.Flatten(),
            nn.Linear(128 * 7 * 7, 1)
        )
    def forward(self, x):
        return self.net(x)

# Training loop step showing Label Smoothing fix
criterion = nn.BCEWithLogitsLoss()
lr_g = 0.0002
lr_d = 0.00005 # Discriminator learning rate slowed down 4x!

# Inside training batch:
# Use 0.9 instead of 1.0 to prevent Discriminator over-confidence
real_labels = torch.full((batch_size, 1), 0.9).cuda() 
fake_labels = torch.zeros((batch_size, 1)).cuda()
            

---

3. Seeing My First Generated Digits

On epoch 25 after applying the learning rate ratio and label smoothing, I looked at the saved image grid generated from random noise vector $z$. The result was breathtaking:

[Epoch 01] Generator output: Uniform gray noise static
[Epoch 05] Generator output: Mode collapse (blurry digit 1 everywhere)
[Epoch 15] Generator output: Rough faint loops and vertical strokes
[Epoch 30] Generator output: Crisp, distinct, novel handwritten 0s, 3s, 7s, and 9s!
            

Seeing crisp handwritten numbers appear out of pure Gaussian noise—numbers that were never drawn by a human being—felt like magic. It converted me from a passive observer into a lifelong fan of generative modeling.

GAN Symptom	Root Cause	Effective Fix
Mode Collapse	Generator finds single easy trick to fool Discriminator	Use Minibatch Discrimination or Wasserstein Loss (WGAN-GP).
Dying Gradients	Discriminator reaches 100% accuracy early	Use LeakyReLU(0.2), lower $D$ learning rate, smooth labels to 0.9.

Training a GAN is like balancing a pencil on its tip. When both networks advance at the exact same pace, the results are extraordinary.