Why My 20-Layer ResNet Refused to Train (And How Batch Norm & Skip Connections Saved It)

I spent three straight days staring at a terminal screen where my loss curve refused to move. The training loss was frozen at exactly 2.302, epoch after epoch. On CIFAR-10—a dataset with 10 balanced classes—a cross-entropy loss of 2.302 means -ln(0.1). In plain terms, my custom 20-layer residual network wasn't learning anything; it was performing uniform random guessing.

I tweaked the learning rate from 0.1 down to 1e-5. I tried Adam, SGD with momentum, weight decay, and Xavier initialization. Nothing worked. The loss line stayed flat like a dead heart monitor. I realized I couldn't just keep swapping high-level hyperparameters in PyTorch like a black box—I had to pull back the hood and look at what was happening to the numbers flowing through the network.

The 20-Layer Gradient Black Hole

To diagnose why layer 20 refused to update, I registered forward and backward hooks across every single layer to log the mean and standard deviation of both activations and weight gradients. What I found in the logs made me choke on my coffee.

By the time backpropagation traveled from layer 20 back down to layer 1, the gradient signal had vanished into mathematical insignificance. Because each matrix multiplication scaled down the incoming gradients slightly, multiplying those fractional derivatives 20 times in a row squashed the backprop signal to zero. Early layers were essentially frozen, unable to extract basic low-level features like edges or color blobs. Worse yet, as inputs passed through successive ReLU layers, activations collapsed toward zero—a phenomenon known as internal covariate shift.

Building Batch Normalization From Scratch

Rather than dropping in torch.nn.BatchNorm2d and calling it a day, I sat down with paper and a Jupyter notebook to implement batch normalization from scratch. I wanted to see every single step of how normalizing intermediate feature maps across mini-batches stabilized the activations.

Here is the exact PyTorch implementation I wrote to test my understanding, complete with comments on the bugs I ran into along the way:

import torch
import torch.nn as nn

class ScratchBatchNorm1d(nn.Module):
    def __init__(self, num_features, eps=1e-5, momentum=0.1):
        super().__init__()
        self.eps = eps
        self.momentum = momentum
        
        # Learnable scale (gamma) initialized to 1 and shift (beta) initialized to 0
        self.gamma = nn.Parameter(torch.ones(num_features))
        self.beta = nn.Parameter(torch.zeros(num_features))
        
        # Running statistics kept for evaluation phase (non-trainable buffers)
        self.register_buffer('running_mean', torch.zeros(num_features))
        self.register_buffer('running_var', torch.ones(num_features))

    def forward(self, x):
        if self.training:
            # Step 1: Calculate mini-batch mean and variance along batch dimension (dim 0)
            # LESSON LEARNED: Using unbiased=False matches PyTorch official BatchNorm
            batch_mean = x.mean(dim=0)
            batch_var = x.var(dim=0, unbiased=False)
            
            # Step 2: Normalize activations to 0 mean and unit variance
            # Epsilon prevents zero-division when feature variance collapses
            x_hat = (x - batch_mean) / torch.sqrt(batch_var + self.eps)
            
            # Step 3: Update running statistics for inference with exponential moving average
            # MUST wrap in torch.no_grad() or autograd tracks the buffer history and leaks memory!
            with torch.no_grad():
                self.running_mean = (1 - self.momentum) * self.running_mean + self.momentum * batch_mean
                self.running_var = (1 - self.momentum) * self.running_var + self.momentum * batch_var
        else:
            # During eval(), use saved dataset-level running statistics
            x_hat = (x - self.running_mean) / torch.sqrt(self.running_var + self.eps)
            
        # Step 4: Scale and shift - allows network to learn non-identity transform if optimal
        return self.gamma * x_hat + self.beta

When I inserted this custom batch norm layer after each linear transformation in my deep network, activations no longer decayed to zero or exploded to infinity. Every layer received inputs centered around zero with stable variance, allowing gradients to flow back predictably without hitting extreme saturation regions.

Why Skip Connections Feel Like Cheating (But Make Mathematical Sense)

Even with batch norm, stacking 20 layers created another subtle problem: degradation. You would assume that adding more layers can only increase or equal a network's capacity. Yet in practice, my deeper 20-layer model was getting higher training error than an 8-layer model. That made no intuitive sense—until I realized how hard it is for deep layers to learn even simple identity functions H(x) = x.

When I first saw skip connections in ResNet papers—adding input x directly to block output F(x) to produce y = F(x) + x—it felt like a lazy cheat. I thought, "If your deep block couldn't figure out a meaningful representation, you're just bypassing it!"

Then I worked through the chain rule on paper and had an 'aha' moment. Consider the backpropagation gradient of the loss L with respect to input x through a residual block:

Notice that + 1 term? That single addition changes everything. Expanding the equation gives:

Even if the weight layers in F(x) have vanishing gradients where ∂F(x)/∂x approaches zero, the backpropagated error signal ∂L / ∂y flows straight through the + 1 term completely unimpeded! Skip connections create a gradient superhighway back to layer 1. The network doesn't have to learn identity functions from scratch; it only needs to learn residual adjustments F(x) = 0 around identity mappings.

The Breakthrough Moment

I refactored my entire model to combine both elements: residual blocks with skip connections and batch normalization before each non-linear activation. I ran the exact same training script that had previously stalled at loss 2.302.

Within 5 minutes, the loss plunged under 1.0 and training accuracy shot up past 90%. What felt like an impossible debugging wall was resolved once I stopped treating deep architectures as black boxes and understood how variance propagation and residual gradient highways keep backprop healthy across deep network graphs.