Back to Articles Speech Processing • 15 min read

Running Whisper Locally: Noise Filters, Out-of-Memory Errors, and Audio Chunking

When OpenAI released Whisper in late 2022, I was determined to build a local Python application that could automatically transcribe my 2-hour university lecture recordings without sending private audio files to a paid cloud API. I downloaded the `whisper-medium` model, passed a raw `.wav` file recorded on my laptop microphone, and hit execute in my terminal.

Ten seconds later, my desktop GPU froze and crashed with a massive `torch.cuda.OutOfMemoryError`. When I downgraded to the lightweight `whisper-base` model to save GPU RAM, the transcription outputted hallucinated sentences in Welsh and repeated "Thank you for watching!" fifty times in a row because of background HVAC fan noise on my mic recording!

---

1. Why Raw Audio Crashed My GPU

I didn't realize that audio machine learning models don't process raw `.wav` sound pressure waves directly. They convert 1D time-domain audio samples (sampled at 16,000 Hz) into 2D **Log-Mel Spectrograms**—visual representations of frequency intensity over time.

Passing a 2-hour continuous spectrogram tensor directly into a Transformer encoder requires computing an $N imes N$ self-attention matrix over thousands of time frames, causing VRAM usage to explode exponentially. I realized I had to write a Python pipeline to chunk long audio into 30-second sliding windows with Voice Activity Detection (VAD).

---

2. The Noise Filter & Audio Chunking Script I Wrote

Here is the exact Python pipeline I built using `librosa`, `scipy`, and `whisper` that processed my long audio files cleanly without ever crashing VRAM:

import whisper
import librosa
import numpy as np
from scipy.signal import butter, lfilter

def butter_lowpass_filter(data, cutoff=4000, fs=16000, order=5):
    # Filter out high-frequency laptop fan hiss before feeding audio to Whisper
    nyq = 0.5 * fs
    normal_cutoff = cutoff / nyq
    b, a = butter(order, normal_cutoff, btype='low', analog=False)
    return lfilter(b, a, data)

def transcribe_long_audio_safely(file_path):
    print("Loading Whisper model on local GPU...")
    model = whisper.load_model("base", device="cuda")
    
    # Load audio sampled at 16kHz
    raw_audio, sr = librosa.load(file_path, sr=16000)
    
    # Filter background fan noise
    clean_audio = butter_lowpass_filter(raw_audio)
    
    # Chunk audio into 30-second frames (30 * 16000 = 480,000 samples)
    chunk_samples = 30 * 16000
    total_samples = len(clean_audio)
    
    full_transcript = []
    
    for start in range(0, total_samples, chunk_samples):
        end = min(start + chunk_samples, total_samples)
        chunk = clean_audio[start:end]
        
        # Pad short tail chunks to exactly 30 seconds
        if len(chunk) < chunk_samples:
            chunk = np.pad(chunk, (0, chunk_samples - len(chunk)))
            
        # Transcribe chunk safely within 2GB GPU RAM limit
        result = model.transcribe(chunk, fp16=True, language="en", condition_on_previous_text=False)
        full_transcript.append(result["text"])
        
    return " ".join(full_transcript)

# Setting condition_on_previous_text=False stopped Whisper from repeating hallucinated text loops!
            

---

3. Terminal Log & Debugging Comparison

Here is what my terminal output looked like before and after applying noise filtering and chunking:

BEFORE FIX (Raw 45-min file on whisper-medium):
[00:00.000 -> 00:15.000] torch.cuda.OutOfMemoryError: Tried to allocate 8.42 GiB (GPU 0; 8.00 GiB capacity)

BEFORE FIX (Raw noisy audio on whisper-base):
[00:01.000 -> 00:30.000] "Thank you for watching! Thank you for watching! Thank you for watching!" (Hallucination loop)

AFTER FIX (Low-pass filtered + 30s chunking + condition_on_previous_text=False):
[00:00.000 -> 00:30.000] "Welcome back everyone. Today we are going to dive into database indexing algorithms..."
[00:30.000 -> 01:00.000] "Specifically, we will look at B-trees vs Hash indexes and why disk I/O matters."
[Peak VRAM Usage: 1.64 GB | Total Time: 42 seconds]
            

---

Real-World Lessons in Speech AI

Issue	What Caused It	Engineering Solution
OOM CUDA Crash	Passing continuous 45-min spectrograms into Transformer self-attention.	Chunk audio into 30s sliding windows with zero-padding.
Repetition Loops	Autoregressive decoder getting stuck on background static noise.	Set `condition_on_previous_text=False` & apply low-pass filter.

Audio AI is full of subtle real-world edge cases. Once I mastered spectrogram preprocessing, low-pass noise filtering, and chunked inference, local models like Whisper became an indispensable part of my personal workflow.