Building a Low Latency Piano Denoiser for Music Education

An industry–academia collaboration with Queen Mary University of London

The task of piano denoising addresses the challenge of removing unwanted noise from recordings while preserving the instrument’s quality. Typical applications range from live scenarios such as a student playing piano over a microphone in a noisy home environment while a teacher listens remotely, to the restoration of century-old historical piano recordings affected by recording artifacts and environmental noise.

In the domain of audio denoising, speech denoising has become highly effective, as seen in the noise suppression used on platforms like Teams. However, advancements in music denoising have been less extensive and represent a more niche area. The reason for this gap is that musical instruments such as the piano are less predictable compared to speech. A piano note is complex and has a distinct onset, a sustain, and a decay. Each note possesses a dynamic spectral envelope, meaning the sound’s character changes as the note rings out. Noise such as a sudden cough, a door slam, or a dropped household object can mimic these characteristics, thereby confusing a denoiser. It can be seen from the spectrograms in Figure 1 that the multi-frequency structure of the piano’s harmonics spans over a broad range of frequencies, a characteristic similar to that of environmental noise.

Because harmonic content overlaps with noise, denoisers either (a) remove desirable musical content (over-attenuation) or (b) leave residual noise. Both outcomes degrade audio quality.

To solve this, we need a model geared to musical structure – one capable of understanding the difference between a musical crash and a literal one.

The Solution: Adapting HTDemucs for Music

To tackle the complexity of piano acoustics, we reframed the denoising problem as a Source Separation problem. Imagine the audio recording as a duet between two “performers”: the pianist and the noise environment. Our goal was not to filter the audio, but to train a model to understand the harmonic and temporal structure of the piano and cleanly “separate” it from the unwanted noise source. 

To achieve this, we use a state-of-the-art architecture known as HTDemucs (Hybrid Transformer Demucs). Originally designed to separate full music tracks into individual stems (like drums, bass, and vocals), this model operates in two distinct domains: spectrogram and waveform and utilizes the best of both worlds. We modified the HTDemucs architecture by introducing a trainable linear layer at the end of the network, which acts as a Source Mapper. This layer functions as a funnel, taking the separated features from the HTDemucs core and mapping them into a single output: the denoised piano signal. This modification allowed us to repurpose a well trained, pre-existing model for piano denoising without having to reinvent the core architecture. The figure below shows this architectural modification:

Loss Functions and Fine-Tuning

After choosing the right base model (HTDemucs), the next step was to make it work for piano denoising. To achieve this, we had to tune the model precisely. We focused on two important areas: designing a Loss Function that determines how the model learns from mistakes and determining the optimal Fine-Tuning Strategy that controls how much of the model to retrain. In order to evaluate our trajectory, we used two metrics: ViSQOL (a perceptual quality metric by Google) and FAD (Fréchet Audio Distance).

The Loss Function: 

We used a composite loss function and conducted experiments to measure the impact of the different loss terms by tuning weights of different loss terms. The 3 key terms that proved to be useful were:

• SI-SDR (Scale-Invariant Signal-to-Distortion Ratio): SI-SDR is a widely recognized differentiable loss function for training end-to-end neural architectures for source separation and denoising algorithms. It computes the distortion after removing the influence of time-invariant scaling, ensuring that perceptually irrelevant amplitude mismatches do not affect the loss.
• Waveform L1 Loss: It measures the absolute error between the predicted waveform and the clean target. It ensures the raw data points are accurate.
• Temporal Smoothness: Piano notes evolve smoothly over time; they don’t jump around randomly. We added this term to ensure that consecutive frames of the spectrogram transition naturally, preventing abrupt changes in the audio which is a characteristic of noise. Mathematically, we penalize the model when the change in the prediction does not match the change in the ground-truth piano recording (See the equation below):

Where Spred​ and Starg​ represent the magnitude spectra at consecutive time frames t and t+1.

Fine Tuning: 

The original HTDemucs model was trained to separate four different instruments (drums, bass, vocals, other). For our Piano Denoiser, we didn’t need all that broad capability. We had to decide how many layers of the base model to unfreeze. This is a trade-off between Specialization and Generalization:

Unfreezing too few layers: The model cannot adapt to the nuances of piano denoising.

Unfreezing too many layers: We risk forgetting the feature extraction capabilities learned during original training.

By monitoring the evaluation metrics, we found that unfreezing two layers of the base model combined with the weights learned for the source mapper allowed the model to specialize in piano denoising without destroying its pre-learned knowledge.

Real-Time Implementation: Bringing the piano denoiser to Life

After we adapted the HTDemucs architecture for piano denoising, we were left with one last challenge on our hands. Our denoiser model accepted chunks of noisy piano audio and returned chunks of denoised piano audio. In order to convert this to a real-time system, we wanted a system that accepts a continuous audio stream, runs the denoiser on the incoming sound, and plays the cleaned audio back as a continuous stream. To achieve this continuity, it’s important to create a system that decouples capture from processing so capture is non-blocking and processing runs concurrently. To achieve this, we implemented a Producer-Consumer architecture.

The Producer: Capturing the Stream

The Producer acts as the system’s ears, responsible for capturing live audio. A callback function listens to the input audio stream that delivers sound at a fixed pace. As soon as a chunk of audio arrives from the microphone, it is placed into a shared queue without heavy processing. Instead of trying to process the audio immediately, it simply drops the data into the audio_queue and instantly returns to listening. This queue acts as a temporary holding area that decouples audio capture from audio processing.

The Consumer: Processing in the Background

The Consumer is responsible for preprocessing and running the model on a separate background thread. While the Producer continuously captures the sound, the Consumer monitors the audio_queue. When a noisy chunk arrives, the Consumer pops it out from the audio_queue, applies preprocessing and passes it through the fine-tuned denoiser model. Finally, it deposits the cleaned audio into a second queue for playback through the speakers.

Figure 3: The producer-consumer architecture decouples audio capture from inference. A thread-safe queue buffers noisy input (Red), allowing the Consumer to generate clean audio (Green) in parallel without blocking the recording stream.

This structure reduces the risk that model inference blocks audio capture and causes dropouts.

Audio samples:

Below are some audio examples of noisy and cleaned piano recordings in different environment: