Voice vs BGM Separation: How AI Stem Splitting Works

You have a recording where someone is talking over a music bed, and you need them apart. Maybe you want to drop the music for a clean voice track, or keep the tune and lose the speech, or remix the two at a different balance. A few years ago this meant hunting for the original multitrack, which usually did not exist. Today you can feed a mixed file into an online tool and get two clean files back in under a minute. That is stem splitting, and understanding how it works helps you get far better results out of it — and know when it will fail.

The problem: one file, many sources

A finished audio file is a single stream of numbers. If a voice and a piano were recorded together, those two sounds are mathematically merged into one waveform, and there is no tag on the file that says "this sample is voice, this sample is piano." Separation means estimating which parts of that merged signal belong to which source.

For decades the standard answer was to split the file by frequency or by stereo position. That works when sources live in different places — a bass guitar in the low end, a cymbal in the high end, a vocal panned centre. It collapses the moment sources overlap, and voice and music overlap constantly. A male voice sits in the same 100 Hz to 4 kHz region as a guitar, a piano, and a synth pad. Frequency-based separation cannot tell them apart, which is why old "vocal remover" tools always sounded thin and phasey.

How AI separation actually works

AI stem splitting replaces the rule-based approach with a learned model. Instead of being told what a voice looks like, the model figures it out from data.

Step 1: Convert sound into a spectrogram

The first move is a short-time Fourier transform, or STFT. It slices the audio into short overlapping windows — typically 20 to 50 milliseconds — and measures how much energy sits at each frequency within each window. Stack those windows side by side and you get a spectrogram: a picture where the horizontal axis is time, the vertical axis is frequency, and brightness is energy.

The key insight is that a voice and a snare drum look different in this picture even when they overlap. A sustained note is a smooth horizontal band; speech has distinctive wavy patterns called formants that shift as the mouth changes shape; a cymbal is a fine speckled spray. The model learns these visual signatures.

Step 2: Predict a mask for each source

The model does not reconstruct the voice from scratch. It predicts a mask — a per-pixel instruction that says, for each time-frequency cell, how much of the energy in this cell belongs to the voice and how much to the music. If a cell is all voice, the mask value is 1; if it is all music, 0; if it is a mix, something in between.

This is why the approach is called masking rather than filtering. The tool is not inventing a new voice; it is deciding, cell by cell, which parts of the existing signal to keep for the voice stem and which to keep for the music stem.

Step 3: Reconstruct audio from the mask

Once the mask is applied, you have two spectrograms — one for voice, one for music. To turn a spectrogram back into listenable audio you use an inverse STFT, which rebuilds the waveform. The tricky part is phase. A spectrogram mostly captures magnitude, and the phase information needed to reconstruct a waveform is largely discarded in the transform. Models handle this by reusing the phase from the original mix and letting the reconstructed magnitude carry the separation, or by estimating phase directly. Imperfect phase is the main reason separated stems sometimes sound slightly hollow or smeared.

Why the models got good

The turning point was architecture and data. Modern separators are usually built on a U-Net, a network shaped like an hourglass: it compresses the spectrogram to find patterns, then expands it back to full resolution while combining both views. This lets the model see fine detail (a single consonant) and broad context (the overall texture of the music) at the same time.

Training needs paired data: the same song as a mix and as separate stems. Early datasets were small and built from a narrow range of music. Later releases expanded to hundreds of tracks across many genres, and researchers also trained on synthetic mixes made by combining clean speech with music at random volumes. That diversity is what makes a good model separate an acoustic ballad and a heavy rock track equally well.

The results are measured with a metric called SDR, or signal-to-distortion ratio, which compares the separated stem against the true stem. Reported numbers keep climbing, but the honest headline is simpler: on a well-recorded voice-over-music track, a good modern model can separate voice and music cleanly enough for most podcast and video work.

What separates well — and what does not

Separation quality depends far more on the input than on the tool. Expect good results when:

• The voice is loud and dry, sitting clearly above the music.
• The music is instrumental rather than heavily vocal.
• The two sources differ in stereo position or character.

Expect trouble when:

• Both sources are singing — the model often cannot tell two voices apart.
• The voice is quiet and drenched in reverb, so its signature is smeared.
• The file is heavily compressed, low-bitrate, or clipped.
• The music has a busy, percussive texture that masks the voice.

One useful trick: if you only need the voice and the music is a simple loop, separation will work almost perfectly. If both parts are complex, listen on headphones and check for artefacts — a faint ghost of the music under the voice, or a warbly, robotic edge. A small amount of residue is normal and usually inaudible under speech.

Practical workflow

1. Start from the best source file you have. Separation amplifies whatever is already there, including noise, so clean the recording first if you can.
2. Run the split and listen to each stem on its own.
3. For the voice stem, follow with a high-pass filter around 80–100 Hz and a light noise reduction if any music bleed is audible in the pauses.
4. For the music stem, do not expect perfection — it will usually carry a little vocal residue, which is fine if you are using it as a low bed under new narration.
5. Blend to taste rather than deleting one side outright; sometimes keeping 10–15 percent of the music under the voice sounds more natural than a hard cut.

You can do this online without installing anything — Offex Audio's free tool separates voice from background music using AI stem splitting, which is enough for a podcast intro, a YouTube narration, or rescuing an interview recorded in a noisy café.

Conclusion

AI stem splitting is not magic, but it is a genuine leap over the old frequency-based tricks. It works by turning sound into a picture, learning the visual signature of a voice, and masking out everything that does not match — then rebuilding the audio from the result. Knowing that, you can predict when it will shine (a clear voice over instrumental music) and when it will struggle (two overlapping voices, heavy reverb, a low-bitrate file). Feed it good audio, judge the stems on headphones, and clean up each one afterwards, and you will get results that once required the original multitrack.

Frequently asked questions

Try the free audio tools →

Related guides and tools