SOLIPSIUMAlpha

How the engine works

Inside one recording.

Follow a damaged recording through Solipsium's engine, from the diagnosis to what comes back. The pictures are the engine's real output.

The four steps

  1. Spectrogram of the damaged recording.

    This is one spoken sentence, damaged on purpose. Time runs left to right, low notes at the bottom, high at the top. Colour follows pitch; brighter is louder.

    Step 1 of 4

    It names the damage first.

    Before anything changes, the diagnosis lists each problem and where it is. The numbers match the marks.

    • 1.59 sDropout. 38 ms of silence.
    • 3.57 sDropout. 38 ms of silence.
    • ThroughoutMains hum. 60 Hz and its harmonics.
    • ThroughoutHiss. A steady floor 22 dB under the loudest speech.
    • ThroughoutMissing treble. Little above 8.1 kHz.
  2. Spectrogram of the damaged recording.Spectrogram after Dropout filling.Spectrogram after Hum removal.Spectrogram after Full-band speech cleanup.Spectrogram after Treble rebuilding (AudioSR).

    Step 2 of 4

    It repairs one problem at a time.

    The repairs run in the engine's own order. Each picture is the sound after that repair.

    1. The recording after Dropout filling.RanDropout filling. Short holes of lost audio were found, so they are filled from nearby sound.
    2. The recording after Hum removal.RanHum removal. Mains hum was found, so it is notched out before any cleanup.
    3. The recording after Full-band speech cleanup.RanFull-band speech cleanup. A steady noise floor was found, so it is removed.
    4. The recording after Treble rebuilding (AudioSR).SeparateTreble rebuilding (AudioSR). Suggested repair ends before this step. AudioSR ran separately on the GPU, and what it adds is inferred.
  3. Spectrogram of the recording handed back.

    Step 3 of 4

    It marks what it guessed.

    Filled gaps are estimated and rebuilt treble is inferred, so both are marked here.

    • EstimatedThe gap at 1.59 s. Continued from the sound on each side.
    • EstimatedThe gap at 3.57 s. Continued from the sound on each side.
    • InferredThe treble above 8.1 kHz. Rebuilt by AudioSR as a separate step. It may differ from what was lost.
  4. Spectrogram of the recording handed back.

    Step 4 of 4

    You get the recording back.

    The frequencies fold back into one beam. The faint beam behind it is the damaged recording; both play below.

    • EstimatedDropout filling. 2 gaps filled, 76 ms in all, from the sound on each side.
    • RemovedHum removal. Hum notched out at 60, 120, 180 and 240 Hz.
    • ReducedFull-band speech cleanup. The quietest moments fell from -49 dB to silence.
    • InferredTreble rebuilding (AudioSR). Treble rebuilt above the old limit. It may differ from what was lost.

Hear it

All versions play at the same loudness, and you can switch between them while one plays.

Original
Repaired
With rebuilt treble
Scores for this example, worked out by formula against the clean recording. No listening test was done.
MeasureOriginalRepairedWith rebuilt treble
Intelligibility (ESTOI)0 to 1, higher is better0.800.880.88
Signal to distortion (SI-SDR)decibels, higher is better. AudioSR redraws the whole waveform, so this measure falls9.09.58.6
Treble distance above 7 kHz (LSD)lower is closer3.753.261.89

About this example

How this example was made.

This is a single recording, damaged on purpose. The numbers above are for this recording only.

The recording
VCTK 0.92, sentence p238_029, spoken by one of the four speakers in the September 12 check set. That set was seen during development. Yamagishi, Veaux & MacDonald (2019). Source · CC BY 4.0. Solipsium added the damage, the repairs, the level matching and the MP3 encoding. The recording is supplied without warranties, and no endorsement is implied.
The damage, in order
  1. Mains hum. 60 Hz and its harmonics, 16 dB below the voice's average.
  2. Hiss. Steady white noise, 16 dB below the voice's average.
  3. Two dropouts. 40 ms of silence each, at 1.59 s and 3.57 s.
  4. A low sample rate. Saved at 14 kHz, like an old upload, which removes the sound above 7 kHz. A little sound still leaks past the resampling filter. Step 1 counts any band no more than 45 dB below the speech band (0.5 to 4 kHz), so it puts the top at 8.1 kHz.

The random seed is 3, so every run gets the same damage.

The pictures
Height is pitch, on a scale that follows the ear, and colour follows height. Brightness is level, on the same scale in every picture: black is 75 dB below the loudest moment of the damaged recording.
The scores
The repository's own code compares each version with the clean recording, using ESTOI for intelligibility, SI-SDR for distortion and log-spectral distance above 7 kHz for the treble. Original here means the damaged recording as it arrived. AudioSR redraws the whole waveform, including the parts below the treble. SI-SDR compares waveforms sample by sample, so it can fall while the treble gets closer.
What else it saw
No clicks were added to this example, but the editor's scan, a separate check, flagged 14 possible clicks between 1.69 s and 3.84 s. Of those, 12 fall within 10 ms of a click the same check finds in the clean recording, so those are sharp sounds in the voice. Suggested repair does not act on clicks. The data behind this page.

On 1,344 damaged test clips (112 sentences from twelve speakers, each damaged 12 ways), the engine of September 29 made 90.8% better and 1.4% worse. The strongest single tool made 51.8% better and 46.4% worse. Measured on September 29, on a sealed set that was opened once, after the rules and settings were fixed. The damage was simulated on speech and the scores come from formulas. That evening it became the engine this site runs. The 90.8% and 1.4% measure the engine's own output. The file the site saved at the time scored lower, 89.7% better and 3.1% worse, because a repaired clipped recording was saved about 1.5 dB quieter and the formula counts level. Suggested repair has changed since, and so has the rule that saved those repairs quieter. The sealed set is scored only once, so there is no new number yet.

Hear what it does to your recording.

Solipsium is a free alpha. It takes files up to 30 minutes and 1 GB, mono or stereo, and you can compare every change before you keep it.

Try it on your recording How the site measured