Implicit Neural Representations

One way to think of INRs is as a very fancy form of curve-fitting. Curve-fitting arbitrary, complicated nonlinear functions is where neural nets really shine.

If we have a time-domain audio signal as an array $x[i]$ where $i$ is the sample index, and let $t[i]$ be a time variable which is a linear ramp from 0 to 1. To produce an INR, we use a standard feedforward neural network, one with a single input neuron and a single output neuron, and any number of hidden layers. The entire network is a scalar function \(f: \mathbb{R} \rightarrow \mathbb{R}\). To train the INR, the parameters of the network are optimized so that $x[i]$ and $f(t[i])$ are as similar to each other as possible. I emphasize here that the “dataset” is not a set of audio files, but rather the individual samples in one audio buffer.

The neural net is nearly a textbook one, alternating layers of linearities and nonlinearities. I will give a formal description in the next section, but overall, there are two key practical differences from conventional uses of neural nets. First, INRs are meant to overfit, certainly overfit more than traditional nets used for prediction and classification. Usually you want the network to fit the data with relatively few parameters, but with INRs you can get away with a lot more parameters than normal, since the goal is not to generalize.

The other difference from textbook neural nets is the activation function. Classically, \(\sigma\) is a sigmoid function, and these days ReLU is popular, especially in deep learning. But a big breakthrough in INRs was realizing that they work best when you use a nonmonotonic, oscillatory \(\sigma\). This is the insight in the paper on SIREN, which sets \(\sigma(x) = \sin(x)\). (You could think of SIREN INR as doing a highly scaled-up form of phase modulation synthesis with stages of matrix mixers in between. I’m not sure if that leads to any real insights.) A bit later came WIRE, which uses complex instead of real signals for the hidden layers and a Gabor wavelet \(\sigma(x) = \exp(j \omega x + |sx|^2)\) where \(\omega\) and $s$ are two hyperparameters. We’re not using actual wavelet transforms here — all you need to know is that it’s a sinusoidal function which oscillates and is tapered on both ends. Because of the tapering, the Gabor wavelet is localized in time, making it a little less brittle than SIREN. (Although both methods overfit, WIRE overfits less badly than SIREN does.)

I won’t go into more technical details on SIREN and WIRE, since the kind of person who cares about such things should just read the original papers on them. In short, I went with WIRE.

Network architecture

In the interest of keeping this article self-contained, I will quickly give a formal definition of the INR I used for this project. Mathematics-averse readers are free to skip this section, as it it’s optional.

Let \(x \in \mathbb{R}^N\) be the time-domain values of an $N$-sample digital audio signal, with the $i$th zero-indexed sample written as $x[i]$. $x[i]$ is preprocessed so that its mean (dc offset) is zero and its maximum absolute value is 1, which makes the INR less sensitive to changes in amplitude. In the resynthesized output signal, we post-multiply by the peak amplitude to reverse the effects of normalization. (Normalization improved performance quite a bit.)

Let $t[i] = i / n$ be a time variable with linearly ramps from 0 to 1. A linear layer with $N$ inputs and $M$ outputs is given by \(f(x) = A x + b\) where \(A \in \mathbb{C}^{M \times N}\) and \(b \in \mathbb{C}^M\) are parameters. A WIRE nonlinearity is an element-wise nonlinearity \(f(x) = \exp(j\omega x + |sx|^2)\) where $omega = 20$ and $s = 10$. A “real part” is another nonlinearity where we simply take the real part of the result. The network architecture comprises four linear layers in series, with WIRE nonlinearities between them and a final layer that converts complex numbers to real ones:

• 1xH linear layer

• WIRE nonlinearity

• HxH linear layer

• WIRE nonlinearity

• HxH linear layer

• WIRE nonlinearity

• Hx1 linear layer

• Real part

where $H$ is the hidden layer size, which I have set to 300. I have arbitrarily set the hidden layers sizes to be equal, but this is not required. Also, remember that here the INR has only one input neuron and one output neuron, not $N$ neurons (a conceptual mistake that I made once during development). This is specific to mono audio files. In a stereo audio file, the INR would have 1 input neuron for time and 2 outputs for the L and R channels; in an RGB image, it would have 2 spatial inputs for the X and Y coordinates and 3 color channel outputs.

We also define a signal $w[i]$ of the same size as $x[i]$ which is interpreted as the weight of each sample, to tell the loss function that some samples are more important than others. In an unweighted case, all $w[i] = 0$.

We define two errors for our loss function. The first is MSE loss between the two signals:

The second is the MSE loss between the first differences (derivatives) of the two signals:

This comes from the INR literature where it is referred to generally as “gradient loss,” and I found that it definitely improves the INR’s ability to capture high-frequency information (recall that a first derivative is a type of high-pass filter). The final loss function is the weighted sum $E = E_1 + kE_2$, where $k$ is a hyperparameter that controls the INR’s sensitivity to higher frequencies.

Parameters are initialized using a uniform distribution from -1 to 1. Optimization is performed with an ADAM optimizer. (Stochastic Gradient Descent performed significantly worse.)

After the parameters have been optimized, the resynthesized signal is $f(t[i])$.

INRs for audio

At a high level, my goal was to find some application of INRs for audio that produces interesting musical effects, but under the following constraints:

• I don’t have a fancy GPU.

• I don’t want to wait hours to process a short audio file. I don’t need real time, but I’m impatient.

• The INR is trained from only the signal itself and does not require a dataset. (There’s a cool paper on using a “hypernetwork” so that the INR weights are themselves trained with a conventional supervised neural net, but I’d rather avoid a dataset and see what I can do with a lone signal.)

My approach to INRs and audio is to process the signal in overlapping windows, optimizing a new INR for each window. I did try INR in a non-windowed manner on long, multi-second signals, but the training process was very slow. Also, the behavior of the algorithm is quite sensitive to the length of the audio signal in samples. I went with windowed processing of a fixed length, making it easier to keep behavior consistent between different audio examples. (There may well be interesting musical results lurking in non-windowed processing.)

I settled for downsampling the signal to 22050 Hz and doing 4,000-sample, 2x overlap Hann windows. The square root of each window is pre-multiplied with the signal $x$ prior to processing it with INR. The INR’s error function weights $w[i]$ are also set to a Hann window, because we don’t really care about the beginning and ending samples, mostly the middle ones. After the INR resynthesizes the frame, it is again windowed by the square root of the Hann window. The output windows are overlapped and summed to produce the output signal.

For each window, I set the optimizer to a fixed number of only 50 iterations and a fairly fast learning rate so that the loss converges quickly to a local minimum. These are aggressive settings that produce suboptimal results. These are the Impatient Man’s Hyperparameters – I’m not waiting hours to process an audio file, even if it’s for art. The whole point is to listen to the artifacts anyway, so I’m okay with things not being optimal.

The agony of basically all forms of windowed processing is that windowing introduces artifacts of its own. These are tough to eliminate and they are sometimes audible as a tremolo or “rattle” in the results.

Okay, let’s hear it

Because of the suboptimal settings, we do not achieve perfect reconstruction, so running an audio file through the windowed INR processor already alters how it sounds. There’s a lowpass-like effect on the highs, which is expected since the curve-fitting smooths out oscillations, and we get weird signal-dependent noisy crackles that sound a bit like frying oil.

I remind you that INRs do not have to be this way. In a “correct,” “professional” implementation of INRs, you could get a perfect match with the target signal — you’d just need to put up with much slower training.

The effects are most interesting on simple, synthesized signals, since the artifacts are a lot more exposed.

Altering parameters for effects

What if, after training an INR, we alter its parameters before resynthesis? Can we design audio effects in the INR domain? Unfortunately, INRs are pretty brittle and can be sensitive to small changes in weights, especially the parameters toward the beginning of the neural net which have a domino effect on the later ones. Mostly the results tend to blow up into a textured noise. The noise follows the amplitude envelope of the signal, but that’s not because of the INR but because of the amplitude normalization.

Even if we only tweak the later stages of the INR in hopes of finding less destructive effects, the results are pretty disappointing – largely they just change the amount of crackling artifacts.

I did find an effect that’s pretty interesting. The final linear stage is a “mixdown” which takes 300 signals and adds them together to one signal. If we zero out entries in the final weight and bias vectors, we can mute some of those 300 channels. If I solo a single channel, I get just crackles, but they aren’t random – they seem to follow the brightness of the signal. Enabling more and more channels, the crackles get quieter and gradually rebuild the resynthesized signal. It sounds quite interesting if the number of enabled channels is less than 20; you can hear the signal faintly. With more than 50 channels enabled, it doesn’t sound as cool because it mostly just sounds like the signal with raspy noise added in – there is no longer an impression that the signal and noise are interacting.

## Further exploration

• Try it on multichannel signals.

• What if we use non-fully connected linear layers, like convolutional layers?

• Subtracting the resynthesized signal from the original signal produces a residual. What if you train an INR on a residual?

• Are there nonlinearities other than WIRE that might be more suitable for audio?

• What schemes are there for initializing the weights? I played with initializing the weights from the previous window, which was sort of interesting but mainly caused the windowing artifacts to be more prominent.

• Can the loss function be improved? I toyed around with using the frequency domain but didn’t have much success. Maybe you can.

• Pitch-synchronous windowing.

• Try converting the signal to a different representation before learning the INR, such as a Fourier transform.