Higher-order numerical integration methods

[stefano]

So far in virtual analog modeling I've only seen first-order numerical integration methods being used (Euler's, trapezoidal/bilinear, alpha/Mobius transform), but AFAIK SPICE normally uses implicit Runge-Kutta 4 by default to get more accurate results.

Did any of you ever experiment with higher-order numerical integration methods? If so, any remarks on static and time-varying stability? Any references?

[stefano]

@Jan Wilczek, I just found out you used RK4 in https://arxiv.org/pdf/2205.01897.pdf. Any comments?

[Jan Wilczek]

Hi Stefano, thanks for the introduction. I have thought about this and I don't think I can bring something new to the table.

Indeed in our experiments, explicit, higher-order methods resulted in a noticeable improvement of the accuracy (the context was solving ODEs learned by neural networks for Virtual Analog modeling). The trade-off was of course an increased processing time. The whole method was, however, not implemented for real-time processing.

What I can say is that implicit methods failed completely, not only being prohibitively slow but also diverging time and time again. In that regard, explicit methods proved more reliable in the neural ODE case.

A nice discussion of applied integration methods for VA can be found in David Yeh's diode clipper paper "Simulation of the diode limiter in guitar distortion circuits by numerical solution of ordinary differential equations."

[stefano]

What I can say is that implicit methods failed completely, not only being prohibitively slow but also diverging time and time again. In that regard, explicit methods proved more reliable in the neural ODE case.

A nice discussion of applied integration methods for VA can be found in David Yeh's diode clipper paper "Simulation of the diode limiter in guitar distortion circuits by numerical solution of ordinary differential equations."

That is interesting. In classic simulation algorithms it's usually the opposite, implicit methods have better stability than explicit ones, as pointed out in the same paper you mentioned (IIRC, also Tim Stilson's PhD thesis goes deep into this topic).

In the meanwhile I found this paper by Ducceschi and Bilbao (free PDF available on that page) which seems to touch on the topic. At first sight it seems like something that takes quite some time to read, understand, and digest, and unluckily I'm busy to death these days.

[Archit3ch]

I recall reading that higher order methods result in more accuracy, but at the expense of extra computation. For SPICE it won't matter, because it's not real-time. However, taking a more accurate step and having all that work rejected because the implicit solve failed might be worse than taking 2 less accurate steps with e.g. trapezoidal.

I can also confirm from personal experience and discussions that everyone eventually hits computation/convergence problems with implicit integration. The variation is in how they deal with it.

The best option will vary depending on the specific system you are solving, the integrator, tolerances, the CPU and so on.

[stefano]

I can also confirm from personal experience and discussions that everyone eventually hits computation/convergence problems with implicit integration. The variation is in how they deal with it.

Weird. W.r.t. LTI systems the theory certainly tells the opposite (e.g., bilinear transform/trapezoidal rule is implicit), and in my experience I never had stability issues because of implicit discretization.

In the context of NN I have no idea, I tend to trust what Jan reports.

[Archit3ch]

I meant in the context of nonlinear systems (hence the implicit solve).

For linear systems you can pre-solve symbolically and it reduces to a biquad (or the 3rd order equivalent in this example).