Regularized Dynamical Parametric Approximation for Stiff Evolution Problems

Evolutionary deep neural networks have emerged as a rapidly growing field of research. This talk discusses numerical integrators for such and other classes of nonlinear parametrizations u(t) = Φ(θ(t)) where the evolving parameters θ(t) are to be computed. The primary focus is on tackling the challenges posed by the combination of stiff evolution problems and irregular parametrizations, which typically arise with neural networks, tensor networks, flocks of evolving Gaussians, and in further cases of overparametrization. Regularized parametric versions of the implicit Euler method and higher-order implicit Runge–Kutta methods for the time integration of the parameters in nonlinear approximations to evolutionary partial differential equations are presented. At each time step, an ill-conditioned nonlinear optimization problem is solved approximately with a few regularized Gauß–Newton iterations. Error bounds for the resulting parametric integrator are derived by relating the computationally accessible Gauß–Newton iteration for the parameters to the computationally inaccessible Newton iteration for the underlying non-parametric time integration scheme. Numerical experiments that are designed to show key properties of the proposed parametric integrators are discussed.