eduzhai > Applied Sciences > Engineering >

Optimizing Neural Networks via Koopman Operator Theory

  • Save

... pages left unread,continue reading

Document pages: 18 pages

Abstract: Koopman operator theory, a powerful framework for discovering the underlyingdynamics of nonlinear dynamical systems, was recently shown to be intimatelyconnected with neural network training. In this work, we take the first stepsin making use of this connection. As Koopman operator theory is a lineartheory, a successful implementation of it in evolving network weights andbiases offers the promise of accelerated training, especially in the context ofdeep networks, where optimization is inherently a non-convex problem. We showthat Koopman operator theoretic methods allow for accurate predictions ofweights and biases of feedforward, fully connected deep networks over anon-trivial range of training time. During this window, we find that ourapproach is >10x faster than various gradient descent based methods (e.g. Adam,Adadelta, Adagrad), in line with our complexity analysis. We end byhighlighting open questions in this exciting intersection between dynamicalsystems and neural network theory. We highlight additional methods by which ourresults could be expanded to broader classes of networks and larger trainingintervals, which shall be the focus of future work.

Please select stars to rate!

         

0 comments Sign in to leave a comment.

    Data loading, please wait...
×