eduzhai > Applied Sciences > Engineering >

Accelerating Training in Artificial Neural Networks with Dynamic Mode Decomposition

  • Save

... pages left unread,continue reading

Document pages: 13 pages

Abstract: Training of deep neural networks (DNNs) frequently involves optimizingseveral millions or even billions of parameters. Even with modern computingarchitectures, the computational expense of DNN training can inhibit, forinstance, network architecture design optimization, hyper-parameter studies,and integration into scientific research cycles. The key factor limitingperformance is that both the feed-forward evaluation and the back-propagationrule are needed for each weight during optimization in the update rule. In thiswork, we propose a method to decouple the evaluation of the update rule at eachweight. At first, Proper Orthogonal Decomposition (POD) is used to identify acurrent estimate of the principal directions of evolution of weights per layerduring training based on the evolution observed with a few backpropagationsteps. Then, Dynamic Mode Decomposition (DMD) is used to learn the dynamics ofthe evolution of the weights in each layer according to these principaldirections. The DMD model is used to evaluate an approximate converged statewhen training the ANN. Afterward, some number of backpropagation steps areperformed, starting from the DMD estimates, leading to an update to theprincipal directions and DMD model. This iterative process is repeated untilconvergence. By fine-tuning the number of backpropagation steps used for eachDMD model estimation, a significant reduction in the number of operationsrequired to train the neural networks can be achieved. In this paper, the DMDacceleration method will be explained in detail, along with the theoreticaljustification for the acceleration provided by DMD. This method is illustratedusing a regression problem of key interest for the scientific machine learningcommunity: the prediction of a pollutant concentration field in a diffusion,advection, reaction problem.

Please select stars to rate!


0 comments Sign in to leave a comment.

    Data loading, please wait...