eduzhai > Applied Sciences > Engineering >

Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory

  • king
  • (0) Download
  • 20210506
  • Save

... pages left unread,continue reading

Document pages: 21 pages

Abstract: In this paper, we explain the universal approximation capabilities of deepresidual neural networks through geometric nonlinear control. Inspired byrecent work establishing links between residual networks and control systems,we provide a general sufficient condition for a residual network to have thepower of universal approximation by asking the activation function, or one ofits derivatives, to satisfy a quadratic differential equation. Many activationfunctions used in practice satisfy this assumption, exactly or approximately,and we show this property to be sufficient for an adequately deep neuralnetwork with $n+1$ neurons per layer to approximate arbitrarily well, on acompact set and with respect to the supremum norm, any continuous function from$ mathbb{R}^n$ to $ mathbb{R}^n$. We further show this result to hold for verysimple architectures for which the weights only need to assume two values. Thefirst key technical contribution consists of relating the universalapproximation problem to controllability of an ensemble of control systemscorresponding to a residual network and to leverage classical Lie algebraictechniques to characterize controllability. The second technical contributionis to identify monotonicity as the bridge between controllability of finiteensembles and uniform approximability on compact sets.

Please select stars to rate!

         

0 comments Sign in to leave a comment.

    Data loading, please wait...
×