eduzhai > Applied Sciences > Engineering >

Policy Gradient-based Algorithms for Continuous-time Linear Quadratic Control

  • Save

... pages left unread,continue reading

Document pages: 34 pages

Abstract: We consider the continuous-time Linear-Quadratic-Regulator (LQR) problem interms of optimizing a real-valued matrix function over the set of feedbackgains. The results developed are in parallel to those in Bu et al. [1] fordiscrete-time LTI systems. In this direction, we characterize severalanalytical properties (smoothness, coerciveness, quadratic growth) that arecrucial in the analysis of gradient-based algorithms. We also point outsimilarities and distinctive features of the continuous time setup incomparison with its discrete time analogue. First, we examine three types ofwell-posed flows direct policy update for LQR: gradient flow, natural gradientflow and the quasi-Newton flow. The coercive property of the corresponding costfunction suggests that these flows admit unique solutions while the gradientdominated property indicates that the underling Lyapunov functionals decay atan exponential rate; quadratic growth on the other hand guarantees that thetrajectories of these flows are exponentially stable in the sense of Lyapunov.We then discuss the forward Euler discretization of these flows, realized asgradient descent, natural gradient descent and quasi-Newton iteration. Wepresent stepsize criteria for gradient descent and natural gradient descent,guaranteeing that both algorithms converge linearly to the global optima. Anoptimal stepsize for the quasi-Newton iteration is also proposed, guaranteeinga $Q$-quadratic convergence rate--and in the meantime--recovering theKleinman-Newton iteration. Lastly, we examine LQR state feedback synthesis witha sparsity pattern. In this case, we develop the necessary formalism andinsights for projected gradient descent, allowing us to guarantee a sublinearrate of convergence to a first-order stationary point.

Please select stars to rate!


0 comments Sign in to leave a comment.

    Data loading, please wait...