eduzhai > Applied Sciences > Engineering >

MM-KTD Multiple Model Kalman Temporal Differences for Reinforcement Learning

  • Save

... pages left unread,continue reading

Document pages: 14 pages

Abstract: There has been an increasing surge of interest on development of advancedReinforcement Learning (RL) systems as intelligent approaches to learn optimalcontrol policies directly from smart agents interactions with the environment.Objectives: In a model-free RL method with continuous state-space, typically,the value function of the states needs to be approximated. In this regard, DeepNeural Networks (DNNs) provide an attractive modeling mechanism to approximatethe value function using sample transitions. DNN-based solutions, however,suffer from high sensitivity to parameter selection, are prone to overfitting,and are not very sample efficient. A Kalman-based methodology, on the otherhand, could be used as an efficient alternative. Such an approach, however,commonly requires a-priori information about the system (such as noisestatistics) to perform efficiently. The main objective of this paper is toaddress this issue. Methods: As a remedy to the aforementioned problems, thispaper proposes an innovative Multiple Model Kalman Temporal Difference (MM-KTD)framework, which adapts the parameters of the filter using the observed statesand rewards. Moreover, an active learning method is proposed to enhance thesampling efficiency of the system. More specifically, the estimated uncertaintyof the value functions are exploited to form the behaviour policy leading tomore visits to less certain values, therefore, improving the overall learningsample efficiency. As a result, the proposed MM-KTD framework can learn theoptimal policy with significantly reduced number of samples as compared to itsDNN-based counterparts. Results: To evaluate performance of the proposed MM-KTDframework, we have performed a comprehensive set of experiments based on threeRL benchmarks. Experimental results show superiority of the MM-KTD framework incomparison to its state-of-the-art counterparts.

Please select stars to rate!


0 comments Sign in to leave a comment.

    Data loading, please wait...