Multi-task Reinforcement Learning in Partially Observable Stochastic Environments

摘要

We consider the problem of multi-task reinforcement learning (MTRL)in multiple partially observable stochastic environments. Weintroduce the regionalized policy representation (RPR) tocharacterize the agent's behavior in each environment. The RPR is aparametric model of the conditional distribution over currentactions given the history of past actions and observations; theagent's choice of actions is directly based on this conditionaldistribution, without an intervening model to characterize theenvironment itself. We propose off-policy batch algorithms to learnthe parameters of the RPRs, using episodic data collected whenfollowing a behavior policy, and show their linkage to policyiteration. We employ the Dirichlet process as a nonparametric priorover the RPRs across multiple environments. The intrinsic clusteringproperty of the Dirichlet process imposes sharing of episodes amongsimilar environments, which effectively reduces the number ofepisodes required for learning a good policy in each environment,when data sharing is appropriate. The number of distinct RPRs andthe associated clusters (the sharing patterns) are automaticallydiscovered by exploiting the episodic data as well as thenonparametric nature of the Dirichlet process. We demonstrate theeffectiveness of the proposed RPR as well as the RPR-based MTRLframework on various problems, including grid-world navigation andmulti-aspect target classification. The experimental results showthat the RPR is a competitive reinforcement learning algorithm inpartially observable domains, and the MTRL consistently achievesbetter performance than single task reinforcement learning. color="gray">