On using discretized Cohen-Grossberg node dynamics for model-free actor-critic neural learning in non-Markovian domains

摘要

We describe how multi-stage non-Markovian decision problems can be solved using actor-critic reinforcement learning by assuming that a discrete version of Cohen-Grossberg node dynamics describes the node-activation computations of neural network (NN). Our NN is capable of rendering the process Markovian implicitly and automatically in a totally model-free fashion without learning by how much the state apace must be augmented so that the Markov property holds. This serves as an alternative to using Elman or Jordan-type function as a history memory in order to develop sensitivity to non-Markovian dependencies. We shall demonstrate our concept using a small-scale non-Markovian deterministic path problem, in which our actor-critic NN finds an optimal sequence of actions, although it needs much iteration due to the nature of neural model-free learning. This is, in spirit, a neuro-dynamic programming approach.