Approximate Dynamic Programming: From Optimization in Policy Space to Actor-Critic Methods

摘要

We consider simulation-based approximation methods for large scale dynamic programming problems. We discuss briefly methods relying on value function approximation ('reinforcement learning' or 'neuro-dynamic programming'). We then consider policy-space methods in which we start with a parametric class of policies, and tune the policy parameters using gradient descent. Such methods may suffer from a large variance, and we indicate approaches for variance reduction. We conclude with a discussion of actor-critic algorithms. These are two-time-scale algorithms in which the critic uses temporal difference (TD) learning with a linearly parameterized approximation architecture, and the actor is updated in an approximate gradient direction based on information provided by the critic. We indicate that the features for the critic should ideally span a subspace prescribed by the choice of parameterization of the actor, and state some convergence results.