Policy gradients with memory-augmented critic: Stabilizing off-policy policy gradients via differentiable memory

Deep reinforcement learning has been investigated in high-dimensional continuous control tasks. Deep Deterministic Policy Gradients (DDPG) is known as a highly sample-efficient policy gradients algorithm. However, it is reported that DDPG is unstable during training due to bias and variance problems of learning its action-value function. In this paper, we propose Policy Gradients with Memory Augmented Critic (PGMAC) that builds action-value function with the memory module previously proposed as Differentiable Neural Dictionary (DND). Although the DND is only studied in discrete action-space problems, we propose Action-Concatenated Key, which is a technique to combine DDPG-based policy gradient methods and DND. Furthermore, the remarkable advantage of PGMAC is shown that long-term reward calculation and weighted summation of value estimation at DND has an essential mechanism to solve the bias and variance problem. In experiment, PGMAC significantly outperformed baselines in continuous control tasks. The effects of hyperparameters were also investigated to show that the memory-augmented action-value function reduces the bias and variance in policy optimization.