Reinforcement learning with symmetry reduction for rotating cylinder wakes: finding unstable low-drag fixed points

Reinforcement learning (RL) is a powerful tool for discovering complex control strategies in fluid problems. Unfortunately, RL is often slow to train, does not respect system symmetries, and is difficult to interpret. Here we seek to overcome these challenges in RL for flow around a cylinder by using transfer learning from parallel, coarse environments, using symmetry reduction, and investigating the relationship between the long-time dynamics and unsteady fixed points of the system. We apply these methods to RL in the canonical case of flow around a cylinder to highlight the importance of our approach in a well-studied benchmark. We discover that using transfer learning and symmetry reduction results in greater drag reduction than observed in previous studies, and we observe that this drag reduction is due to the RL pushing states toward unstable fixed points. Specifically, we perform RL control on a rotating circular cylinder at Re_D = 100 using a Soft Actor-Critic (SAC) algorithm. The RL agent inthis work observes the vorticity field via sensors behind the cylinder and controls the flow by changing its rotational velocity. We perform transfer learning by training an agent first on parallelized low-resolution direct numerical simulations (DNS). We then deploy the agent to a high-resolution DNS to reduce drag. To this end, we use a reward function that in corporates the drag coefficient (C_D) and additional penalty terms, lift coefficient (C_L), and time derivative of the rotational velocity. By varying the CL penalty, the RL agent learns how to push the state toward different unstable fixed points. The combination of SAC, parallelization, transfer learning, and symmetry reduction results in dramatic drag reduction.