Overview of AIWolfDial 2019 Shared Task: Contest of automatic dialog agents to play the werewolf game through conversations

The AIWolf project has been holding contests to play the Werewolf game (“Mafia”) by automatic agents for a couple of years. A difficulty of the Werewolf game is that the game is an imperfect information game, where a player’s role is hidden from other players. Players are required to infer the roles of other players through free conversations; players of a specific role should tell a lie, while others try to break through lies. We employ this werewolf game as a novel way of evaluations for dialog systems. Because the werewolf game forces players to deceive, persuade, and detect lies, neither inconsistent nor vague response are evaluated as “unnatural”, losing in the game. Our werewolf game competition and evaluation could be a new interesting evaluation criteria for dialog systems, but also for imperfect information game theories. In addition, the werewolf game allows any conversation, so the game includes both task-oriented and non-task-oriented conversations. This aspect would provide a handy intermediate goal rather than to create a general dialog system from scratch. In this AIWolfDial 2019 shared task, five participant agents played games in English and Japanese. We performed subjective evaluations on these game logs.