Achieving Human–Robot Collaboration with Dynamic Goal Inference by Gradient Descent

Collaboration with a human partner is a challenging task expected of intelligent robots. To realize this, robots need the ability to share a particular goal with a human and dynamically infer whether the goal state is changed by the human. In this paper, we propose a neural network-based computational framework with a gradient-based optimization of the goal state that enables robots to achieve this ability. The proposed framework consists of convolutional variational autoencoders (ConvVAEs) and a recurrent neural network (RNN) with a long short-term memory (LSTM) architecture that learns to map a given goal image for collaboration to visuomotor predictions. More specifically, visual and goal feature states are first extracted by the encoder of the respective ConvVAEs. Visual feature and motor predictions are then generated by the LSTM based on their current state and are conditioned according to the extracted goal feature state. During collaboration after the learning process, the goal feature state is optimized by gradient descent to minimize errors between the predicted and actual visual feature states. This enables the robot to dynamically infer situational (goal) changes of the human partner from visual observations alone. The proposed framework is evaluated by conducting experiments on a human–robot collaboration task involving object assembly. Experimental results demonstrate that a robot equipped with the proposed framework can collaborate with a human partner through dynamic goal inference even when the situation is ambiguous.