Achieving Human–Robot Collaboration with Dynamic Goal Inference by Gradient Descent

Shingo Murata, Wataru Masuda, Jiayi Chen, Hiroaki Arie, Tetsuya Ogata, Shigeki Sugano

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Citations (Scopus)


Collaboration with a human partner is a challenging task expected of intelligent robots. To realize this, robots need the ability to share a particular goal with a human and dynamically infer whether the goal state is changed by the human. In this paper, we propose a neural network-based computational framework with a gradient-based optimization of the goal state that enables robots to achieve this ability. The proposed framework consists of convolutional variational autoencoders (ConvVAEs) and a recurrent neural network (RNN) with a long short-term memory (LSTM) architecture that learns to map a given goal image for collaboration to visuomotor predictions. More specifically, visual and goal feature states are first extracted by the encoder of the respective ConvVAEs. Visual feature and motor predictions are then generated by the LSTM based on their current state and are conditioned according to the extracted goal feature state. During collaboration after the learning process, the goal feature state is optimized by gradient descent to minimize errors between the predicted and actual visual feature states. This enables the robot to dynamically infer situational (goal) changes of the human partner from visual observations alone. The proposed framework is evaluated by conducting experiments on a human–robot collaboration task involving object assembly. Experimental results demonstrate that a robot equipped with the proposed framework can collaborate with a human partner through dynamic goal inference even when the situation is ambiguous.

Original languageEnglish
Title of host publicationNeural Information Processing - 26th International Conference, ICONIP 2019, Proceedings
EditorsTom Gedeon, Kok Wai Wong, Minho Lee
Number of pages12
ISBN (Print)9783030367107
Publication statusPublished - 2019
Externally publishedYes
Event26th International Conference on Neural Information Processing, ICONIP 2019 - Sydney, Australia
Duration: 2019 Dec 122019 Dec 15

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11954 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference26th International Conference on Neural Information Processing, ICONIP 2019


  • Deep learning
  • Human–robot collaboration
  • Prediction error minimization
  • Predictive coding
  • Robot learning

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science


Dive into the research topics of 'Achieving Human–Robot Collaboration with Dynamic Goal Inference by Gradient Descent'. Together they form a unique fingerprint.

Cite this