Semi-automation of gesture annotation by machine learning and human collaboration

Naoto Ienaga, Alice Cravotta, Kei Terayama, Bryan W. Scotney, Hideo Saito, M. Grazia Busà

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)


Gesture and multimodal communication researchers typically annotate video data manually, even though this can be a very time-consuming task. In the present work, a method to detect gestures is proposed as a fundamental step towards a semi-automatic gesture annotation tool. The proposed method can be applied to RGB videos and requires annotations of part of a video as input. The technique deploys a pose estimation method and active learning. In the experiment, it is shown that if about 27% of the video is annotated, the remaining parts of the video can be annotated automatically with an F-score of at least 0.85. Users can run this tool with a small number of annotations first. If the predicted annotations for the remainder of the video are not satisfactory, users can add further annotations and run the tool again. The code has been released so that other researchers and practitioners can use the results of this research. This tool has been confirmed to work in conjunction with ELAN.

Original languageEnglish
Pages (from-to)673-700
Number of pages28
JournalLanguage Resources and Evaluation
Issue number3
Publication statusPublished - 2022 Sept


  • Active learning
  • Gesture detection
  • Machine learning
  • Video annotation

ASJC Scopus subject areas

  • Language and Linguistics
  • Education
  • Linguistics and Language
  • Library and Information Sciences


Dive into the research topics of 'Semi-automation of gesture annotation by machine learning and human collaboration'. Together they form a unique fingerprint.

Cite this