Consistent Smile Intensity Estimation from Wearable Optical Sensors

Katsutoshi Masai, Monica Perusquia-Hernandez, Maki Sugimoto, Shiro Kumano, Toshitaka Kimura

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Smiling plays a crucial role in human communication. It is the most frequent expression shown in daily life. Smile analysis usually employs computer vision-based methods that use data sets annotated by experts. However, cameras have space constraints in most realistic scenarios due to occlusions. Wearable electromyography is a promising alternative; however, issue of user comfort is a barrier to long-term use. Other wearable-based methods can detect smiles, but they lack consistency because they use subjective criteria without expert annotation. We investigate a wearable-based method that uses optical sensors for consistent smile intensity estimation while reducing manual annotation cost. First, we use a state-of-art computer vision method (OpenFace) to train a regression model to estimate smile intensity from sensor data. Then, we compare the estimation result to that of OpenFace. We also compared their results to human annotation. The results show that the wearable method has a higher matching coefficient (r=0.67) with human annotated smile intensity than OpenFace (r=0.56). Also, when the sensor data and OpenFace output were fused, the multimodal method produced estimates closer to human annotation (r=0.74). Finally, we investigate how the synchrony of smile dynamics among subjects and their average smile intensity are correlated to assess the potential of wearable smile intensity estimation.

Original languageEnglish
Title of host publication2022 10th International Conference on Affective Computing and Intelligent Interaction, ACII 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665459082
DOIs
Publication statusPublished - 2022
Event10th International Conference on Affective Computing and Intelligent Interaction, ACII 2022 - Nara, Japan
Duration: 2022 Oct 182022 Oct 21

Publication series

Name2022 10th International Conference on Affective Computing and Intelligent Interaction, ACII 2022

Conference

Conference10th International Conference on Affective Computing and Intelligent Interaction, ACII 2022
Country/TerritoryJapan
CityNara
Period22/10/1822/10/21

Keywords

  • affective computing
  • optical sensors
  • smart eyewear
  • smile intensity estimation
  • wearable computing

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Computer Vision and Pattern Recognition
  • Human-Computer Interaction

Fingerprint

Dive into the research topics of 'Consistent Smile Intensity Estimation from Wearable Optical Sensors'. Together they form a unique fingerprint.

Cite this