FacialPen: Using Facial Detection to Augment Pen-Based Interaction

Xinrui Fang, Chengshuo Xia, Yuta Sugiura

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Pen-based interactions have been ubiquitously adopted on mobile and stationary devices, but the usability can be further augmented through the use of advanced techniques. In this work, we propose FacialPen, a prototype that uses facial gestures to trigger commands for pen-based manipulation. In our prototype, a fisheye camera is mounted to the end of a stylus that provides a broad view from which to capture the human face. We facilitated an elicitation study to identify natural and user-defined gestures for interactions with facial expressions. Different gestures can be further discerned via face detection and a classification pipeline. We designed a sketching demonstration application to explore usage scenarios and evaluated the effectiveness of FacialPen through a qualitative study. The user study posits that FacialPen supports efficiency by reducing screen widgets, enabling the continuity of creation work and liberating the user's stylus holding postures when switching sketch functions.

Original languageEnglish
Title of host publication5th Asian CHI Symposium 2021
EditorsJosh B. Tedjasaputra, Briane Paul V. Samson, Masitah Ghazali, Eunice Sari, Eunice Sari, Sayan Sarcar, Dilrukshi Gamage, Yohannes Kurniawan
PublisherAssociation for Computing Machinery, Inc
Number of pages8
ISBN (Electronic)9781450382038
Publication statusPublished - 2021 May 8
Event5th Asian CHI Symposium 2021 - Virtual, Online, Japan
Duration: 2021 May 72021 May 8

Publication series

Name5th Asian CHI Symposium 2021


Conference5th Asian CHI Symposium 2021
CityVirtual, Online


  • elicitation study
  • facial expression detection
  • stylus input

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Human-Computer Interaction
  • Information Systems
  • Software


Dive into the research topics of 'FacialPen: Using Facial Detection to Augment Pen-Based Interaction'. Together they form a unique fingerprint.

Cite this