FacialPen: Using Facial Detection to Augment Pen-Based Interaction

Pen-based interactions have been ubiquitously adopted on mobile and stationary devices, but the usability can be further augmented through the use of advanced techniques. In this work, we propose FacialPen, a prototype that uses facial gestures to trigger commands for pen-based manipulation. In our prototype, a fisheye camera is mounted to the end of a stylus that provides a broad view from which to capture the human face. We facilitated an elicitation study to identify natural and user-defined gestures for interactions with facial expressions. Different gestures can be further discerned via face detection and a classification pipeline. We designed a sketching demonstration application to explore usage scenarios and evaluated the effectiveness of FacialPen through a qualitative study. The user study posits that FacialPen supports efficiency by reducing screen widgets, enabling the continuity of creation work and liberating the user's stylus holding postures when switching sketch functions.