Vision-based detection of guitar players' fingertips without markers

This paper proposes a vision-based method for detecting the positions of fingertips of a hand playing a guitar. We detect the skin color of a guitar player's hand by using on-line adaptation of color probabilities and a Bayesian classifier which can cope with considerable illumination changes and a dynamic background. The results of hand segmentation are used to train an artificial neural network. A set of Gabor filters is utilized to compute a lower-dimensional representation of the image. Then an LLM (Local-Linear- Mapping)-network is applied to map and estimate fingertip positions smoothly. The system enables us to visually detect the fingertips even when the fingertips are in front of skin-colored surfaces and/or when the fingers are not fully stretched out. Representative experimental results are also presented.