An Affective Video Generation System Supporting Impromptu Musical Performance

Anri Kobayashi, Issei Fujishiro

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)


When a musical instrument player performs music, the accompanying visual information can have a significant effect on the performance. For example, several players in a jam session may change their style of playing immediately by closely examining the co-players' expressions and behaviors and predicting their emotion and intention on the fly. In this paper, we propose a system that generates videos in response to the impromptu performance of a single musical instrumental player. The system evaluates the input signals in an affective way and generates a corresponding video based on the results of the evaluation. The player tends to change his/her performance while being inspired by the generated video and giving further triggers for the system to modify the video. The system aims to establish such an affective loop, where it is expected to act as a «co-player» who continues having an influence on the performance of the real player. The final goal of this study is to improve the quality of the player's performing experience through such interactions between the player and the system. By conducting a user evaluation, it was proven that we were able to inspire an amateur guitarist as the subject through the affective video generation and provide a cyberworld where he is allowed to experience a better performance than playing alone.

Original languageEnglish
Title of host publicationProceedings - 2016 International Conference on Cyberworlds, CW 2016
EditorsAlexei Sourin
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages8
ISBN (Electronic)9781509023035
Publication statusPublished - 2016 Nov 23
Event2016 International Conference on Cyberworlds, CW 2016 - Chongqing, China
Duration: 2016 Sept 282016 Sept 30

Publication series

NameProceedings - 2016 International Conference on Cyberworlds, CW 2016


Other2016 International Conference on Cyberworlds, CW 2016


  • Affective computing
  • cell automata
  • classifier
  • music visualization
  • musical performance
  • video generation

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications
  • Computer Vision and Pattern Recognition
  • Human-Computer Interaction
  • Signal Processing


Dive into the research topics of 'An Affective Video Generation System Supporting Impromptu Musical Performance'. Together they form a unique fingerprint.

Cite this