TY - GEN
T1 - TimToShape
T2 - 28th International Conference on Intelligent User Interfaces, IUI 2023
AU - Arai, Kota
AU - Hirao, Yutaro
AU - Narumi, Takuji
AU - Nakamura, Tomohiko
AU - Takamichi, Shinnosuke
AU - Yoshida, Shigeo
N1 - Publisher Copyright:
© 2023 Owner/Author.
PY - 2023/3/27
Y1 - 2023/3/27
N2 - Timbre is high-dimensional and sensuous, making it difficult for musical-instrument learners to improve their timbre. Although some systems exist to improve timbre, they require expert labeling for timbre evaluation; however, solely visualizing the results of unsupervised learning lacks the intuitiveness of feedback because human perception is not considered. Therefore, we employ crossmodal correspondences for intuitive visualization of the timbre. We designed TimToShape, a system that visualizes timbre with 2D shapes based on the user's input of timbre-shape correspondences. TimToShape generates a shape morphed by linear interpolation according to the timbre's position in the latent space, which is obtained by unsupervised learning with a variational autoencoder (VAE). We confirmed that people perceived shapes generated by TimToShape to correspond more to timbre than randomly generated shapes. Furthermore, a user study of six violin players revealed that TimToShape was well-received in terms of visual clarity and interpretability.
AB - Timbre is high-dimensional and sensuous, making it difficult for musical-instrument learners to improve their timbre. Although some systems exist to improve timbre, they require expert labeling for timbre evaluation; however, solely visualizing the results of unsupervised learning lacks the intuitiveness of feedback because human perception is not considered. Therefore, we employ crossmodal correspondences for intuitive visualization of the timbre. We designed TimToShape, a system that visualizes timbre with 2D shapes based on the user's input of timbre-shape correspondences. TimToShape generates a shape morphed by linear interpolation according to the timbre's position in the latent space, which is obtained by unsupervised learning with a variational autoencoder (VAE). We confirmed that people perceived shapes generated by TimToShape to correspond more to timbre than randomly generated shapes. Furthermore, a user study of six violin players revealed that TimToShape was well-received in terms of visual clarity and interpretability.
KW - crossmodal correspondences
KW - musical instrumental practice
KW - timbre
KW - timbre-shape correspondences
KW - variational autoencoder
UR - https://www.scopus.com/pages/publications/85152147956
UR - https://www.scopus.com/pages/publications/85152147956#tab=citedBy
U2 - 10.1145/3581641.3584053
DO - 10.1145/3581641.3584053
M3 - Conference contribution
AN - SCOPUS:85152147956
T3 - International Conference on Intelligent User Interfaces, Proceedings IUI
SP - 850
EP - 865
BT - IUI 2023 - Proceedings of the 28th International Conference on Intelligent User Interfaces
PB - Association for Computing Machinery
Y2 - 27 March 2023 through 31 March 2023
ER -