Skip to main navigation Skip to search Skip to main content

BGM2Pose: Active 3D Human Pose Estimation With Non-Stationary Sounds

Research output: Contribution to journal › Article › peer-review

Abstract

We propose BGM2Pose, a non-invasive 3D human pose estimation method using pre-selected everyday music (e.g., background music) as active sensing signals. Unlike existing approaches that significantly limit practicality by employing intrusive chirp signals within the audible range, our method utilizes user-selected natural music that causes minimal discomfort to listeners. Estimating human poses from standard music presents significant challenges. In contrast to sound sources specifically designed for measurement, regular music varies in both volume and pitch. These dynamic changes in signals caused by music are inevitably mixed with alterations in the sound field resulting from human motion, making it hard to extract reliable cues for pose estimation. To address these challenges, BGM2Pose introduces a Contrastive Pose Extraction Module that employs contrastive learning and hard negative sampling to eliminate musical components from the recorded data, isolating the pose information. Additionally, we propose a Frequency-wise Attention Module that enables the model to focus on subtle acoustic variations attributable to human movement by dynamically computing attention across frequency bands. Experiments suggest that our method outperforms the existing methods, demonstrating substantial potential for real-world applications. Our datasets and code will be made publicly available.

Original languageEnglish
Pages (from-to)790-799
Number of pages10
JournalIEEE Open Journal of Signal Processing
Volume7
DOIs
Publication statusPublished - 2026

Keywords

  • Human pose estimation
  • deep learning
  • sound

ASJC Scopus subject areas

  • Signal Processing

Fingerprint

Dive into the research topics of 'BGM2Pose: Active 3D Human Pose Estimation With Non-Stationary Sounds'. Together they form a unique fingerprint.

Cite this