TY - GEN
T1 - Real-Time Dynamic Digital Twin Generation Using Multiple First-Person Views
AU - Yazaki, Hiroki
AU - Hirokawa, Shinnosuke
AU - Hori, Ryosuke
AU - Saito, Hideo
AU - Sawada, Azusa
AU - Nakanoya, Manabu
N1 - Publisher Copyright:
© 2026 IEEE.
PY - 2026
Y1 - 2026
N2 - The growing interest in the Metaverse drives demand for digital twins that accurately replicate physical environments. While capturing real-time human dynamics in static environments is essential for immersion, it is limited by the scalability of conventional motion capture. Furthermore, multi-user spatial consistency is challenging when using only first-person view (FPV) streams. We propose a real-time system generating dynamic digital twins from multiple FPV videos. Our method achieves robust localization by integrating a pre-scanned 3D Gaussian Splatting map, ORB-SLAM3, and YOLO-based dynamic object removal. We integrate individual motion synthesis with spatial synchronization; leveraging Ego-Poser for full-body reconstruction, we unify disparate streams into a global coordinate system. Experimental results show our system operates at 30 to 60 FPS, enabling synchronized visualization of multiple 3D human models in a photorealistic environment.
AB - The growing interest in the Metaverse drives demand for digital twins that accurately replicate physical environments. While capturing real-time human dynamics in static environments is essential for immersion, it is limited by the scalability of conventional motion capture. Furthermore, multi-user spatial consistency is challenging when using only first-person view (FPV) streams. We propose a real-time system generating dynamic digital twins from multiple FPV videos. Our method achieves robust localization by integrating a pre-scanned 3D Gaussian Splatting map, ORB-SLAM3, and YOLO-based dynamic object removal. We integrate individual motion synthesis with spatial synchronization; leveraging Ego-Poser for full-body reconstruction, we unify disparate streams into a global coordinate system. Experimental results show our system operates at 30 to 60 FPS, enabling synchronized visualization of multiple 3D human models in a photorealistic environment.
KW - 3D Gaussian Splatting
KW - Computing methodologies - Artificial intelligence - Computer vision - Computer vision problems - Reconstruction
KW - Digital twin
KW - First-person view
UR - https://www.scopus.com/pages/publications/105038742329
UR - https://www.scopus.com/pages/publications/105038742329#tab=citedBy
U2 - 10.1109/VRW70859.2026.00139
DO - 10.1109/VRW70859.2026.00139
M3 - Conference contribution
AN - SCOPUS:105038742329
T3 - Proceedings - 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2026
SP - 718
EP - 724
BT - Proceedings - 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2026
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2026 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2026
Y2 - 21 March 2026 through 25 March 2026
ER -