Free viewpoint video synthesis and presentation from multiple sporting videos

This paper presents a new framework for observation of sporting events using HMD by applying the method of free viewpoint video synthesis for dynamic events to mixed reality. According to the viewpoint position of an observer, a virtual view image is generated by view interpolation among multiple sporting videos captured in a stadium, and then overlaid onto the real world via HMD. This makes the observer feel as if the match is played in front of his/her eyes. The proposed method performs virtual view synthesis and geometric registration between the real world and the virtual view images using projective geometry between cameras, which can be estimated from correspondence of natural feature points. It does not require calibration of multiple cameras imaging the sporting match and HMD camera capturing the real world. In this paper, we have constructed a mixed reality presentation system for a soccer match in order to demonstrate the utility of the proposed method. This system can overlay soccer scenes captured with multiple cameras in a soccer stadium onto a desktop field model with HMD.