Arbitrary viewpoint image synthesis based on light field construction from uncalibrated image sequence

Recently, image-based rendering in which the actual image is efficiently utilized to synthesize a new image has been attracting attention. There have been many studies of arbitrary viewpoint image synthesis based on the light field, which is an image-based rendering procedure. It is now possible to synthesize images with a high degree of realism. However, in the method based on the theory of the light field, the camera parameters used in the imaging must be known (with calibration). In cases such as imaging by a free hand-held camera, the theory of the light field cannot be applied to the sequence of uncalibrated images. Consequently, this paper proposes an image generation method using the theory of the light field, which can be applied to a sequence of uncalibrated images. In the proposed method, the ray in each frame can be handled appropriately, using only the correspondence between frames, which is derived by tracking the feature points in the uncalibrated image sequence. This is possible because a special space, the projective grid space, which is determined by the two cameras, rather than Euclidean space, that is, ordinary three-dimensional space, is used for light field construction in the proposed method. In the projective grid space, the general geometrical relations among straight lines and planes in Euclidean space do not apply, but the position of the ray in space can be calculated by using the fundamental matrix. The proposed method is applied to uncalibrated image sequences taken by free hand-held video cameras. In experiments, the image from an arbitrary viewpoint is generated in a satisfactory way, indicating the usefulness of the proposed method.