Direct estimation of motion and extended scene structure from a moving stereo rig

We investigate the relationship between the kinematics (infinitesimal motion model) of a calibrated Stereo Rig and point and line image feature measurements seen at two time instances of the rig's motion (four images in all). In particular, we are interested in the byproduct of this analysis providing a direct connection between the spatio-temporal derivatives of the images at two time instances and kinematics of the 3D motion of the Rig. We establish a fundamental result showing that 3 quadruples of point-line-line-line matches (i.e., point in the reference image and lines coincident with the corresponding points in the remaining three images) are sufficient for a unique linear solution for the kinematics of the rig. In other words, the projected instantaneous motion of 'one and a half' 3D lines is sufficient for recovering the kinematics of the moving rig. In particular, spatio-temporal derivatives across 3 points are sufficient for a direct estimation of the rig's motion. Consequently, we describe a new direct estimation method for motion estimation and 3D reconstruction from stereo image sequences obtained by a stereo rig moving through a rigid world. Correspondences (optic flow) are not required as spatio-temporal derivative are used instead. One can then use the images from both pairs combined, to compute a dense depth map. Finally, since the basic equations are linear, we combine the contribution coming from all pixels in the image using a Least Squares approach.