The roots of the Structure from Motion community can be traced back to two key fields, photogrammetry and computer vision. Although SfM problems account for a large portion of contemporary computer vision work, they have a long history and span several schools of thought.
Photogrammetry is a relatively old technique for measuring and processing lengths and angles in photographs for mapping purposes [47]. Reconstruction efforts were initially attempted using a pair of ground cameras separated by a fixed baseline. Pioneers in the 1840's include Arago, Jordan, Stolze and Laussedat who used cameras for estimating the shape of terrain from ground and aerial photographs, coining the name 'photogrammetrie' [39]. The arrival of airplane and space photography techniques spurred further development in the area. Estimates of motion from 2D photographs were used to rectify images into appropriate coordinates, mosaic multiple frames as well as estimate structure and elevation.
In the vision community, which was traditionally driven more from biology and AI roots, early achievements include the recovery of 3D scene structure from stereo by Marr and Poggio [37] where the correspondence is established automatically from two images via an iterative cooperative algorithm. The algorithm searches for unique matches of points between two images and recovers smooth disparity (an intermediate form of 3D depth) between them. Ullman [58] pioneered work on motion based reconstruction. The approach showed that four point correspondences over three views yield a unique solution 2 to motion and structure which could be solved via a nonlinear algorithm.
The formalism derived in photogrammetry and earlier vision research provided an important foundational theory for the SfM community. However, issues of implementation, stability, accuracy and so on have spurred numerous developments in the field. In addition, the goals and applications confronting the vision community have changed and hence emphasized different ways of thinking about the SfM problem. One of the key issues in the community is the use of linear versus nonlinear techniques. We discuss linear techniques, some of the critical issues in motion based structure estimation and the nonlinear techniques. Along the way, several camera models will be described as they are needed.