Publications - Fitting a 3D Morphable Model to Edges

Fitting a 3D Morphable Model to Edges: A Comparison Between Hard and Soft Correspondences


Summary: A simple, novel, robust and fully automatic algorithm for fitting a 3D Morphable Model to a single image using landmarks and edge features.

Estimating 3D face shape from one or more 2D images is a longstanding problem in computer vision. It has a wide range of applications from pose-invariant face recognition to creation of 3D avatars from 2D images. One of the most successful approaches to this problem is to use a statistical model of 3D face shape. This transforms the problem of shape estimation to one of model fitting and provides a strong statistical prior to constrain the problem.

Image edges convey important information about a face. The occluding boundary provides direct information about 3D shape, for example a profile view reveals strong information about the shape of the nose. Internal edges, caused by texture changes, high curvature or self occlusion, provide information about the position and shape of features such as lips, eyebrows and the nose. This information provides a cue for estimating 3D face shape from 2D images or, more generally, for fitting face models to images.

Fitting a 3DMM to a 2D image using only geometric features (i.e. landmarks and edges) is essentially a non-rigid alignment problem. Surprisingly, the idea of employing an iterated closest point approach with hard edge correspondences (in a similar manner to ASM fitting) has been discounted in the literature. In this study, we pursue this idea and develop an iterative 3DMM fitting algorithm that is fully automatic, simple and efficient. Instead of working in a transformed distance-to-edge space and treating correspondences as "soft", we compute an explicit correspondence between model and image edges. This allows us to treat the model edge vertices as a landmark with known 2D position, for which optimal pose or shape estimates can be easily computed.


Our approach is as follows:

  1. 1. Detect facial landmarks
  2. 2. Initialise shape and pose estimates by fitting to landmarks only
  3. 3. Obtain high quality fit using iterated closest edge fitting
  4. 4. Nonlinear optimisation of hybrid objective function
    containing landmark, edge and prior terms

We show qualitative examples from the CMU PIE and the Labeled Faces in the Wild datasets. In the first example, we fit to images (first row) in a non-frontal pose using automatically detected landmarks and show the reconstruction in the second row. We texture map the image onto the mesh, rotate to frontal pose (bottom row) and compare to an actual frontal view (third row). In the second example, again, we texture map the image to the mesh and show a range of poses.

Our Matlab implementation is available here. The summary video is available on YouTube.