We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time.

http://www.graphics.stanford.edu/~niessner/thies2016face.html

Abstract

An e cient method has been presented to achieve an accurate dense 3D reconstruction of ob-

jects using photometric stereo without any prior knowledge of illumination settings i.e. light

source direction and light source intensity. Using the data of photometric images, an image

matrix

I

is constructed with its columns equal to number of photometric images captured and

rows equal to number of pixels in a photometric image. A per pixel initial surface normal esti-

mate is computed based upon Singular Value Decomposition (SVD) of the image matrix

I

. A

novel regularization technique has been applied on the initial normal estimate within the Graph

Cuts framework to regularize them and preserve the underlying discontinuities better. Thus,

photometric stereo problem has been solved under energy minimization framework. Finally, the

regularized surface normals are integrated to recover the surface of the object. The algorithm

has been tested on synthetic as well as real datasets and very encouraging results have been

obtained.