Programming Project #3 (proj3)CS180/280A: Intro to Computer Vision and Computational Photography
Programming Project #3 (proj3)
In this assignment you will produce a "morph" animation of your face into someone else's face, compute the mean of a population of faces and extrapolate from a population mean to create a caricature of yourself.
A morph is a simultaneous warp of the image shape and a cross-dissolve of the image colors. The cross-dissolve is the easy part; controlling and doing the warp is the hard part. The warp is controlled by defining a correspondence between the two pictures. The correspondence should map eyes to eyes, mouth to mouth, chin to chin, ears to ears, etc., to get the smoothest transformations possible.
To start with, you should take a pictures of yourself on a uniform background (for instance, white). Your image should be the same size and aspect ratio as your target face (for instance, this beautiful portrait of George, taken by Martin Schoeller). Treat your target face as a passport photo template -- your face in the picture should be about where their face is. You can use photo editing software to resize/crop your image such that this is the case. This will make your morphing result more pleasing to the eye.
Use your photo as Picture A and your target person's photo as Picture B. You'll morph still picture A into still picture B and produce 45 frames of animation numbered
0-45, where frame 0 must be identical to picture A and frame 45 must be
identical to picture B. In the video, each frame will be displayed for 1/30 of a second (ie. 30 fps). Create a video from your sequence of frames, either a YouTube or an animated gif.
First, you will need to define pairs of corresponding points on the two images by hand (the more points, the better the morph, generally). The simplest way is probably to use the cpselect (matlab) tool or write your own little tool using ginput (matlab or python) and plot commands (with hold on and hold off ). In order for the morph to work you will need a consistent labeling of the two faces. So label your faces A and B in a consistent manner using the same ordering of keypoints in the two faces. It's strongly recommended that you save the points once you obtain something you are happy with so that you don't have to do all that clicking more than once!
Now, you need to provide a triangulation of these points that will be used for morphing. You can compute a triangulation any way you like, or even define it by hand. A Delaunay triangulation (see delaunay and related functions) is a good choice since it does not produce overly skinny triangles. You can compute the Delaunay triangulation on either of the point sets (but not both -- the triangulation has to be the same throughout the morph!). But the best approach would probably be to compute the triangulation at a midway shape (i.e. mean of the two point sets) to lessen the potential triangle deformations.
Note if you don't want to implement your own labeling tool, you can use this one from a last year's student.
Before computing the whole morph sequence, compute the mid-way face of your images A and B. This would involve: 1) computing the average shape (a.k.a the average of each keypoint location in the two faces), 2) warping both faces into that shape, and 3) averaging the colors together. The main task in warping the faces into the average shape is implementing an affine warp for each triangle in the triangulation from the original images into this new shape. This will involve computing an affine transformation matrix A between two triangles:
A = computeAffine(tri1_pts,tri2_pts)
(You will write this function.)
A set of these transformation matrices will then need to be used to implement an inverse warp (as discussed in class) of all pixels. One way to do so is to generate a mask directly using polygon in python or roipoly in MATLAB. Interpolation functions may be useful for this implementation, but you are not permitted to use any built-in functions for computing transforms.
Note that you only need to write one loop here, looping over all of the triangles. Don't loop over the pixels!!!Show us the original A and B images as well as the image of the mid-way face that you got.
You need to write a function:
morphed_im = morph(im1, im2, im1_pts, im2_pts, tri, warp_frac, dissolve_frac);
that produces a warp between im1 and im2 using point correspondences defined in im1_pts and im2_pts (which are both n-by-2 matrices of (x,y) locations) and the triangulation structure tri . The parameters warp_frac and dissolve_frac control shape warping and cross-dissolve, respectively. In particular, images im1 and im2 are first warped into an intermediate shape configuration controlled by warp_frac, and then cross-dissolved according to dissolve_frac. For interpolation, both parameters lie in the range [0,1]. They are the only parameters that will vary from frame to frame in the animation. For your starting frame, they will both equal 0, and for your ending frame, they will both equal 1.
The output of this part should be a video sequence of a morph from your image A to image B. Please do NOT put a video file on your website! Either include a link to a YouTube video, or create an animated gif.
Pick a freely available dataset of annotated faces (for instance the Danes or this one or something from here or ask for permission to use this one). Using the keypoints already annotated on the data:
Show the mean image that you got, as well as 1) your face warped into the average geometry, and 2) the average face warped into your geometry.
Produce a caricature of your face by extrapolating from the population mean you calculated in the last step. This might work better on a gender specific mean or some other characteristics specific mean.
In summary, this project will require you to perform the following and describe any implementation or design details on your project webpage:
