Deep learning - The "Average" Facial Representation

Hi all,

Question: I am doing some research on learning the "average" representation of a person's face from various facial images from the subjects.

I am thinking of maybe doing it like this:

1) Get an arbitrary small number (like 5) of the most clean frontal-face images of the person as targets (Y). Remaining images go in the X set.

2) Permute X and Y so that for each target of the subject in the set Y will be used as target for each entry in X. So if X has 20 images for subject1 and Y has 5 images of subject1 this means that the final training set would have 100 entries, 5 for each entry in X, for each subject.

3) Use a deep network in order to learn a facial representation for each subject that approximates all the targets for that subject with the least error. Thus acquiring (hopefully) an "average" of each of the subject's images in Y. (with optionally some extra constraints such as penalizing asymmetry

Now I do realize this is probably not very optimal, especially when it comes to approximating the average representation using 5 different targets for each entry in X, and using MSE will probably not do very well in this case.

So my question is: Has there been any research on this? Is there a better error measure, or method I could use?

Note: I am not looking to do sample reduction either. The targets Y will have the same resolution as inputs X