Question about classification and novelty detection for facial recognition

I'm trying to write a facial recognition system for a personal project.

The system ideally should be able to classify ~20 people individually, but also be able to recognize when its given an image that is not of any of the 20 people. To further clarify, let there be people A, B, C, and D, and my system wants to be able to uniquely identify A and B. If given an image of A's face, the system should respond "A", given B respond "B", and given C or D should respond "unknown".

I have a pretty good idea about how to do the actual classification (know which models I'm going to use), but I'm wondering what the overall flow of the system should be. I have training data for all of the people I want to recognize. Anyone have any thoughts/experience on whether I should download some face database and train my model with all of those faces as "unknown", or would I be better off using some sort of novelty detection and only train on the faces I want to (uniquely) recognize? I'm a little bit worried about training on some arbitrary face database because then I'm giving the model some prior distribution about what fraction of faces are "unknown".