Gary B. Huang, Honglak Lee, and Erik Learned-Miller.
Learning Hierarchical Representations for Face Verification with Convolutional Deep Belief Networks.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.

This page shows all pairs from View 2 of LFW that were incorrectly classified by all representations used in our system. As mentioned in the paper, the pairs that are correctly classified by few to none of the representations are heavily skewed toward matched pairs. Of the 50 pairs incorrectly classified by all representations, only three are of mismatched pairs, highlighted in red below. This highlights the large intra-class variations as a primary challenge in unconstrained face verification, and suggests either trying to incorporate such knowledge directly into the classifier, or learning such knowledge in a less supervised manner, such as from video.

The above pair is actually correctly labeled as mismatched - see LFW Errata, Jim_OBrien