What's in a Face? Metric Learning for Face Characterization

O. Sendik, D. Lischinski, D. Cohen-Or

Research output: Contribution to journalArticlepeer-review

3 Scopus citations

Abstract

We present a method for determining which facial parts (mouth, nose, etc.) best characterize an individual, given a set of that individual's portraits. We introduce a novel distinctiveness analysis of a set of portraits, which leverages the deep features extracted by a pre-trained face recognition CNN and a hair segmentation FCN, in the context of a weakly supervised metric learning scheme. Our analysis enables the generation of a polarized class activation map (PCAM) for an individual's portrait via a transformation that localizes and amplifies the discriminative regions of the deep feature maps extracted by the aforementioned networks. A user study that we conducted shows that there is a surprisingly good agreement between the face parts that users indicate as characteristic and the face parts automatically selected by our method. We demonstrate a few applications of our method, including determining the most and the least representative portraits among a set of portraits of an individual, and the creation of facial hybrids: portraits that combine the characteristic recognizable facial features of two individuals. Our face characterization analysis is also effective for ranking portraits in order to find an individual's look-alikes (Doppelgängers).

Original languageAmerican English
Pages (from-to)405-416
Number of pages12
JournalComputer Graphics Forum
Volume38
Issue number2
DOIs
StatePublished - May 2019

Bibliographical note

Publisher Copyright:
© 2019 The Author(s) Computer Graphics Forum © 2019 The Eurographics Association and John Wiley & Sons Ltd. Published by John Wiley & Sons Ltd.

Keywords

  • CCS Concepts
  • Computing methodologies → Neural networks
  • Image processing
  • face recognition
  • facial hybrids
  • feature polarization
  • neural networks

Fingerprint

Dive into the research topics of 'What's in a Face? Metric Learning for Face Characterization'. Together they form a unique fingerprint.

Cite this