arXiv Paper Spotlight: Automated Inference on Criminality Using Face Images

This recent paper addresses the use of still facial images in an attempt to differentiate criminals from non-criminals, doing so with the help of 4 different classifiers. Results are as troubling as they are unsettling.

Are the faces of a society's criminals significantly different than those of the non-criminals?


A recent paper by Xiaolin Wu (McMaster University, Shanghai Jiao Tong University) and Xi Zhang (Shanghai Jiao Tong University), titled "Automated Inference on Criminality using Face Images," explores this very idea. The research is based on the study of still images of the faces of criminals and non-criminals, and uses 4 classification techniques in an attempt to discern, namely logistic regression, K-Nearest Neighbors, Support Vector Machines, and Convolutional Neural Networks.

The study controls for race, gender, age, and facial expressions, and "nearly half" of the faces in the dataset were of convicted criminals. All 4 classification methods performed consistently well, with the Convolutional Neural Network outperforming the other methods:

As expected, the state-of-the-art CNN classifier performs the best, achieving 89.51% accuracy. The relatively high accuracy of CNN is also paralleled by all other three classifiers which are only few percentage points behind in the success rate of classification.

Classifier accuracy

There has been significant historical controversy surrounding the idea of appearance-based inference of criminality, yet this research claims to empirically establish its validity. The study also finds common discriminating structural features in the pursuit of criminality prediction, including "lip curvature, eye inner corner distance, and the so-called nose-mouth angle."

Average faces

What is striking from the above images are that the "average" criminal and non-criminal faces certainly appear to be quite different from one another. From the abstract:

The variation among criminal faces is significantly greater than that of the non-criminal faces. The two manifolds consisting of criminal and non-criminal faces appear to be concentric, with the non-criminal manifold lying in the kernel with a smaller span, exhibiting a law of normality for faces of non-criminals. In other words, the faces of general law-biding public have a greater degree of resemblance compared with the faces of criminals, or criminals have a higher degree of dissimilarity in facial appearance than normal people.

Criminal vs. non-criminal spectrums

This all leads to various questions and concerns, however, especially those from a civil liberties point of view. Recall from this past spring when a face recognition app sparked outrage and concern that public anonymity may be a thing of the past, given that images could be compared to 200+ million profiles on a Russian social networking site and identify individuals with 70% accuracy. And there was no insinuation of criminality in that case. A study such as this only reiterates that care and caution are needed when deciding when, where, and how to utilize technologies like these. Research of this type and of this level of discomfort is not going away, and so it must be dealt with head-on.

Certainly further work needs to be done to strengthen this research. The samples were ethnically homogeneous, the same sex, and on a narrowly bounded age spectrum, and so future work would need to expand on, at a minimum, these features. Larger datasets would also obviously need to be investigated, regardless of the ethnic, age, and sex makeup of the individual faces involved.

The abstract can be found here, while this is a direct link to the paper.