Can Deep Learning Help you Find the Perfect Girl? – Part 2

Using Deep Learning to find the perfect match, PhD student Harm de Vries describes the process of data collection and analysis. Finally, the results from matching algorithm are compared to human assessment for identifying an individual's dating preferences.



By Harm de Vries

online-date First part of article

Collecting data

Neural networks work especially well when we provide them a lot of labeled examples. So, all I needed to do was intensively use Tinder for some time.

But how do you save the liked and disliked pictures from the Tinder app? The application is communicating with a server to get new profile pictures, and by inspecting how the app did this communication, I could build a program on my laptop that mimics the tinder app (but saves the pictures). You can see the program at work in the screenshot. I have to say that the real Tinder app was a bit more convenient because it can be used anywhere and anytime. Now I really had to sit down and make time to swipe profiles. Nevertheless, I managed to label 9364 profile pictures in less than two months.

deep-neural-netAlmost 10K profile pictures sounds a lot, but in practice it was not enough to successfully train a deep neural network. What happened was that the network worked very well for the pictures it was trained on, but did not generalize to new profile pictures. Looking at the images, I could see that there was a lot of variation in the profile pictures (compared to the clean datasets we work with in academia). One way to account for those variations is to collect more data, but I didn't want to wait half a year before I had labeled enough profile pictures (or spent my whole day using Tinder..).

Instead, I focused on another way to overcome this issue: we can use another neural network that is successfully trained for a related task (for which there was enough data). The lower-layer filters learned by that network are recognizing patterns that might be useful for our task. We simply copy the trained network, and then only adjust the last layers to predict my attractiveness preferences.

I thought it would be a good idea to use a network to predict gender from profile pictures for two reasons. First, it is easy to collect data for this task. I scraped over 400 000 (!) male and female profile pictures from OKCupid.com in a couple of weeks. Second, the filters extracted by this network could be useful to predict my preferences, since I am probably more attracted to girls that do not look like men.

The data never lie..

Now that I had collected the data, I had to face the facts about my own Tinder behavior:
  • It was very confronting that I liked more than half (53%) of all profile pictures. Maybe I'm not that picky, but I also believe that Tinder shows you the popular profiles more frequently. There is a clear incentive for Tinder since you keep using the app when you hope to match with pretty girls. I can also imagine that more attractive people are using Tinder, and my swiped profile pictures are thus not representatives of the whole population (I think it would be weird if I was willing to date more than half of the population).
  • "What was I thinking when I liked this picture??" This feeling popped up several times when scrolling through the labeled pictures. It was tempting to do another round of labeling to filter out this unwanted likes. I realized that the problem is more general: for some pictures I do not have a clear opinion, and my mood determines the label. So, instead I estimated how much "noise" entered the labeling process. I relabeled a 100 random profile pictures a couple of weeks later and compared the results with the original labeling. I judged 12 pictures differently.
  • From the 12 errors I made, we could assume that I have guessed 24% of the time (because 24*0.5=12). You have to realize that this makes it difficult for any model to learn my preferences. If you are not consistent in the labeling, you are confusing the network what are important image patterns to learn.

 
The results

I'll spare you the details of processing all this data, and simply summarize the results:
  • The neural network trained on profile pictures from OKcupid predicted 92.7% of the time correctly if there was a male or female in the image. It took about 3 days to train this network on one of the fastest graphical cards today. It achieves near human performance because I achieved myself around 95% on this dataset. There were a lot of pictures with both males and females in the picture, and also encountered some pictures without somebody in the picture (apparently there are people who put a picture of a plate of food as their profile picture).
  • Given the success of the gender network, I had good hope that this network could be used to predict my preferences. Unfortunately, this wasn't the case, this network only achieved 61.5% accuracy.
  • However, I used another network that was trained (not by me) to distinguish all kinds of animals, plants and vehicles. Somewhat surprising, predicting attractiveness using this network worked quite well: 68.1% accuracy.


deep-learning-jason-prediction

Almost 70% does not sound that impressive, right? But how well could someone else predict to which girls I am attracted?

My co-author Jason Yosinski tried it. He 'trained' by first looking at all 50 dislike and 50 like pictures side by side, scrolling through them all a few times. Then the training set was shuffled, pictures were displayed one a time, and he produced a label prediction after each photo. The correct label was shown after each picture so that he could learn from his mistakes. See his study program in the screenshot above. He achieved 86%, 82%, 88% and 88% on 4 training rounds.

He then predicted the labels for 100 unseen profile pictures (the same ones as in my consistency test). He made 24 errors, and thus achieved 76% accuracy. Given that's also hard for other people to predict my preferences, the 68% is not that bad!

The future

The techniques I have used are not tailored to my preferences, and could be used for other people whenever they provide their labeled profiles pictures. In fact, deep learning is known to be working much better with bigger datasets, so I am pretty confident that this approach would be even more successful when I train a neural network on data gathered from many people (e.g. when I have access to all data from Tinder). As far as I know, the most popular dating websites (like OKCupid.com) still use questionaires to suggest potential partners. I think the success of Tinder has shown that people care about somebody's appearance, and with the rise of deep learning we now have a tool to actually incorporate this kind of information in matching algorithms.

The time that you were predetermined to marry one of the 50 girls/boys of your town is over. New technologies will continue to change the way how we find our romantic partners. Although randomness sometimes determines love, we should embrace the great opportunities in machine learning to help anyone looking for a soulmate!

harmHarm de Vries is a PhD student supervised by Aaron Courville and Roland Memisevic, and a member of the Montreal Institute for Learning Algorithms headed by Yoshua Bengio. His main interests are in the theory and application of deep learning.

Original.

Related: