Data Mining History: The Invention of Support Vector Machines

The story starts in Paris in 1989, when I benchmarked neural networks against kernel methods, but the real invention of SVMs happened when Bernhard decided to implement Vladimir Vapnik algorithm.



By Isabelle GuyonIsabelle Guyon.

The invention of Support Vector Machines using the kernel trick in 1991 is my most important contribution.

I had the chance of working in the 1990's in Larry Jackel's group at Bell Labs with great people, some of whom became famous ( Vladimir Vapnik, Yann LeCun, Leon Bottou, Yoshua Bengio, and Corinna Cortes). This was a very stimulating and competitive environment.

The story starts in Paris in 1989, when I benchmarked neural networks against kernel methods (Parzen windows, potential functions, and least-square kernel regression from Tommy Poggio's work) with my PhD advisors Gerard Dreyfus and Leon Personnaz. The same year, two physicists working close-by (Marc Mezard and Werner Krauth) published a paper on an optimal margin algorithm called 'minover,' which attracted my attention. But it was not until I joined Bell Labs that I put things together.

At the time, everybody was working on multi-layer Perceptrons (the ancestors of deep learning), and my first work on optimal margin algorithms was simply some active set method to enlarge their margin, inspired by the 'minover.' Bernhard Boser, my husband, was even making dedicated hardware for MLPs! I had heated discussions with Vladimir Vapnik, who shared my office and was pushing another optimal margin algorithm that he invented in the 1960's.

guyon-svm-1 The invention of SVMs happened when Bernhard decided to implement Vladimir's algorithm in the three months we had left before we moved to Berkeley. After some initial success of the linear algorithm, Vladimir suggested introducing products of features. I proposed to rather use the kernel trick of the 'potential function' algorithm. Vladimir initially resisted the idea because the inventors of the 'potential functions' algorithm ( Aizerman, Braverman, and Rozonoer) were from a competing team of his institute back in the 1960's in Russia!

But Bernhard tried it anyways, and the SVMs were born!

guyon-svm-kernel


Gregory Piatetsky: I sent Isabelle the link to History of Data Mining, which mentions her and Vapnik historic invention of Support Vector Machines, and asked if she had a description of that discovery. She sent me a link to a recent Facebook post which had this story, but I felt that it deserved a wider audience. So, with her permission, we reposted the story here.


Original.

Bio: After many years working as a consultant in Machine Learning Isabelle Guyon joined Paris-Saclay University as a full professor while continuing to serve as President of ChaLearn, a California non-profit dedicated to organizing challenges in Machine Learning. She was born in Paris and currently commutes between Paris and California.