LIONbook Chapter 10: Statistical Learning Theory and Support Vector Machines (SVM)
The LIONbook on machine learning and optimization, written by co-founders of LionSolver software, is provided free for personal and non-profit usage. Chapter 10 looks at Statistical Learning Theory and Support Vector Machines (SVM).
Here is the latest chapter from LIONbook, a new book dedicated to "LION" combination of Machine Learning and Intelligent Optimization, written by the developers of LionSolver software, Roberto Battiti and Mauro Brunato.
This book will available for free from the web, chapter after chapter.
Here are previous chapters:
- Chapters 1-2: Introduction and nearest neighbors.
- Chapter 3: Learning requires a method
- Chapter 4: Linear models
- Chapter 5: Mastering generalized linear least-squares
- Chapter 6: Rules, decision trees, and forests
- Chapter 7: Ranking and selecting features
- Chapter 8: Specific nonlinear models
- Chapter 9: Neural networks, shallow and deep.
You can also download the entire book here.
The latest chapter is Chapter 10: Statistical Learning Theory and Support Vector Machines (SVM).
The order of chapters in this book has some connections with the history of machine learning. Before 1980, most learning methods concentrated either on symbolic "rule-based" expert systems, or on simple sub-symbolic linear discrimination techniques, with clear theoretical properties. In the eighties, decision trees and neural networks paved the way to efficient learning of nonlinear models, but with little theoretical basis and naive optimization techniques (based on gradient descent).
In the nineties, efficient learning algorithms for non-linear functions based on statis- tical learning theory developed, mostly through the seminal work by Vapnik and Chervonenkis.
Statistical learning theory (SLT) deals with fundamental questions about learning from data.
- Under which conditions can a model learn from examples?
- How can the measured performance on a set of examples lead to bounds on the generalization performance?
These theoretical results are everlasting, although the conditions for the theorems to be valid are almost impossible to check for most practical problems. In another direction, the same researchers proposed a resurrection of linear separability methods, with additional ingredients intended to improve generalization, with the name of Support Vectors Machines (SVM).