LionBook Chapter 3: Learning requires a method
Real learning is associated with extracting the deep and basic relationships in a phenomenon, with summarizing with short models a wide range of events, with unifying different cases by discovering the underlying explanatory laws. This chapter explains the bias-variance dilemma.
Here is the latest chapter from LIONbook, a new book dedicated to "LION" combination of Machine Learning and Intelligent Optimization, written by the developers of LionSolver software, Roberto Battiti and Mauro Brunato.
This book will available for free from the web, chapter after chapter.
Chapters 1-2 cover Introduction and nearest neighbors.
Here is an excerpt:
Learning, both human and machine-based, is a powerful but subtle instrument. Real learning is associated with extracting the deep and basic relationships in a phenomenon, with summarizing with short models a wide range of events, with unifying different cases by discovering the underlying explanatory laws.
We are mostly interested in inductive inference, a kind of reasoning that constructs general propositions derived from specific examples, in a bottom-up manner.
In other words, learning from examples is only a means to reach the real goal: generalization, the capability of explaining new cases, in the same area of application but not already encountered during the learning phase.
One of the key issues is the bias-variance dilemma:
- Models with too few parameters are inaccurate because of a large bias: they lack flexibility.
- Models with too many parameters are inaccurate because of a large variance: they are too sensitive to the sample details (changes in the details will produce huge variations).
- Identifying the best model requires identifying the proper "model complexity", i.e., the proper architecture and number of parameters
Read Chapter 3 for an explanation how to deal with this dilemma.
After reading Chapter 3, do an exercise on training and testing (green building).