Must-Know: What is the idea behind ensemble learning?
In ensemble methods, more diverse the models used, more robust will be the ultimate result.
Editor's note: This post was originally included as an answer to a question posed in our 17 More Must-Know Data Science Interview Questions and Answers series earlier this year. The answer was thorough enough that it was deemed to deserve its own dedicated post.
"In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone."
– Wikipedia.
Imagine you are playing the game “Who wants to be millionaire?” and reached up to last question of 1 million dollars. You have no clue about the question, but you have audience poll and phone a friend life lines. Thank God. At this stage you don’t want to take any risk, so what will you do to get sure-shot right answer to become millionaire?
You will use both life lines, isn’t it? Let’s say 70% audience is saying right answer is D and your friend is also saying the right answer is D with 90% confidence because he is an expert in the area of the question. Use of both life lines gives you an average 80% confidence that D is correct and gets you closer to becoming a millionaire.
This is the approach of ensemble methods.
The famous Netflix Prize competition took almost 3 years before the goal of 10% improvement was reached. The winners used gradient boosted decision trees to combine over 500 models.
In ensemble methods, the more diverse the models used, the more robust will be the ultimate result.
Different models used in ensemble improves overall variance from difference in population, difference in hypothesis generated, difference in algorithms used and difference in parametrization. There are main 3 widely used ensembles techniques:
- Bagging
- Boosting
- Stacking
Boosting
So if you have different models built for same data and same response variable, you can use one of the above methods to build ensemble model. As every model used in the ensemble has its own performance measures, some of the models may perform better than ultimate ensemble model and some of them may perform poorer than or equal to ensemble model. But overall the ensemble methods will improve overall accuracy and stability of the model, although at the expense of model understandability.
For more on ensemble methods see:
- Ensemble Methods: Elegant Techniques to Produce Improved Machine Learning Results
- Data Science Basics: An Introduction to Ensemble Learners
Related:
