Bayesian Hyperparameter Optimization with tunesklearn in PyCaret
PyCaret, a low code Python ML library, offers several ways to tune the hyperparameters of a created model. In this post, I'd like to show how Ray Tune is integrated with PyCaret, and how easy it is to leverage its algorithms and distributed computing to achieve results superior to default random search method.
By Antoni Baum, Core Contributor to PyCaret and Contributor to Ray Tune
Here’s a situation every PyCaret user is familiar with: after selecting a promising model or two from compare_models()
, it’s time to tune its hyperparameters to squeeze out all of the model’s potential with tune_model()
.
from pycaret.datasets import get_data
from pycaret.classification import *
data = get_data("juice")
exp = setup(
data,
target = "Purchase",
)
best_model = compare_models()
tuned_best_model = tune_model(best_model)
(If you would like to learn more about PyCaret — an opensource, lowcode machine learning library in Python, this guide is a good place to start.)
By default, tune_model()
uses the tried and tested RandomizedSearchCV
from scikitlearn. However, not everyone knows about the various advanced options tune_model()
provides.
In this post, I will show you how easy it is to use other stateoftheart algorithms with PyCaret thanks to tunesklearn, a dropin replacement for scikitlearn’s model selection module with cutting edge hyperparameter tuning techniques. I’ll also report results from a series of benchmarks, showing how tunesklearn is able to easily improve classification model performance.
Random search vs Bayesian optimization
Hyperparameter optimization algorithms can vary greatly in efficiency.
Random search has been a machine learning staple and for a good reason: it’s easy to implement, understand and gives good results in reasonable time. However, as the name implies, it is completely random — a lot of time can be spent on evaluating bad configurations. Considering that the amount of iterations is limited, it’d make sense for the optimization algorithm to focus on configurations that it considers promising by taking into account already evaluated configurations.
That is, in essence, the idea of Bayesian optimization (BO). BO algorithms keep track of all evaluations and use the data to construct a “surrogate probability model”, which can be evaluated a lot faster than a ML model. The more configurations have been evaluated, the more informed the algorithm becomes, and the closer the surrogate model becomes to the actual objective function. That way the algorithm can make an informed choice about which configurations to evaluate next, instead of merely sampling random ones. If you would like to learn more about Bayesian optimization, check out this excellent article by Will Koehrsen.
Fortunately, PyCaret has builtin wrappers for several optimization libraries, and in this article, we’ll be focusing on tunesklearn.
tunesklearn in PyCaret
tunesklearn is a dropin replacement for scikitlearn’s model selection module. tunesklearn provides a scikitlearn based unified API that gives you access to various popular state of the art optimization algorithms and libraries, including Optuna and scikitoptimize. This unified API allows you to toggle between many different hyperparameter optimization libraries with just a single parameter.
tunesklearn is powered by Ray Tune, a Python library for experiment execution and hyperparameter tuning at any scale. This means that you can scale out your tuning across multiple machines without changing your code.
To make things even simpler, as of version 2.2.0, tunesklearn has been integrated into PyCaret. You can simply do pip install "pycaret[full]"
and all of the optional dependencies will be taken care of.
How it all works together
!pip install "pycaret[full]"
from pycaret.datasets import get_data
from pycaret.classification import *
data = get_data("juice")
exp = setup(
data,
target = "Purchase",
)
best_model = compare_models()
tuned_best_model_hyperopt = tune_model(
best_model,
search_library="tunesklearn",
search_algorithm="hyperopt",
n_iter=20
)
tuned_best_model_optuna = tune_model(
best_model,
search_library="tunesklearn",
search_algorithm="optuna",
n_iter=20
)
Just by adding two arguments to tune_model()
you can switch from random search to tunesklearn powered Bayesian optimization through Hyperopt or Optuna. Remember that PyCaret has builtin search spaces for all of the included models, but you can always pass your own, if you wish.
But how well do they compare to random search?
A simple experiment
In order to see how Bayesian optimization stacks up against random search, I have conducted a very simple experiment. Using the Kaggle House Prices dataset, I have created two popular regression models using PyCaret — Random Forest and Elastic Net. Then, I tuned both of them using scikitlearn’s Random Search and tunesklearn’s Hyperopt and Optuna Searchers (20 iterations for all, minimizing RMSLE). The process was repeated three times with different seeds and the results averaged. Below is an abridged version of the code — you can find the full code here.
from pycaret.datasets import get_data
from pycaret.regression import *
data = get_data("house")
exp = setup(
data,
target = "SalePrice",
test_data=data, # so that the entire dataset is used for cross validation  do not normally do this!
session_id=42,
fold=5
)
rf = create_model("rf")
en = create_model("en")
tune_model(rf, search_library = "scikitlearn", optimize="RMSLE", n_iter=20)
tune_model(rf, search_library = "tunesklearn", search_algorithm="hyperopt", n_iter=20)
tune_model(rf, search_library = "tunesklearn", search_algorithm="optuna", optimize="RMSLE", n_iter=20)
tune_model(en, search_library = "scikitlearn", optimize="RMSLE", n_iter=20)
tune_model(en, search_library = "tunesklearn", search_algorithm="hyperopt", n_iter=20)
tune_model(en, search_library = "tunesklearn", search_algorithm="optuna", optimize="RMSLE", n_iter=20)
Isn’t it great how easy PyCaret makes things? Anyway, here are the RMSLE scores I have obtained on my machine:
Experiment’s RMSLE scores
And to put it into perspective, here’s the percentage improvement over random search:
Percentage improvement over random search
All of that using the same number of iterations in comparable time. Remember that given the stochastic nature of the process, your mileage may vary. If your improvement is not noticeable, try increasing the number of iterations (n_iter
)from the default 10. 20–30 is usually a sensible choice.
What’s great about Ray is that you can effortlessly scale beyond a single machine to a cluster of tens, hundreds or more nodes. While PyCaret doesn’t support full Ray integration yet, it is possible to initialize a Ray cluster before tuning — and tunesklearn will automatically use it.
exp = setup(
data,
target = "SalePrice",
session_id=42,
fold=5
)
rf = create_model("rf")
tune_model(rf, search_library = "tunesklearn", search_algorithm="optuna", optimize="RMSLE", n_iter=20) # Will run on Ray cluster!
Provided all of the necessary configuration is in place ( RAY_ADDRESS
environment variable), nothing more is needed in order to leverage the power of Ray’s distributed computing for hyperparameter tuning. Because hyperparameter optimization is usually the most performanceintensive part of creating an ML model, distributed tuning with Ray can save you a lot of time.
Conclusion
In order to speed up hyperparameter optimization in PyCaret, all you need to do is install the required libraries and change two arguments in tune_model()
— and thanks to builtin tunesklearn support, you can easily leverage Ray’s distributed computing to scale up beyond your local machine.
Make sure to check out the documentation for PyCaret, Ray Tune and tunesklearn as well as the GitHub repositories of PyCaret and tunesklearn. Finally, if you have any questions or want to connect with the community, join PyCaret’s Slack and Ray’s Discourse.
Thanks to Richard Liaw and Moez Ali for proofreading and advice.
Bio: Antoni Baum is a Computer Science and Econometrics MSc student, as well as a core contributor to PyCaret and contributor to Ray Tune.
Original. Reposted with permission.
Related:
 Algorithms for Advanced HyperParameter Optimization/Tuning
 5 Tools for Effortless Data Science
 5 Things You Are Doing Wrong in PyCaret
Top Stories Past 30 Days

