Hyperparameter Optimization: 10 Top Python Libraries

Become familiar with some of the most popular Python libraries available for hyperparameter optimization.

Hyperparameter Optimization: 10 Top Python Libraries
Image by Author


Hyperparameter optimization plays a crucial role in determining the performance of a machine learning model. They are one the 3 components of training.


3 Components of Models


Training data

Training data is what the algorithm leverages (think: instructions to build a model) to identify patterns.



Algorithm 'learns' by adjusting parameters, such as weights, based on training data to make accurate predictions, which are saved as part of the final model.



Hyperparameters are variables that regulate the process of training and are constant during the training process.


Different Types of Search


Grid Search

Training models with every possible combination of the provided hyperparameter values a time-consuming process.


Random Search

Training models with randomly samples hyperparameter values from the defined distributions, a more effective search.


Having Grid Search

Training models with all values, and then repeatedly "halving" the search space by only considering the parameter values that performed the best in the previous round.


Bayesian Search

Starting with an initial guess of values, using performance of the model to the values. It's like how a detective might start with a list of suspects, then use new information to narrow down the list.


Python Libraries for Hyperparameter Optimization

I found these 10 Python libraries for hyperparameter optimization.



You can tune estimators of almost any ML, DL package/framework, including Sklearn, PyTorch, TensorFlow, Keras, XGBoost, LightGBM, CatBoost, etc with a real-time Web Dashboard called optuna-dashboard.



Optimizing using Bayesian optimization, including conditional dimensions.



Different searches such as GridSearchCV or HalvingGridSearchCV.



AutoML and a drop-in replacement for a scikit-learn estimator.



Very easy to learn but extremly versatile providing intelligent optimization.



Provides distinct approaches such plethora of score functions.



Automatic save/learn from Experiments for persistent optimization



AutoML creating Markdown reports from ML pipeline



With Bayesian Optimization, Hyperband, and Random Search algorithms built-in



Hyperparameter Optimization for TensorFlow, Keras and PyTorch.

Have I forgotten any libraries?


Maryam Miradi is an AI and Data Science Lead with a PhD in Machine Learning and Deep learning, specialised in NLP and Computer Vision. She has 15+ years of experience creating successful AI solutions with a track record of delivering over 40 successful projects. She has worked for 12 different organisations in a variety of industries, including Detecting Financial Crime, Energy, Banking, Retail, E-commerce, and Government.