# Optimization (103)

**15 Python Snippets to Optimize your Data Science Pipeline**- Aug 25, 2021.

Quick Python solutions to help your data science cycle.**Numerics V: Integrality – When Being Close Enough is not Always Good Enough**- Jun 10, 2021.

Wow, already the fifth blog in this series…What is left to tell about numerics? There is another place where a MIP solver can sneak in minor violations that we have not yet discussed: The integrality conditions.**State of Mathematical Optimization Report, 2021**- May 28, 2021.

Download your copy of Gurobi's first-ever "State of Mathematical Optimization Report," which is based on data from a survey of commercial mathematical optimization users. Get yours now.**A Simple Way to Time Code in Python**- Mar 18, 2021.

Read on to find out how to use a decorator to time your functions.**Automating Machine Learning Model Optimization**- Mar 17, 2021.

This articles presents an overview of using Bayesian Tuning and Bandits for machine learning.**Speeding up Scikit-Learn Model Training**- Mar 5, 2021.

If your scikit-learn models are taking a bit of time to train, then there are several techniques you can use to make the processing more efficient. From optimizing your model configuration to leveraging libraries to speed up training through parallelization, you can build the best scikit-learn model possible in the least amount of time.**Bayesian Hyperparameter Optimization with tune-sklearn in PyCaret**- Mar 5, 2021.

PyCaret, a low code Python ML library, offers several ways to tune the hyper-parameters of a created model. In this post, I'd like to show how Ray Tune is integrated with PyCaret, and how easy it is to leverage its algorithms and distributed computing to achieve results superior to default random search method.**How to Speed up Scikit-Learn Model Training**- Feb 11, 2021.

Scikit-Learn is an easy to use a Python library for machine learning. However, sometimes scikit-learn models can take a long time to train. The question becomes, how do you create the best scikit-learn model in the least amount of time?**Optimization Algorithms in Neural Networks**- Dec 18, 2020.

This article presents an overview of some of the most used optimizers while training a neural network.**Algorithms for Advanced Hyper-Parameter Optimization/Tuning**- Nov 17, 2020.

In informed search, each iteration learns from the last, whereas in Grid and Random, modelling is all done at once and then the best is picked. In case for small datasets, GridSearch or RandomSearch would be fast and sufficient. AutoML approaches provide a neat solution to properly select the required hyperparameters that improve the model’s performance.**From Y=X to Building a Complete Artificial Neural Network**- Nov 13, 2020.

In this tutorial, we will start with the most simple artificial neural network (ANN) and move to something much more complex. We begin by building a machine learning model with no parameters—which is Y=X.**Strategies of Docker Images Optimization**- Oct 8, 2020.

Large Docker images lengthen the time it takes to build and share images between clusters and cloud providers. When creating applications, it’s therefore worth optimizing Docker Images and Dockerfiles to help teams share smaller images, improve performance, and debug problems.**Making Python Programs Blazingly Fast**- Sep 25, 2020.

Let’s look at the performance of our Python programs and see how to make them up to 30% faster!**Showcasing the Benefits of Software Optimizations for AI Workloads on Intel® Xeon® Scalable Platforms**- Sep 1, 2020.

The focus of this blog is to bring to light that continued software optimizations can boost performance not only for the latest platforms, but also for the current install base from prior generations. This means customers can continue to extract value from their current platform investments.**Rapid Python Model Deployment with FICO Xpress Insight**- Aug 20, 2020.

The biggest hurdle in the use of data to create business value, is indeed the ability to operationalize analytics throughout the organization. Xpress Insight is geared to reduce the burden on IT and address their critical requirements while empowering business users to take ownership of decisions and change management.**Autotuning for Multi-Objective Optimization on LinkedIn’s Feed Ranking**- Aug 19, 2020.

In this post, the authors share their experience coming up with an automated system to tune one of the main parameters in their machine learning model that recommends content on LinkedIn’s Feed, which is just one piece of the community-focused architecture.**Facebook Uses Bayesian Optimization to Conduct Better Experiments in Machine Learning Models**- Aug 10, 2020.

A research from Facebook proposes a Beyasian optimization method to run A/B tests in machine learning models.**Linear algebra and optimization and machine learning: A textbook**- May 18, 2020.

This book teaches linear algebra and optimization as the primary topics of interest, and solutions to machine learning problems as applications of these methods. Therefore, the book also provides significant exposure to machine learning.**Hyperparameter Optimization for Machine Learning Models**- May 7, 2020.

Check out this comprehensive guide to model optimization techniques.**Optimize Response Time of your Machine Learning API In Production**- May 1, 2020.

This article demonstrates how building a smarter API serving Deep Learning models minimizes the response time.**How to Do Hyperparameter Tuning on Any Python Script in 3 Easy Steps**- Apr 8, 2020.

With your machine learning model in Python just working, it's time to optimize it for performance. Follow this guide to setup automated tuning using any optimization library in three steps.**Google Open Sources TFCO to Help Build Fair Machine Learning Models**- Mar 12, 2020.

A new optimization framework helps to incorporate fairness constraints in machine learning models.**TensorFlow 2.0 Tutorial: Optimizing Training Time Performance**- Mar 5, 2020.

Tricks to improve TensorFlow training time with tf.data pipeline optimizations, mixed precision training and multi-GPU strategies.**Practical Hyperparameter Optimization**- Feb 13, 2020.

An introduction on how to fine-tune Machine and Deep Learning models using techniques such as: Random Search, Automated Hyperparameter Tuning and Artificial Neural Networks Tuning.**How to Optimize Your Jupyter Notebook**- Jan 30, 2020.

This article walks through some simple tricks on improving your Jupyter Notebook experience, and covers useful shortcuts, adding themes, automatically generated table of contents, and more.**How To “Ultralearn” Data Science: summary, for those in a hurry**- Dec 30, 2019.

For those of you in a hurry and interested in ultralearning (which should be all of you), this recap reviews the approach and summarizes its key elements -- focus, optimization, and deep understanding with experimentation -- geared toward learning Data Science.**How To “Ultralearn” Data Science: optimization learning, Part 3**- Dec 20, 2019.

This third part in a series about how to "ultralearn" data science will guide you through how to optimize your learning through five valuable techniques.**Enabling the Deep Learning Revolution**- Dec 5, 2019.

Deep learning models are revolutionizing the business and technology world with jaw-dropping performances in one application area after another. Read this post on some of the numerous composite technologies which allow deep learning its complex nonlinearity.**Building an intelligent Digital Assistant**- Oct 18, 2019.

In this second part we want to outline our own experience building an AI application and reflect on why we chose not to utilise deep learning as the core technology used.**There is No Such Thing as a Free Lunch**- Oct 11, 2019.

You have heard the expression “there is no such thing as a free lunch” – well in machine learning the same principle holds. In fact there is even a theorem with the same name.**The State of Transfer Learning in NLP**- Sep 13, 2019.

This post expands on the NAACL 2019 tutorial on Transfer Learning in NLP organized by Matthew Peters, Swabha Swayamdipta, Thomas Wolf, and Sebastian Ruder. This post highlights key insights and takeaways and provides updates based on recent work.**There is No Free Lunch in Data Science**- Sep 12, 2019.

There is no such thing as a free lunch in life or data science. Here, we'll explore some science philosophy and discuss the No Free Lunch theorems to find out what they mean for the field of data science.**Lagrange multipliers with visualizations and code**- Aug 6, 2019.

In this story, we’re going to take an aerial tour of optimization with Lagrange multipliers. When do we need them? Whenever we have an optimization problem with constraints.**From Data Pre-processing to Optimizing a Regression Model Performance**- Jul 19, 2019.

All you need to know about data pre-processing, and how to build and optimize a regression model using Backward Elimination method in Python.**XGBoost and Random Forest® with Bayesian Optimisation**- Jul 8, 2019.

This article will explain how to use XGBoost and Random Forest with Bayesian Optimisation, and will discuss the main pros and cons of these methods.**Top 8 Data Science Use Cases in Construction**- Jul 5, 2019.

This article considers several of the most efficient and productive data science use cases in the construction industry.**Optimization with Python: How to make the most amount of money with the least amount of risk?**- Jun 26, 2019.

Learn how to apply Python data science libraries to develop a simple optimization problem based on a Nobel-prize winning economic theory for maximizing investment profits while minimizing risk.**10 Gradient Descent Optimisation Algorithms + Cheat Sheet**- Jun 26, 2019.

Gradient descent is an optimization algorithm used for minimizing the cost function in various ML algorithms. Here are some common gradient descent optimisation algorithms used in the popular deep learning frameworks such as TensorFlow and Keras.**How to Automate Hyperparameter Optimization**- Jun 12, 2019.

A step-by-step guide into performing a hyperparameter optimization task on a deep learning model by employing Bayesian Optimization that uses the Gaussian Process. We used the gp_minimize package provided by the Scikit-Optimize (skopt) library to perform this task.**Monash University: Lecturer / Senior Lecturer – Optimisation [Melbourne, Australia]**- May 15, 2019.

Seeking outstanding academics to join our world-class team to deliver high-quality teaching and research that will help shape the future of optimisation for energy, transport, health and many other application domains. Positions are available for academics with strong skills in the development and application of techniques discrete optimisation research.**Linear Programming and Discrete Optimization with Python using PuLP**- May 8, 2019.

Knowledge of such optimization techniques is extremely useful for data scientists and machine learning (ML) practitioners as discrete and continuous optimization lie at the heart of modern ML and AI systems as well as data-driven business analytics processes.**KDnuggets™ News 19:n16, Apr 24: Data Visualization in Python with Matplotlib & Seaborn; Getting Into Data Science: The Ultimate Q&A**- Apr 24, 2019.

Best Data Visualization Techniques for small and large data; The Rise of Generative Adversarial Networks; Approach pre-trained deep learning models with caution; How Optimization Works; Building a Flask API to Automatically Extract Named Entities Using SpaCy**How Optimization Works**- Apr 18, 2019.

Optimization problems are naturally described in terms of costs - money, time, resources - rather than benefits. In math it's convenient to make all your problems look the same before you work out a solution, so that you can just solve it the one time.**Checklist for Debugging Neural Networks**- Mar 22, 2019.

Check out these tangible steps you can take to identify and fix issues with training, generalization, and optimization for machine learning models.**Deep Compression: Optimization Techniques for Inference & Efficiency**- Mar 20, 2019.

We explain deep compression for improved inference efficiency, mobile applications, and regularization as technology cozies up to the physical limits of Moore's law.**Artificial Neural Networks Optimization using Genetic Algorithm with Python**- Mar 18, 2019.

This tutorial explains the usage of the genetic algorithm for optimizing the network weights of an Artificial Neural Network for improved performance.**Deep Multi-Task Learning – 3 Lessons Learned**- Feb 15, 2019.

We share specific points to consider when implementing multi-task learning in a Neural Network (NN) and present TensorFlow solutions to these issues.**The Intuitions Behind Bayesian Optimization with Gaussian Processes**- Oct 19, 2018.

Bayesian Optimization adds a Bayesian methodology to the iterative optimizer paradigm by incorporating a prior model on the space of possible target functions. This article introduces the basic concepts and intuitions behind Bayesian Optimization with Gaussian Processes.**Recent Advances for a Better Understanding of Deep Learning**- Oct 1, 2018.

A summary of the newest deep learning trends, including Non Convex Optimization, Overparametrization and Generalization, Generative Models, Stochastic Gradient Descent (SGD) and more.**Essential Math for Data Science: ‘Why’ and ‘How’**- Sep 6, 2018.

It always pays to know the machinery under the hood (even at a high level) than being just the guy behind the wheel with no knowledge about the car.**Optimization 101 for Data Scientists**- Aug 8, 2018.

We show how to use optimization strategies to make the best possible decision.**Only Numpy: Implementing GANs and Adam Optimizer using Numpy**- Aug 6, 2018.

This post is an implementation of GANs and the Adam optimizer using only Python and Numpy, with minimal focus on the underlying maths involved.**Deep Learning Tips and Tricks**- Jul 4, 2018.

This post is a distilled collection of conversations, messages, and debates on how to optimize deep models. If you have tricks you’ve found impactful, please share them in the comments below!**Simple Tips for PostgreSQL Query Optimization**- Jun 22, 2018.

A single query optimization tip can boost your database performance by 100x. Although we usually advise our customers to use these tips to optimize analytic queries (such as aggregation ones), this post is still very helpful for any other type of query.**An Intuitive Introduction to Gradient Descent**- Jun 21, 2018.

This post provides a good introduction to Gradient Descent, covering the intuition, variants and choosing the learning rate.**Optimization Using R**- May 18, 2018.

Optimization is a technique for finding out the best possible solution for a given problem for all the possible solutions. Optimization uses a rigorous mathematical model to find out the most efficient solution to the given problem.**KDnuggets™ News 18:n12, Mar 21: Will GDPR Make Machine Learning Illegal?; 5 Things You Need to Know about Big Data**- Mar 21, 2018.

Also: A Beginner's Guide to Data Engineering - Part II; Introduction to Optimization with Genetic Algorithm; Introduction to Markov Chains; Your free 70-page guide to a career in data science**Introduction to Optimization with Genetic Algorithm**- Mar 14, 2018.

This article gives a brief introduction about evolutionary algorithms (EAs) and describes genetic algorithm (GA) which is one of the simplest random-based EAs.**3 principles for solving AI Dilemma: Optimization vs Explanation**- Feb 14, 2018.

We propose 3 principles for maximizing the benefits of machine learning without sacrificing its intelligence.**Using AutoML to Generate Machine Learning Pipelines with TPOT**- Jan 29, 2018.

This post will take a different approach to constructing pipelines. Certainly the title gives away this difference: instead of hand-crafting pipelines and hyperparameter optimization, and performing model selection ourselves, we will instead automate these processes.**Data Science vs Addiction: Estimating Opioid Abuse by Location**- Jan 26, 2018.

Data science can help find the optimal locations for drug treatment facilities, even in the face of major data challenges.**Managing Machine Learning Workflows with Scikit-learn Pipelines Part 3: Multiple Models, Pipelines, and Grid Searches**- Jan 24, 2018.

In this post, we will be using grid search to optimize models built from a number of different types estimators, which we will then compare and properly evaluate the best hyperparameters that each model has to offer.**Managing Machine Learning Workflows with Scikit-learn Pipelines Part 2: Integrating Grid Search**- Jan 19, 2018.

Another simple yet powerful technique we can pair with pipelines to improve performance is grid search, which attempts to optimize model hyperparameter combinations.**The LION WAY, v. 3.0: Machine Learning plus Intelligent Optimization – Free Download**- Jan 16, 2018.

This newly revised book presents two topics which are in most cases separated: machine learning (the design of flexible models from data) and intelligent optimization (the automated creation and selection of improving solutions). Free download!**Custom Optimizer in TensorFlow**- Jan 8, 2018.

How to customize the optimizers to speed-up and improve the process of finding a (local) minimum of the loss function using TensorFlow.**Understanding Objective Functions in Neural Networks**- Nov 23, 2017.

This blog post is targeted towards people who have experience with machine learning, and want to get a better intuition on the different objective functions used to train neural networks.**TensorFlow: What Parameters to Optimize?**- Nov 9, 2017.

Learning TensorFlow Core API, which is the lowest level API in TensorFlow, is a very good step for starting learning TensorFlow because it let you understand the kernel of the library. Here is a very simple example of TensorFlow Core API in which we create and train a linear regression model.**AI Conference in San Francisco, Sep 2017 – highlights and key ideas**- Sep 28, 2017.

Highlights from recent AI Conference include the inevitable merger of IQ and EQ in computing, Deep learning to fight cancer, AI as the new electricity and advice from Andrew Ng, Deep reinforcement learning advances and frontiers, and Tim O’Reilly analysis of concerns that AI is the single biggest threat to the survival of humanity.**Closing the Insights-to-Action Gap**- Sep 5, 2017.

There are many types of analytics for getting insight out of data, but the bigger and more difficult challenge is turning that insight into action. What should we do differently based on your findings?**What Is Optimization And How Does It Benefit Business?**- Aug 10, 2017.

Here we explain what Mathematical Optimisation is, and discuss how it can be applied in business and finance to make decisions.**When not to use deep learning**- Jul 24, 2017.

Despite DL many successes, there are at least 4 situations where it is more of a hindrance, including low-budget problems, or when explaining models and features to general public is required.

**Optimizing Web sites: Advances thanks to Machine Learning**- Jul 17, 2017.

Machine learning has revitalized a nearly dormant method, leading to a powerful approach for optimizing Web pages, finding the best of thousands of alternatives.**Optimization in Machine Learning: Robust or global minimum?**- Jun 30, 2017.

Here we discuss how convex problems are solved and optimised in machine learning/deep learning.**Introducing Dask-SearchCV: Distributed hyperparameter optimization with Scikit-Learn**- May 12, 2017.

We introduce a new library for doing distributed hyperparameter optimization with Scikit-Learn estimators. We compare it to the existing Scikit-Learn implementations, and discuss when it may be useful compared to other approaches.**Learning to Learn by Gradient Descent by Gradient Descent**- Feb 2, 2017.

What if instead of hand designing an optimising algorithm (function) we*learn*it instead? That way, by training on the class of problems we’re interested in solving, we can learn an optimum optimiser for the class!**5 Machine Learning Projects You Can No Longer Overlook, January**- Jan 2, 2017.

There are a lot of popular machine learning projects out there, but many more that are not. Which of these are actively developed and worth checking out? Here is an offering of 5 such projects, the most recent in an ongoing series.

**Game Theory Reveals the Future of Deep Learning**- Dec 29, 2016.

This post covers the emergence of Game Theoretic concepts in the design of newer deep learning architectures. Deep learning systems need to be adaptive to imperfect knowledge and coordinating systems, 2 areas with which game theory can help.**The hard thing about deep learning**- Dec 1, 2016.

It’s easy to optimize simple neural networks, let’s say single layer perceptron. But, as network becomes deeper, the optmization problem becomes crucial. This article discusses about such optimization problems with deep neural networks.**How to Make Your Database 200x Faster Without Having to Pay More**- Nov 22, 2016.

Waiting long for a BI query to execute? I know it’s annoyingly frustrating… It’s a major bottle neck in day-to-day life of a Data Analyst or BI expert. Let’s learn some of the easy to use solutions and a very good explanation of why to use them, along with other advanced technological solutions.**The Hard Problems AI Can’t (Yet) Touch**- Jul 11, 2016.

It's tempting to consider the progress of AI as though it were a single monolithic entity, advancing towards human intelligence on all fronts. But today's machine learning only addresses problems with simple, easily quantified objectives**Three Impactful Machine Learning Topics at ICML 2016**- Jul 1, 2016.

This post discusses 3 particular tutorial sessions of impact from the recent ICML 2016 conference held in New York. Check out some innovative ideas on Deep Residual Networks, Memory Networks for Language Understanding, and Non-Convex Optimization.**Angoss 9.6 Data Science Software Suite**- Apr 29, 2016.

Angoss software provides users with comprehensive scorecard building functionality that is fast, reliable, accurate, and business centric.**Metrics Gone Wrong – How Companies Are Optimizing The Wrong Way**- Apr 20, 2016.

A critique of the over-abundant and misguided pursuit of metric completeness, and how it can result in incorrect "optimization."**Stanford: Data Mining, Data Science Online Courses, Certificate**- Feb 22, 2016.

Take online Data Mining and Data Science courses with top Stanford faculty that count toward a Stanford Graduate Certificate. Spring quarter starts in March - enroll now.**Stanford Big Data Mining Certificates Online**- Nov 10, 2015.

Earn online certificates in Mining Massive Data Sets, Data Mining, Optimization, and more while learning from world-renowned Stanford experts.**YCML Machine Learning library on Github**- Aug 24, 2015.

YCML is a new Machine Learning library available on Github as an Open Source (GPLv3) project. It can be used in iOS and OS X applications, and includes Machine Learning and optimization algorithms.**HP Big Data Helps Ford to Better Manage Fleets and Personalize Employee Drives**- Jul 23, 2015.

HP-Ford partnership is leveraging Big Data for the next level of Telematics insights based intelligence.**Interview: Ksenija Draskovic, Verizon on Dissecting the Anatomy of Predictive Analytics Projects**- Apr 15, 2015.

We discuss Predictive Analytics use cases at Verizon Wireless, advantages of a unified data view, model selection and common causes of failure.**Interview: Lei Shi, ChinaHR.com on Analytics behind the Perfect Match**- Mar 6, 2015.

We discuss analytics at ChinaHR, matching job seekers and employers, traditional job fairs vs online recruitment, key metrics and analytical insights.**Interview: Jason Bloomberg, Intellyx on the Tricky Balance of Optimization and Innovation**- Feb 6, 2015.

We discuss Agile Digital Transformation, Optimization vs Innovation trade-off, best innovations of 2014, trends, advice and more.**FICO Analytics Competition: Helping Santa Helpers**- Dec 11, 2014.

FICO is sponsoring a Kaggle competition to optimize Santa's scheduling algorithm. Competitors can win in multiple categories and the best solution using FICO Xpress Optimization Suite will receive a special prize.**LION Intelligent Learning and Optimization News**- Nov 26, 2014.

LION intelligent learning and optimization adds full support for Java packages, new visualization neatly explains overfitting, and get "The LION way" book on Kindle (free if you qualify).**LIONoso, a non-profit tool for Machine Learning and Intelligent Optimization**- Nov 3, 2014.

LIONoso 2.1, developed by LIONlab for non-profit research and academic use, offers flexible training and experimentation for researchers who know what they are doing and want freedom in their ML experimentation.**Book: Modern Optimization with R**- Oct 10, 2014.

Learn the most relevant concepts related to modern optimization methods and how to apply them using multi-platform, open source, R tools in this new book on metaheuristics.**Resonate: Data Scientist**- Aug 29, 2014.

Performing complex data analysis on large datasets to further develop our campaign optimization and targeting algorithms.**Interview: Pallas Horwitz, Blue Shell Games on Why Data Science is So Critical for Gaming Studios**- Aug 14, 2014.

We discuss the role of data science at Blue Shell Games, the importance of "Lean Data", key metrics for online games, cross-product projects and optimizing meeting the data needs across an organization.**XLMiner solves Big Data Problems in Excel**- Jun 26, 2014.

XLMiner, a part of Analytic Solver Platform integrated software for predictive and prescriptive analytics - forecasting, data mining, optimization and simulation, lets you solve small or Big Data problems in Excel.**LION Resources for Teaching Machine Learning and Optimization**- Jun 23, 2014.

A great collection of resources for "LION: Learning and Intelligent Optimization" textbook includes slides, tutorial movies, exercises, use cases, and LIONoso - an academic version of LIONsolver software.**The Algorithm that Runs the World Can Now Run More of It**- Jun 13, 2014.

The most important algorithm, used for optimizing almost everything, is linear programming. New advances allow linear programming problems to be solved faster using the new commercial parallel simplex solver.**MADALGO Summer School on LEARNING AT SCALE, August 11-14, Denmark**- May 22, 2014.

MADALGO Summer School will teach the latest developments in learning at scale as applied to Big Data. Registration is free on a first-come-first serve basis. Denmark, Aug 11 - 14, 2014.**WCAI: Measuring Skill Level and Optimizing Player-Matching Algorithms in Online Games**- Mar 26, 2014.

New research opportunity from Wharton Customer Analytics Initiative (WCAI) involves a unique data set from a major gaming company, with historic behavioral data for 9.5 million users who played 882K games. Learn more on Apr 25.**Top stories for Mar 9-15: How Many Data Scientists?**- Mar 16, 2014.

How Many Data Scientists are out there? LIONbook: Machine Learning + Intelligent Optimization - completed, free personal download; Boston AnalyticsWeek: Big Data and Analytics Unconference, March 24-28; Upcoming Webcasts on Analytics, Big Data, Data Science.**LIONbook: Machine Learning + Intelligent Optimization – completed, free personal download**- Mar 11, 2014.

This book combines two usually separated topics: machine learning and intelligent optimization, and does it with enough technical details to satisfy professionals, but also with concrete examples, vivid images, and fun. Buy a low-cost paperback or ebook (Kindle), or download a free PDF.