KDnuggets Home » News » 2017 » Sep » Tutorials, Overviews » The Search for the Fastest Keras Deep Learning Backend ( 17:n37 )

The Search for the Fastest Keras Deep Learning Backend


This is an overview of the performance comparison for the popular Deep Learning frameworks supported by Keras – TensorFlow, CNTK, MXNet and Theano.



By Jasmeet Bhatia, Microsoft.

Disclaimer: Jasmeet Bhatia is a Data & AI Solution Architect currently working for Microsoft. Opinions expressed in this post are his own.

If there are any doubts in regards to the popularity of Keras among the Data Scientist/Engineer community and the mindshare it commands, you just need to look at the support it has been receiving from all major AI and Cloud players. Currently the official Keras release already supports Google's TensorFlow and Microsoft's CNTK deep learning libraries besides supporting other popular libraries like Theano. Last year Amazon Web Services announced its support for Apache MXNet, another powerful Deep Learning library and few weeks ago support for Keras was added to the MXNet's next release candidate. As of now MXNet only seems to support Keras v1.2.2 and not the current Keras release 2.0.5.  

Image

Although it is possible to deploy Keras models in production with any of the supported backends, developers and solution architects should keep in mind that Keras, by nature of being a high-level API for the different DL frameworks, doesn't yet support tweaking of all underlying parameters offered by the individual libraries. So in use cases where you would want to fine tune all parameters offered by the backend frameworks, you might be better off using one of the deep learning frameworks directly instead of using Keras as the top layer. This of course might change in future as additional functionalities are added to both Keras and the backend libraries. But having said that, Keras still remains an excellent tool that can fit into earlier stages of most deep learning development projects as it enables data scientists and engineers to quickly build and test complex deep learning models.

Keras also enables developers to quickly test relative performance across multiple supported deep learning frameworks. A single parameter in Keras configuration file dictates what deep learning framework would be used as the backend. So, you can build a single model and without changing any code at all, you can run it on TensorFlow, CNTK and Theano. As for MXNet, since it only supports Keras ver1.2.2 right now, some minimal changes in code are required but this might change soon. These individual frameworks can obviously be fine-tuned further using features present in the individual libraries but Keras still provides an excellent opportunity to compare the base performance between these libraries.

There have already been a few articles comparing the relative performance of backends supported by Keras but with each new release for Keras or the individual deep learning libraries, we are seeing drastic improvements in performance.

So let’s see how did the recent releases for the different deep learning frameworks performed in the recent matchup.

Let’s first go over the setup used for the tests.

All performance tests were executed on Azure NC6 VM with Nvidia Tesla K80 GPU. The VM image used was Azure DSVM (Data Science Virtual Machine) on Ubuntu. The image comes pre-installed with Keras, TensorFlow, Theano and MXNet besides other data science tools. For tests, all packages were updated to recent releases. To use MXNet, older Keras package 1.2.2 was used. Additional details on Azure DSVM: https://docs.microsoft.com/en-us/azure/machine-learning/machine-learning-data-science-virtual-machine-overview

 

Configuration:

 
Due to dependencies for each framework, I had to run the tests in three configurations, as shown below:

Image

Image

For all frameworks latest available stable releases were used for testing. All frameworks have their next beta versions available which claim to improve performance and are probably good to use for research purpose, but for production applications, preference is to use the stable releases. Hence these beta versions are not included in the performance tests.

 

Performance Tests:

 
To compare the performance of the DL frameworks, I used 5 different test models described below. To ensure no particular framework got any special treatment, all models were sourced from Keras/examples repository maintained on GitHub.

The test code/notebooks are available in my GitHub repo - https://github.com/jasmeetsb/deep-learning-keras-projects

Note: In two of the tests, MXNet was left out. This is because MXNet doesn’t yet support newer Keras functions and scripts would have needed significant changes before running on MXNet. This would have defeated the purpose of this exercise. Even the 3 tests that were run on MXNet needed some minimal changes to the script, mostly due to renaming of some of the Keras functions in their recent releases.

 
1. Test - CIFAR10 CNN

Learning Model Type: Convolutional Neural Network (CNN)
Datasets/Tasks:  CIFAR10 small images dataset

Objective: Classify images into 10 classes

In terms of training speed per epoch, TensorFlow wins by a slight margin over MXNet.
In terms of accuracy/convergence speed CNTK seems to have a slight edge till 25th iteration but by 50th iteration all frameworks display similar accuracy.

Image

Image

In terms of training speed per epoch, TensorFlow wins by a slight margin over MXNet.

In terms of accuracy/convergence speed CNTK seems to have a slight edge till 25th iteration but by 50th iteration all frameworks display similar accuracy.

 
2. Test - MNIST CNN

Learning Model Type: CNN
Datasets/Tasks: MNIST handwritten digit dataset

Description: Classify images into 10 classes/digits

In this test TensorFlow is the winner in terms of training speed followed closely by Theano but in terms of accuracy/convergence speed, all frameworks showcase similar characteristics.

Image

Image

In this test TensorFlow is a clear winner in terms of training speed but in terms of accuracy/convergence speed, all frameworks showcase similar characteristics.