KDnuggets Home » News » 2017 » Sep » Tutorials, Overviews » The Search for the Fastest Keras Deep Learning Backend ( 17:n37 )

The Search for the Fastest Keras Deep Learning Backend


 
  http likes 67

This is an overview of the performance comparison for the popular Deep Learning frameworks supported by Keras – TensorFlow, CNTK, MXNet and Theano.



3. Test - MNIST MLP 

Learning Model Type: Multilayer Perceptron/Deep NN
Datasets/Tasks: MNIST handwritten digit dataset

Objective: Classify images into 10 classes/digits

In a standard Deep neural network test using MNIST dataset, CNTK, TensorFlow and Theano achieve similar scores (2.5 – 2.7 s/epoch) but MXNet blows it out of the water with 1.4s/epoch timing. MXNet also showcases a slight edge in regards to accuracy/convergence speed.

Image

Image

In a standard Deep neural network test using MNIST dataset, CNTK, TensorFlow and Theano achieve similar scores (2.5 – 2.7 s/epoch) but MXNet blows it out of the water with 1.4s/epoch timing. MXNet also showcases a slight edge in regards to accuracy/convergence speed.

 
4. Test - MNIST RNN

Learning Model Type: Hierarchical Recurrent Neural Network (HRNN)
Datasets/Tasks: MNIST handwritten digit dataset

Objective: Classify images into 10 classes/digits

CNTK and MXNet have similar performance (162 – 164 s/epoch) in terms of training speed followed by TensorFlow at 179s/epoch. Theano seems to lag significantly for RNN models.

Image

Image

CNTK and MXNet have similar performance (162 – 164 s/epoch) in terms of training speed followed by TensorFlow at 179s/epoch. Theano seems to lag significantly for RNN models.

 
5. Test - BABI RNN

Learning Model Type: Recurrent Neural Network (RNN)
Datasets/Tasks: bAbi Project (https://research.fb.com/downloads/babi/)

Objective: Train two recurrent neural networks based upon a story and a question. The resulting merged vector is then queried to answer a range of bAbi tasks.

Results: MXNet was left out as the sample script from Keras repo required changes. TensorFlow and Theano had comparable performance with CNTK being 50% faster at 9.5s/epoch.

Image

Image

 

Summary:

 
Image

  • TensorFlow performed the best in both the CNN tests but lagged behind in the RNN test cases.
  • CNTK performed significantly better than TensorFlow and Theano in the Babi RNN and MNIST RNN tests but was slower than TensorFlow in CNN Test Cases.
  • MXNet seemed to perform slightly better than CNTK and TensorFlow in RNN test and significantly better than all frameworks in the MLP test but the fact that it does not support Keras v2 functions makes it difficult to do a straight up comparison without modifying the test code. This would hopefully be resolved soon.
  • Theano had a marginal edge over TensorFlow and CNTK in the deep Neural Network (MLP) test.

 

Conclusion:

 
As seen in the results above, all frameworks have their areas of strength but at this moment there is no single framework that outperforms all others across the board. CNTK shines in RNN use cases, TensorFlow in CNN and MXNet although showing very promising performance still has some ground to cover to support all Keras functionality. As is often the case in open source world, all these frameworks are constantly being enhanced giving them better performance and making them easier to use and deploy in production. When considering these deep learning frameworks for production, performance is a prime consideration but in most cases, you also need to consider the ease of deployment and other auxiliary tools that come as part of these tools which help you manage your production machine learning models. That discussion will probably require a post on its own but I hope my analysis above at least provides you with some additional insights.

 
AuthorBio: Jasmeet Bhatia is a Data and AI Solution Architect currently working for Microsoft. Jasmeet has over 12 years of combined industry and consulting experience with multiple Fortune 500 clients in building strategy and implementing solutions that focus on solving complex business problems by utilizing information assets. Jasmeet’s experience ranges across AI/Machine Learning, Cloud, Big Data, IoT, Advanced Analytics, Business Intelligence, process improvement and change management.

Related:


Sign Up

By subscribing you accept KDnuggets Privacy Policy