A Breakdown of Deep Learning Frameworks

Deep Learning continues to evolve as one of the most powerful techniques in the AI toolbox. Many software packages exist today to support the development of models, and we highlight important options available with key qualities and differentiators to help you select the most appropriate for your needs.

What is a Deep Learning Framework?

A deep learning framework is a software package used by researchers and data scientists to design and train deep learning models. The idea with these frameworks is to allow people to train their models without digging into the algorithms underlying deep learning, neural networks, and machine learning.

These frameworks offer building blocks for designing, training, and validating models through a high-level programming interface. Widely used deep learning frameworks such as PyTorch, TensorFlow, MXNet, and others can also use GPU-accelerated libraries such as cuDNN and NCCL to deliver high-performance multi-GPU accelerated training.

Why Use a Deep Learning Framework?

  • They supply readily available libraries for defining layers, network types (CNNs, RNNs), and common model architectures
  • They can support computer vision applications; image, speech, and natural language processing
  • They have familiar interfaces via popular programming languages such as Python, C, C++, and Scala
  • Many deep learning frameworks are accelerated by NVIDIA deep learning libraries such as cuDNN, NCCl, and cuBLAS for GPU accelerated deep learning training


Example Frameworks


Framework Qualities Differentiators
  • Easy to use - well defined APIs, documentation
  • Flexible - ideal for researching and prototyping new ideas
  • Multiple tools for building on top of TensorFlow such as TensorFlow Slim, Scikit Flow, PrettyTensor, Keras, and TFLearn
  • TensorFlow Lite allows for deployment on mobile and embedded devices
  • JavaScript library can deploy models via the web browser and Node.js
  • Great community engagement and support
  • Large body of existing TensorFlow samples and code, accelerates research
  • Computational graph visualizations via TensorBoard
  • Python interface
Aesara (successor to Theano)
  • Automatic differentiation as a symbolic expression
  • Computation graph optimizations and on-the-fly code generation for speed, numerical stability, and memory usage
  • Low-level and flexible for research of new ideas
  • Python-based, with NumPy integration
  • No multi GPU
  • Designed for computer vision framework problems
  • Too rigid for researching new algorithms
  • Caffe is in maintenance mode
  • NVIDIA fork of Caffe is maintained and updated by NVIDIA
  • Delivers optimized performance on the latest GPU hardware
  • Caffe2 is now a part of PyTorch, and the APIs are being deprecated
  • PyTorch is based on Python. It is the successor of Torch, which was based on the Lua programming language
  • Primary audience is researchers
  • Supports dynamic computational graphs
  • PyTorch 1.0 is a new iteration that includes PyTorch merged with Caffe2 (current stable version is 1.9.0 from June 2021)
  • Now the primary framework used at Facebook/Used by Facebook FAIR for research
  • Extremely flexible for research
  • Shares the same backend as the popular Torch framework it was based on
  • Models that are fast to prototype and easier to debug
  • CuPy: NumPy-equivalent multi-dimensional CUDA® array-library
  • Extensions & Tools: ChainerRL, ChainerMN, for computer vision
  • Is now in maintenance mode since the company behind it is changing their primary framework to PyTorch
  • Dynamic computation graphs with a Python API are strengths of Chainer and PyTorch
Apache MXNet
  • Primary focus is training and deploying deep neural networks
  • Matlab support
  • Portable and scalable, including multi-GPU and multi-node support
  • Supports 8 language bindings, including C++, Python, Julia, Java, Clojure, R, Scala, and Perl
  • Allows for a mix of symbolic and imperative programming
  • Beyond just DL classifiers: combine image/vision processing with DL
  • Automates distributed training and deployment to a data center and embedded without manual coding
  • High productivity desktop IDE (integrated development environment) that makes research, prototyping, and debugging easy
  • MATLAB expert technical support
  • Easily integrates into existing MATLAB and simulink workflows


More Details on TensorFlow


An open-source software library created by Google, TensorFlow is a popular tool for machine learning, especially for training deep neural networks. TensorFlow’s API primarily supports Python and C, but there are also APIs for C++, JavaScript, and Java; however, only the Python API is guaranteed to be stable.

TensorFlow’s community has also developed support for a number of other languages, including C#, Haskell, Julia, R, Ruby, Rust, and Scala.

The advantage of TensorFlow is that it has so many entry points. Beyond languages, there is a wide range of tools that integrate with or are built on top of TensorFlow.

TensorFlow also has a very large community of users where you can get help, and it’s well documented.


More Details on Keras


Keras is an open-source library that’s focused on providing a simple Python API for neural networks with features such as scalability for GPU clusters. It’s built on top of TensorFlow 2.0 and can also run on Theano.

Keras has the same portability as TensorFlow, meaning you can run it in a browser, as well as mobile and embedded devices. Keras is used by a number of major organizations, including CERN and NASA.


More Details on PyTorch


PyTorch is another product of the FAANG ecosystem coming from Facebook’s AI Research lab (FAIR). PyTorch is largely focused on computer vision and natural language processing (NLP) tasks. Like TensorFlow, PyTorch’s primary interface language is Python, but there is also C++ support.

PyTorch’s community has a number of tools that integrate with the library, such as Skorch for scikit-learn compatibility, extBrewer for NLP, NeMo toolkit for conversational AI, and PyTorch Lightning which is similar in idea to TensorFlow and Keras in that it’s focused on simplifying the coding required to get a model working.

PyTorch is also a good stand-in for NumPy (a popular tool in machine learning and data science) with tensors, which are like NumPy arrays but optimized to run on CPUs or GPUs. PyTorch has an experimental deployment method for mobile devices but is optimized to run on cloud computing platforms, including Amazon Web Services, Google Cloud, and Microsoft Azure.

There are a wide number of deep learning frameworks to choose from. If one of the options listed here doesn’t suit your needs, there are others, including Amazon’s Gluon (based on MXNet), DL4J, and Sonnet.


Original. Reposted with permission.