Francois Chollet on the Future of Keras and Reinforce Conference

Sponsored Post.

Francois Chollet, the creator of Keras, will be speaking at the Reinforce AI conference in Budapest, on April 6-7, about Keras’ evolution and Tensorflow integration. Ahead of the conference, we asked Chollet several questions about the future and the directions of Keras. Read his answers below.

Use coupon code KDNuggets to get 15% off conference tickets.

What are the most important features you plan to add to Keras in 2020?

Francois Chollet: Keras in 2020 is continuing its evolution as an end-to-end framework for deep learning applications. Here are some of the big new features we've launched recently or are about to launch:

Preprocessing layers and redesigned image preprocessing API:
historically we've delegated preprocessing to auxiliary tools written in NumPy and PIL (the Python Imaging Library). However, this kind of external preprocessing makes models less portable: every time someone reuses a model you've trained, they need to also recreate the preprocessing pipeline. And that pipeline has to behave in the exact same way as the original one -- or the model will break. This becomes a significant issue when you want to port your model to JavaScript, for instance, which is otherwise easy to do for Keras models. So, the new approach we're taking is to make preprocessing part of the model, via "preprocessing layers". This covers things like text standardization, tokenization, vectorization, image normalization and random data augmentation. It enables models to accept raw text or raw images as input.

Keras Tuner: this is a next-generation hyperparameter tuning framework built for Keras. We've seen a lot of excitement around this tool already, and very strong adoption at Google. It solves the massive pain point of hyperparameter tuning for ML practitioners and researchers, with a simple and very Kerasic workflow.

AutoKeras: this project brings Automated Machine Learning (AutoML) to the real world, with a super usable, Keras-like API. It's built on top of Keras and Keras Tuner. You can use it to define and train a ML model in just 3 lines of code -- and thanks to automatic search across the space of possible models, that initial 3-line model will already be quite performant. For more advanced users, AutoKeras also gives you a deep level of control over how the configuration of the search space and the search process.

Cloud Keras: this is still at the prototype stage, and will soon go into beta. The vision is to enable you to take any Keras script that can run locally on your laptop or in a Colab notebook, and in a single line of code, launch a distributed training job in the cloud. You'll get essentially the same workflow as developing locally -- no need to worry about cluster configuration and Docker containers -- but your experiments will run extraordinarily fast.

Greater integration with TFX (TensorFlow Extended, a platform for managing production ML apps), and better support for exporting models to TF Lite (a ML execution engine for mobile and embedded devices). We think great support for production use cases is critical to the success of Keras.

How do you see Keras evolution vs PyTorch ?

Francois Chollet: I think comparing TensorFlow/Keras and PyTorch is really comparing apples to oranges. PyTorch is a Python library for defining and training deep learning models. Which is perhaps 10% of a typical ML workflow. But we're more of a ML platform that supports end-to-end use cases for the real world. Dataset management, scaling training to 27,000 GPUs, arbitrarily scalable hyperparameter tuning, deploying a production model to an API, in the browser, on mobile, or on an embedded device -- you name it, we can do it. TensorFlow/Keras is powerful. And we have 3x to 5x more users than PyTorch, which reflects this difference.

Considering that training large Deep Learning models can consume a lot of energy and potentially contribute to global warming, what about Keras or TensorFlow support for energy-efficient computation?

Francois Chollet: Training deep learning models is computationally intensive, especially if you're doing hyperparameter tuning or architecture search. Now, whether this contributes to CO2 emissions is entirely a matter of the source of the electricity used. If you build your datacenter in a place where there's abundant and cheap hydroelectric power, your deep learning models can be carbon-free. In the case of Google -- a big consumer of deep learning model training -- the company actually releases reports about the carbon-intensiveness of its operations, if you're interested in that. Google already purchases an amount of renewable power that matches 100% of its consumption, and has made a commitment to run entirely on renewable power in the near future.

As for Keras specifically -- there are several ways you can reduce your energy footprint. The first one, of course, is to train smaller models and to be more focused in your use of hyperparameter search and architecture search. I really think that Keras Tuner and AutoKeras can help with that, by democratizing more intelligent search methodologies, as opposed to merely brute-forcing a large search space.

For a given training run, one thing you can do is use mixed precision. We have an easy-to-use (currently experimental) API in Keras for mixed precision during training. This can reduce the compute-intensiveness of your model by around 30% on average.

For production models that only do inference, we have a suite of tools to help you optimize them and make them as lightweight as possible: the TensorFlow model optimization toolkit. It makes it easy to do weight pruning and weight quantization. This is especially if you want your model to run on mobile without training the battery, or on low-power embedded devices such as microcontrollers. But even in the case of a model running on a regular server, a well-optimized model can significantly reduce your power consumption and operations costs.

---

Francois Chollet will be speaking at the Reinforce AI conference in Budapest, on April 6-7, about Keras’ evolution and Tensorflow integration. Csaba Szepesvari from DeepMind will also speak next to David Aronchick from Microsoft who previously also worked for Google and co-founded Kubeflow, and Reza Zadeh from Stanford, a member of the Technical Advisory Board for Databricks. In his talk, Reza explains how to turn ML research into real-world products.

Use code KDNuggets to get 15% off conference tickets.