Interactive Machine Learning Experiments

Dive into experimenting with machine learning techniques using this open-source collection of interactive demos built on multilayer perceptrons, convolutional neural networks, and recurrent neural networks. Each package consists of ready-to-try web browser interfaces and fully-developed notebooks for you to fine tune the training for better performance.

comments

By Oleksii Trekhleb, Software Engineer at Uber.

I've open-sourced a new Interactive Machine Learning Experiments project on GitHub. Each experiment consists of Jupyter/Colab notebook (to see how a model was trained) and demo page (to see a model in action right in your browser).

Although the models may be a little dumb (remember, these are just experiments, not production-ready code), they will try to do their best to:

Recognize digits or sketches you draw in your browser
Detect and recognize the objects you'll show to your camera
Classify your uploaded image
Write a Shakespeare poem with you
Play with you in Rock-Paper-Scissors game

I've trained the models on Python using TensorFlow 2 with Keras support and then consumed them for a demo in a browser using React and JavaScript version of Tensorflow.

Models performance

First, let's set our expectations.️ The repository contains machine learning experiments and not a production-ready, reusable, optimized, and fine-tuned code and models. This is rather a sandbox or a playground for learning and trying different machine learning approaches, algorithms, and data-sets. Models might not perform well, and there is a place for overfitting/underfitting.

Therefore, sometimes you might see things like this:

But be patient, sometimes the model might get smarter and give you this:

Background

I'm a software engineer, and for the last several years now, I've been doing mostly frontend and backend programming. In my spare time, as a hobby, I decided to dig into machine learning topics to make it less like magic and more like math to myself.

Since Python might be a good choice to start experimenting with Machine Learning, I decided to learn its basic syntax first. As a result, a Playground and Cheatsheet for Learning Python project came out. This was just to practice Python and, at the same time, to have a cheat sheet of basic syntax once I need it (for things like dict_via_comprehension = {x: x**2 for x in (2, 4, 6)}, etc.).
After learning a bit of Python, I wanted to dig into the basic math behind Machine Learning. So after passing an awesome Machine Learning course by Andrew Ng on Coursera, the Homemade Machine Learning project came out. This time it was about creating a cheatsheet for basic machine learning math algorithms like linear regression, logistic regression, k-means, multilayer perceptron, etc.
The next attempt to play around with basic Machine Learning math was NanoNeuron. It was about 7 simple JavaScript functions that supposed to give you a feeling of how machines can actually "learn."
After finishing yet another awesome Deep Learning Specialization by Andrew Ng on Coursera, I decided to practice a bit more with multilayer perceptrons, convolutional, and recurrent neural networks (CNNs and RNNs). This time instead of implementing everything from scratch, I decided to start using some machine learning framework. I ended up using TensorFlow 2 with Keras. I also didn't want to focus too much on math (letting the framework do it for me), and instead, I wanted to come up with something more practical, applicable, and something I could try to play with right in my browser. As a result, new Interactive Machine Learning Experiments came out that I want to describe a bit more here.

Tech-stack

Models training

I used Keras inside TensorFlow 2 for modelling and training. Since I had zero experience with machine learning frameworks, I needed to start with something. One of the selling points in favor of TensorFlow was that it has both Python and JavaScript flavor of the library with similar API. So eventually, I used Python version for training and JavaScript version for demos.
I trained TensorFlow models on Python inside Jupyter notebooks locally and sometimes used Colab to make the training faster on GPU.
Most of the models were trained on good old MacBook's Pro CPU (2,9 GHz Dual-Core Intel Core i5).
Of course, there is no way you could run away from NumPy for matrix/tensor operations.

Models demo

I used TensorFlow.js to make predictions with previously trained models.
To convert Keras HDF5 models to TensorFlow.js Layers format, I used the TensorFlow.js converter. This might be inefficient to transfer the whole model (megabytes of data) to the browser instead of making predictions through HTTP requests, but again, remember that these are just experiments and not production-ready code and architecture. I wanted to avoid having a dedicated backend service to make architecture simpler.
The Demo application was created on React using a create-react-app starter with a default Flow flavour for type checking.
For styling, I used Material UI. It was, as they say, "to kill two birds" at once and try out a new styling framework (sorry, Bootstrap‍).

Experiments

So, in short, you may access Demo page and Jupyter notebooks by these links:

Experiments with Multilayer Perceptron (MLP)

A multilayer perceptron (MLP) is a class of feedforward artificial neural network (ANN). Multilayer perceptrons are sometimes referred to as "vanilla" neural networks (composed of multiple layers of perceptrons), especially when they have a single hidden layer.

Handwritten Digits Recognition

You draw a digit, and the model tries to recognize it.

Handwritten Sketch Recognition

You draw a sketch, and the model tries to recognize it.

Experiments with Convolutional Neural Networks (CNN)

A convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery (photos, videos). They are used for detecting and classifying objects on photos and videos, style transfer, face recognition, pose estimation, etc.

Handwritten Digits Recognition (CNN)

You draw a digit, and the model tries to recognize it. This experiment is similar to the one from the MLP section, but it uses CNN under the hood.

Handwritten Sketch Recognition (CNN)

You draw a sketch, and the model tries to recognize it. This experiment is similar to the one from the MLP section, but it uses CNN under the hood.

Rock Paper Scissors (CNN)

You play a Rock-Paper-Scissors game with the model. This experiment uses CNN that is trained from scratch.

Rock Paper Scissors (MobilenetV2)

You play a Rock-Paper-Scissors game with the model. This model uses transfer learning and is based on MobilenetV2.

Objects Detection (MobileNetV2)

You show the model your environment through your camera, and it will try to detect and recognize the objects. This model uses transfer learning and is based on MobilenetV2.

Image Classification (MobileNetV2)

You upload a picture, and the model tries to classify it depending on what it "sees" on the picture. This model uses transfer learning and is based on MobilenetV2.

Experiments with Recurrent Neural Networks (RNN)

A recurrent neural network (RNN) is a class of deep neural networks, most commonly applied to sequence-based data like speech, voice, text, or music. They are used for machine translation, speech recognition, voice synthesis, etc.

Numbers Summation

You type a summation expression (i.e., 17+38), and the model predicts the result (i.e., 55). The interesting part here is that the model treats the input as a sequence, meaning it learned that when you type a sequence 1 → 17 → 17+ → 17+3 → 17+38, it "translates" it to another sequence 55. You may think about it as translating a Spanish Hola sequence to English Hello.

Shakespeare Text Generation

You start typing a poem like Shakespeare, and the model will continue it like Shakespeare. At least it will try to do so.

Wikipedia Text Generation

You start typing a Wiki article, and the model tries to continue it.

Future plans

As I've mentioned above, the main purpose of the repository is to be more like a playground for learning rather than for production-ready models. Therefore, the main plan is to continue learning and experimenting with deep-learning challenges and approaches. The next interesting challenges to play with might be:

Emotions detection
Style transfer
Language translation
Generating images (i.e., handwritten numbers)

Another interesting opportunity would be to tune existing models to make them more performant. I believe it might give a better understanding of how to overcome overfitting and underfitting and what to do with the model if it just stuck on a 60% accuracy level for both training and validation sets and doesn't want to improve anymore.

I hope you might find some useful insights for models training from the repository or at least to have some fun playing around with the demos!

Original. Reposted with permission.

Related: