Comet.ml – Machine Learning Experiment Management

By Gideon Mendels, Data Scientist and CEO, Comet.ml

Machine learning has made a significant shift from academia to industry in the last decade. The combination of large datasets, computing resources and significant investments have allowed researchers to push state-of-the-art results on most machine learning benchmarks.

However, we are missing the fundamental tools to manage machine learning teams and processes.

Over the past year, we conducted short interviews with over 200 data scientists from a variety of companies and research institutes. We asked them about their processes and their team dynamics. While we’re working to release a comprehensive post with our analysis, one particular data point stood out to us:

The majority of data scientists do not keep track of their previous work. Institutions and individual data scientists alike run into the risk of losing their experiment results.

How does a company retain its institutional knowledge if it does not keep track of its results?

The underlying issue isn’t best practices or lack of discipline, it’s the current state of tooling. GIT and other code version control systems are designed for software and not machine learning. They’re not designed to track datasets, results and other artifacts.

When it comes to sharing experiments, maintaining knowledge and publishing reproducible work whether if it’s in the public domain or within companies, we have yet to figure out best practices.

This article presents comet.ml – a platform that allows tracking machine learning experiments with an emphasis on collaboration and knowledge sharing. It allows you to keep your current workflow and report your result from any machine - whether it’s your laptop or a GPU cluster.

Your valuable training data will also stay safe and never leave your machine.

How does it work?

To use Comet.ml, simply add a few lines of code to your Python script (R support coming soon!).

For example, consider this short Keras code:

Every time this script would be executed, the following would be reported automatically (but in a configurable manner) to your private dashboard:

1. Hyperparameters: Command line arguments and everything contained within the model definition will be reported automatically. Additional hyper params can be easily added.
2. Metrics: Real time charts of your model converging collected from the ML library code. Additional metrics can be easily customizable.
3. Code (optional): A snapshot of your source code (single script or Github repository diff) to couple results and code together.
4. Standard output/error: Exceptions or outputs displayed without having to SSH to remote machines.
5. Model structure:  Serialized version of your model.
6. Dependencies: All python dependencies and their version.
7. Dataset Hash: A hash associated with your dataset computed locally to avoid cases where the underlying data changed.

Collaboration

Projects on Comet.ml can be shared with your entire team, a single colleague or publicly with the world. By providing visibility to the information collected above, we hope to make reproducibility easier. Collaborators can also comment on your work and ask questions.

Experiment Documentation

You can add rich documentation (Markup/Latex) to each experiment or a single project.

Comparison and analysis

You can compare experiments and pull experiment information to a local Pandas dataframe. This could be useful to look at hyperparameter importance and correlation.

Github Integration

Most of our users operate within a larger software oriented organization. With Comet.ml you can easily create a pull request to GitHub allowing you to commit the top performing model to any git branch.

What’s next?

We’re working on new features that will be released in the next few weeks:

• Hyper-parameter analysis -  Provides measurements of importance and correlation between different hyper-parameters including advanced visualization.
• Plot Builder -  Option to create your own plots and charts visualizing metrics, results, confusion metrics and even neural network activation maps.
• Production Monitoring - Provides measurements of  model drift, differences between training and inference data distributions and more.

Comet.ml is 100% free for public projects and academics. We also offer a 30 day trial for private projects if you want to take it for a spin before you commit. We hope Comet.ml will be helpful for your team and would love to hear your feedback.