ebook: Fundamentals for Efficient ML Monitoring
We've gathered best practices for data science and engineering teams to create an efficient framework to monitor ML models. This ebook provides a framework for anyone who has an interest in building, testing, and implementing a robust monitoring strategy in their organization or elsewhere.
Today's enterprises rely on machine learning-powered predictions to guide business strategy, such as by forecasting demand and mitigating risk. For an increasing number of businesses, machine learning (ML) predictions underpin their core business model, like financial institutions that use ML models to approve or reject loan applications.
As machine learning is drastically different from other software or traditional IT, models risk degrading the moment they are pushed into production – where the hyperdynamic nature of the data meets the hypersensitivity of the models. These “drifts” in the data structure, or other properties that cause model degradation, are too often silent and unobservable.
In the last few months, triggered by the COVID-19 crisis, we have all witnessed companies struggling to fix corrupted, business-critical models. One of the most documented of such issues was Instacart, whose inventory prediction model’s accuracy declined from 93% to 61%, leaving a sour after-taste for their customers and their teams.
Rare are the data science and engineering teams who are prepared for this “Day 2” , the day their models meet the real world; as they invest the majority of their time researching, training, and evaluating models. While it’s clear that teams want to address any potential issues before they arise, there is a lack of clear processes, tools and requirements for production systems. Today, the industry still lacks guidelines of what an optimal ML infrastructure should look like
That's why we've gathered best practices for data science and engineering teams to create an efficient framework to monitor ML models. The ebook provides a framework for anyone who has an interest in building, testing, and implementing a robust monitoring strategy in their organization or elsewhere. You will learn:
- Best practices for monitoring your models in production
- Proven ways to catch drifts, biases and anomalies at the right time
- Recommendations to avoid alert fatigue
List of Fundamentals
- Look beyond performance measurements
- Use different metrics for different features
- Use a granular point of view
- Avoid overflow and detect events automatically
- Comparing different versions in parallel
- Monitor protected features as proxies to ensure fairness
- Ensure your monitoring is platform agnostic
- Empower all stakeholders
- Use your production insights for other stages of your ML process
Download the eBook to learn more about the best practices out there to refine your ML monitoring strategy, and avoid the pitfalls of "Day 2."