4 Steps to ensure your AI/Machine Learning system survives COVID-19

By Zank Bennett, Bennet Data Science.

COVID-19 is all over the news. And rightfully so. It’s a tremendously important topic. And it’s having massive effects on the accuracy of AI models.

Before we dive into how AI has changed, let us first say from all of us here at Bennett Data Science that we sincerely hope that you are well and getting through the hardships as best you can.

 

COVID-19 and AI

 

Businesses are undergoing tremendous change. It’s hard to find an industry that isn’t drastically affected by COVID-19. In attempts to keep up with either precipitous constriction or massive immediate scaling, companies are seeing their tried and tested methods of interacting with their customers rendered almost meaningless. That means marketing personalization, lifetime value calculations, product recommendations, and churn models, are all inaccurate at best and misleading at worst.

 

Here’s Why

 

AI uses past actions to predict future events. And nothing in the (recent) past is anything like what we’re experiencing today. So AI is confused. It’s as though we’re all suddenly speaking a new language to Alexa or Siri or Google Assistant. Of course they wouldn’t “understand”. They’d be useless. Global purchase behavior is now speaking a different language. And AI doesn’t understand.

Companies are scaling the only way they know how, by putting workers (people) in the trenches to make decisions about how to support rapidly changing customer demand. And they’re doing it largely without the AI they’re used to relying on because the models are not accurate anymore.

We’re seeing companies going back to making business decisions based on recent data from dashboards in place of scalable, automated AI. And for good reason. In many cases, AI is useless because it’s never seen anything like COVID-19.

This often means that teams are overwhelmed by all the data to process, sift through, and make actionable. Because it’s not enough to “discover” an insight. Companies need to act on the insightful information. And machines can do that at scale, company-wide. Humans cannot. So, things back up.

While this sounds grim, there’s a lot that companies can do to keep their intelligent systems working for them and their customers in this ever-changing environment. But before we get to this, it’s important to understand the specific hurdles they encounter.

 

COMPANIES ARE STRUGGLING

 

I’ve had several conversations over the past few weeks to discuss how various businesses are fairing, and how AI is working for them, given the current state of the world. The majority of executives and data scientists I talked to pointed towards the importance of spending more time looking at their data to understand how recent developments changed their business.

Data scientist Oleg Żero looked at it holistically, “The upcoming corona-induced crisis will do the same as any other crisis did. It will sink the heavy and stiff and propel the flexible.

From a higher level, reactions to what’s going on are all over the map.”

To his point, some of our clients are doing ok, having had distributed teams from the beginning, but comfort levels vary widely from industry to industry, and many companies are struggling to understand what the future will bring. E-commerce and delivery services, on the other hand, are set to thrive in the short-term. Amazon alone is looking to hire 100k employees.

But for every business thriving, there are many more in dire condition. Either way, everything has changed, and companies need to adapt. Reza Sohrabi from StitchFix points this out as well, “I don't think there are any businesses that are not impacted by the current pandemic. So, I believe companies have to adapt to the new situation and change gears as to how they do data science.”

One of the big challenges we’re seeing and hearing is that AI-generated predictions are unable to forecast future events, given current market and customer behavior fluctuations. Predictive models rely on historical data to predict future events. And when this historical data of customer behavior is nothing like what those customers are doing now, we have a problem. Predictions go to rubbish.

Alan Murray from Fortune explains it well: “… modern-day ‘prediction machines’ are often based on data drawn from past behavior. They aren’t prepared to deal with massive shifts in behavior—for instance when people inexplicably start hoarding toilet paper.”

Many different industries are currently impacted by changing data such as extremely long supermarket lines, erratic shopping behavior, and spikes in traffic to sites like public health and financial centers, reporting what appear to be denial-of-service attacks.

Companies are seeing signals that, just a few weeks ago, would have been anomalies or outliers. These are the new normal now. And most AI is ill-equipped to deal with them.

There is, however, something that can be done.

 

THE SOLUTION

 

When we think about how to help companies in difficult times, we come back to the fundamentals: companies must engage their customers with products they’ll love. It’s more important now than ever to understand what your customers really want and give it to them when they want it. It’s also a great time to look at how organizations work with data and utilize analytics.

Eric King summarizes it nicely, “There's not a better time to pull the lens back and look more organizationally at how companies are going to compete in the long run, as opposed to the disruption that's occurring this quarter.” - Eric A. King, President and Founder for The Modeling Agency

After all, the goal of effective analytics is to continually progress in the maintenance of customer lifetime value and all of its drivers.

These drivers of lifetime value include:

• Giving customers engaging content and offers when they’re looking for them.
• Keeping churn low by identifying patterns that generally lead to churn and using proven churn mitigation tactics.
• Delivering personalized marketing messages.

These are important fundamentals to get right. But it’s early 2020, and customer preferences around the world are quite literally all over the map.

If you have a business where personalization matters, there are a few changes that will help you retain value from data science:

1. Dive into your data
   The most important objective for data-driven companies is to dive into data dashboards. Look at the major assumptions that are driven by your business objectives. Do they still hold up? Is there still data (or enough data) coming in to support these main objectives? For example, if you sell products, have the products that people are buying changed drastically? Is there inventory? How difficult is it for your customers to find the new set of products they suddenly want? Are your personalization tools still working?For example, are you sending out marketing messages that might be uninteresting or worse, embarrassing given the current state of the world? Or are your on-site recommendations still showing products that are suddenly irrelevant?
2. Offer products that reflect the changing preferences of your customers
   Your models that use data from six months ago to predict customer preferences are probably not accurate anymore. Instead, train these models on customer preferences from the past two to four weeks. Even though you might not have much sales data, it may give you a more accurate depiction of current customer needs, as the whole world was different six weeks ago. Also, there may be price sensitivity-related changes, so be sure you’re accounting for that too before you offer more expensive items.You’ll have to experiment with the amount of training data you use, but doing so will pay off as it’ll allow you to find the sweet spot where you include enough new trends while omitting old, irrelevant data.
3. Make sure you’re not saying the wrong things
   Try new messaging strategies. Old messaging is dead. No one is purchasing or behaving the same today. This means your messages need to change to reflect the new concerns, hopes, and desires of your customers.One way to do this is by writing several options for new marketing copy and A/B testing them to see what resonates.If you have previously identified user segments, they may be the same, but even these may have changed. Trust nothing and prove everything with new evidence. The way to do that is by asking your customers or trying different approaches and quantifying the results. If you have an A/B system in place, use it now! What was working before has likely changed.
4. Assess your marketing targets
   As mentioned above, take another look at your customer segments. Are they the same or, more likely, have several segments merged into one or two new segments? Knowing this will help you tailor messages in these changing times.I’ve always been a big fan of reaching out to customers and asking them how their preferences may have changed. If the incentive is big enough (in other words, if they presume they’ll see wonderfully personalized offers, just for them), they’ll likely be willing to spend time helping you serve them better. Look to StitchFix and TrunkClub onboarding for a great example of how companies can ask for lots of user data with the promise of a highly personalized user experience.

If you do these four things, your company will be far out in front of its competitors that may not quickly adapt to change.

 

What others are saying:

 

"Companies need to be even more mindful of things like data freshness, distribution shift, and prediction calibration when running machine learning models in production. With fast-shifting user behavior due to the effects of COVID-19, models are likely to rot faster and make bad predictions on unexpected user behavior." – Eugen Hotaj

 

“[Recommender algorithms that consider] the time of the year, don’t make sense at the moment. For example, the festival season is coming soon; normally, it would be reasonable to stock up on tents and recommend them to the youth. It would be unfortunate to do that now. Weights assigned to past actions must be reduced in a way that recommendations would rely mainly on trends of the last few weeks. As a buyer, what I see now that in some cases, recommender systems are partly shut down.” – Dániel Barta

 

“As much as a model can predict specifics, it is important, in my opinion, to look at the underlying reasons.” – Joseph "Goose" Aranez

 

Bio: Zank Bennett is the CEO and founder of Bennett Data Science.

Related:

• 5 Ways Data Scientists Can Help Respond to COVID-19 and 5 Actions to Avoid
• Why are Machine Learning Projects so Hard to Manage?
• Successful Use Cases of Artificial Intelligence for Businesses