Data Science is Boring (Part 2)

Why I love boring ML problems and how I think about them.



By Ian Xiao, Engagement Lead at Dessa

Figure

 

TLDR: I love boring Machine Learning (ML) problems. They are core to businesses. In this post, I argue why boring problems are good and share two simple tools: ML Archetypes and MJIT. They can help Data Scientists and ML Product Managers to identify meaningful problems, generate effective solutions faster, and articulate powerful messages.

Thank you! Part 1 of this series has over 3.4K claps (still counting). It reached the top 1% on Medium in less than 4 days. I am extremely humbled and really appreciate your support. As always, hope you enjoy this post.

Like What You Read? Follow me on MediumLinkedIn, or Twitter. Also, do you want to learn business thinking and communication skills as a Data Scientist? Check out my “Influence with Machine Learning” guide.

Disclaimer: This post is not endorsed by any of the firms I work for. I use the term “Data Science” and “ML” interchangeably.

 

1. Story Time

 
Episode 1 in “Data Science is Boring”: on a road trip to Shawn’s school (my younger cousin), I shared how I cope with the boredom of deploying ML.

I did a fantastic job in motivating him, so Shawn asked: “What problems do you solve with ML? Real problems, not toy examples I see on the internet.”

“Very good. First of all, you need to recognize something: the reality of real problems is that they are pretty boring. But, I love boring problems! If you want to take the data science career seriously, you should too … ” I said, and the story continues.

 

2. Let’s Get into It

 
In short, I argue that a) boring problems are good and b) we should apply ML to solve more boring problems, but innovatively.

Why are boring problems good? Boring problems are good because they represent steady-state operational issues. These operations drive the core of businesses. The core of the business creates consistent and substantive value. Therefore, businesses prioritize investment to solve boring problems that hinder their cores. Boring problems get real and constant attention.

So what? As a professional Data Scientist or ML Product Manager, it’s easier to convince your boss (clients or customers) to buy into your ML ideas if you can recognize and articulate a) what the real problems are and b) if you can justify why and how ML can solve it better. Many people have likely worked on the same problems given the importance of the problems. The ML Archetypes and MJIT can help differentiate your ideas, please keep reading.

As an aspiring data scientist, it means you can show the hiring managers your abilities to be practical, that you have an intuition of what it takes to solve real-world problems, and most importantly, that you have the right attitude (e.g. you won’t get “bored” after 3 months).

What this post is not about? Businesses work on “innovation projects” with “Zero-to-One type solutions” for many reasons. I love thinking about and solving them too. But, statistically speaking, I don’t have the privilege to work on such projects every day. Message me if you got a Zero-to-One idea and need help. You can find me on MediumLinkedIn, or Twitter.

So, what are the real (boring) problems? And how do people solve them with ML today?

2.1 The ML Archetypes: Understand the History

Many ML solutions we see today are not new despite the recent hype (it will be ignorant to think so). So, it is important to understand the timeless fundamentals — they are true in the past, now, and likely so in the future.

The ML Archetype focuses on the fundamentals by asking:

  • What underlying problem does each ML archetype solve?
  • What are some concrete examples of applications regardless of business functions (e.g. marketing, operations, finance, etc.)?
Figure

Author’s Analysis

 

Like any catch-all frameworks, something is likely to be missing. Please leave a comment below and I will update if needed.

You may ask: how about some cutting edge stuff like “Distributed Reinforcement Learning with Knowledge Transfer Capability” (my own work. Wait, it seems like a toy example) or “Face generation using Generative Adversarial Network”? I think they fall into the Prioritization and Patterning archetypes respectively. These examples are just “techniques” of how to solve the problems better or differently; they are not the problems. The key here is to focus on the underlying problems and types of outputs.

Let’s take this one step further. Each archetype can be a standalone solution. More importantly, each ML archetype can be a building block of a larger, more sophisticated, and “cool” ML application.

To illustrate, here is an example of my favorite iPhone feature — the “For You” personalized photo album. I break down the feature into the archetypes and more granular implementation details. It is over-simplifying (UX, data, and systems are all important), but I hope you can see how the ML archetypes work together and build up to a larger ML application.

Figure

Link to the iPhone screenshot; author’s analysis Caveat: This is my own analysis and for illustration purposes. It may not represent how Apple engineers actually implemented it.

 

So, how can you use it? As an aspiring data scientists, the ML Archetype serves as a “map”. You can use it to derive a study plan, choose the ones that interest you, and dive deeper. If you want to go above and beyond, maybe you can 1) pick a favorite archetype, set up some Open Source project on Github, and apply simple and sophisticated approaches, and 2) combine multiple archetypes in creative ways to do something cool.

As professionals and ML Product Managers in enterprises, you can use this as a “canvas” to layout your ML portfolio (e.g. remove the “Sample Solutions” and replace them with your actual use cases and their performances). In addition, use the “canvas” to explore different possibilities. As a result, you can identify the gaps and opportunities. And this leads to our next topic.

2.2 The MJIT: Finding the Right Edge for Incremental Innovation

Now you get a sense of what has been and can be done, here is a more important question: what is the next immediate opportunity? Many data scientists (or engineers, including myself) immediately think of the craziest, boldest, and complicated ideas.

This is not a bad thing. It just shows an intrinsic drive to push the boundary. However, as I mentioned in my last post, many crazy ideas tend to be over-complicated, so they don’t justify the effort given the incremental return.

So, finding the Minimum Justifiable Improvement (MJI, I call it “M-gee”) is critical. I do so by traversing the following space.

Figure

Link to the original image

 

Okay, just kidding. My brains cannot handle this. So, I developed and use a simpler mental model called the MJI Tree (MJIT). The MJIT is a structured and efficient problem-solving and solution generation tool to address three fundamental questions: 1) why are we doing this? 2) How can we do it differently? 3) What exactly can be done?

Figure

 

Speaking of incremental innovation, the MJIT is built on battle-tested tools including hypothesis-driven problem-solving used in Management Consulting, the Minimum Viable Product (MVP) concept from Software Development, and Tree Search algorithms in Computer Science. The Why-How-What approach is by Simon Sinek. I only take credit for combining and re-purposing them for ML.

Important Note: I only included examples that are most relevant to ML problems. Data Scientists should have a broader view of what other non-ML solutions there are. ML is useful, but not a silver bullet.

What do I do exactly?

A) traverse the tree from top to bottom and by the respective index. I write down all the viable options that are directly relevant to objective(s). I estimate the effort for each option. The effort varies case by case, so apply your best judgment or do it with a team if possible (it helps to de-bias and de-risk).

B) I shortlist the top ideas; I log the ideas that would take too much time.

C) I do a back-of-envelop estimation of the total improvement of my shortlist. It’s often difficult to know until I actually do it; it depends on a combination of experience and gut-feeling.

This is probably the most critical step in determining a project’s success. I may share a follow-up post with a few examples of how I approach this. Please follow me on MediumLinkedIn, or Twitter to stay tuned.

Because of how the problem is framed and this tree is structured, as you progress, the options are typically ranked by expected incremental improvement per unit effort. That said, the first few options are your MJIs. Each MJI can be a small task or a project that may require substantive funding and months of work. Again, this is situational, apply your judgment.

What’s next (again)? Assume you execute the MJIs properly and they are still not good enough, every time you traverse this tree, the options will inevitably get “more innovative, but expensive” because you should have crossed out “the boring, but immediately effective” ones. Traverse, think, execute, repeat until no option is left or your minds explode (or gas out)!

Want to give it a shot? As a thought experiment, why don’t you apply the MJIT to your current ML project, try to generate 10 ideas of how to improve it assume you have 4 weeks, and write them on a note pad in sequence?

If you do not have an ML project, you can try using the iPhone “For Me” example from above. Assume you are an ML engineer at Apple, you are tasked to improve the feature to increase “# of photos shared with your families” by 10%, you have 4 weeks to come up with a prototype, then you need to go present to Tim Cook (assuming he cares).

(Please do not say “quite Apple, join Google to work on Android’s Photo app, get inspired, and come back to Apple with a salary raise”. It’s smart, but that’s cheating in this context.)

To be concrete, here are two examples of how my outputs usually look like:

  • Data idea: develop a customer level 3-month rolling average of spending in dollars using table X, Y, and Z, join on column A and B, and between month 1 and 12.
  • Model idea: replace the XGBoost Model with a LightGBM with the same target and features; benchmark performance by F1 Score while search over max tree depth between 1–10.

This level of details can help you size the effort and prioritizing MJIs better. Once you get familiar with the process, you should develop your own MJIT for your domain problems and priorities. You may have different indexing, and it is fine. The MJIT is not meant to be a precise scientific algorithm.

Wondering what techniques are there? There are lots of good online materials that highlight the spectrums of techniques by complexity and adoption. Here is an example (only applied to modeling though).

Figure

Link to the article

 

 

3. To Sum Up

 
Boring problems are great. They make up the core of businesses and signify real operational needs. We should solve more boring problems because your work will translate into immediate, tangible, and sustainable values. From a tactical standpoint, it is important to 1) recognize what fundamental problems there are and corresponding ML solution(s), 2) how ML archetypes can make up powerful ML applications, 3) identify the next best opportunities to deliver immediate value and gain more traction. I hope the ML Archetype and MJIT are helpful.

As always, feedback is welcome. Please leave a comment below.

Like What You Read? Follow me on MediumLinkedIn, or Twitter. Also, do you want to learn business thinking and communication skills as a Data Scientist? Check out my “Influence with Machine Learning” guide.

You may also like my other writings:

Data Science is Boring
My normal (boring) days in doing Data Science and how I cope with it
 

The Last-Mile Problem of AI
One Thing Many Data Scientists Don’t Think Enough About
 

AI-Based Operation: Learnings for Business and Technical Managers
Using Reinforcement Learning to Tackle CitiBike Rebalancing Problems and Beyond
 

Until next time.

Ian Xiao

 
Bio: Ian Xiao is Engagement Lead at Dessa, deploying machine learning at enterprises. He leads business and technical teams to deploy Machine Learning solutions and improve Marketing & Sales for the F100 enterprises.

Original. Reposted with permission.

Related: