Metrics Gone Wrong  – How Companies Are Optimizing The Wrong Way

comments

By Drew Moxon, NCSOFT.

Peter Drucker famously said “what gets measured gets managed.” This mantra has been embraced by companies worldwide and particularly in Silicon Valley. Firms are trying to optimize their their sales calls, their supply chain, their HR department and anything else they can put a number on. But with the rush to measure everything, many companies don’t think about what the path to optimization looks like and whether their metrics match their goals.

One recent example is Twitter’s new feed algorithm which was supposed to surface the most relevant tweets rather than use the standard timeline. It was released to complaints and suspicion of censorship.

Seriously, though: I'm seeing a lot of women who say they've been shadow banned. Women w/strong views who are likely targets for harassment.

— Ninja Economics (@NinjaEconomics) March 30, 2016

Every optimization algorithm has a metric that it’s trying to maximize or minimize. This can be a score, an average or some combination of factors. But what if you choose the wrong metric? In the example above, what if the algorithm prioritizes a tweet in your feed by the number of retweets it receives? Harassing tweets may get more retweets due to people quote retweeting to denounce the harasser. Responses to harassing tweets may get more likes in support, but fewer retweets. That means the algorithm would prioritize the harasser’s tweets over the victim’s defense. This is a simplification of how Twitter’s algorithm likely works, but it shows how optimizing for the wrong metric can lead to undesirable outcomes.

The Twitter example is a case of optimizing for the wrong metric. But what about optimizing for the “right” metric but taking the wrong path to get there? For example, let’s say the goal for an autonomous car is to have no more than a .00001% chance of getting in an accident on any given trip. This is a laudable goal. One path to that goal is to take routes that are lightly traveled, causing travel time to greatly increase. Without adding the right constraints (travel time or fuel consumption), the algorithm gives an optimal solution with negative side effects. The algorithm must be controlled by implementing the right constraints.

The Facebook news feed is potentially an example of both of these issues. Despite research from their team that says otherwise, many people suspect that the feed algorithm exacerbates political partisanship due to only showing stories that agree with members’ political leanings. Facebook is probably optimizing for a number of metrics including likelihood to engage with an article. From a business perspective this is reasonable, but from a social perspective it might not be. Additionally, if the path to higher article engagement means lower engagement with people of opposing viewpoints, then the road to maximization might have additional negative side effects.

With the rush to hire data scientists to optimize every color on your webpage there have been a few things lost in the process. Metric design is something that very few people (data enthusiasts or not) specialize in. It’s hard and it’s not always the most rewarding task of analytics. You’re not creating the next sexy data product or discovering a new deep learning method. But metrics are how businesses operate day-to-day and if you’re using the wrong one it can be devastating for the company. More product owners, analysts, data scientists and execs should be thinking about how their products should be measured and where their metrics will lead them.

In his farewell post, Kickstarter VP of Data Fred Benenson imparts two pieces of knowledge he learned as a data leader:

Data work consists primarily of a) counting things b) figuring out the denominator and c) the reproducibility of a) and b)

The more you can work with someone to help refine their question the easier it will be to answer

These two points summarize much of the hard work in creating good metrics. The best metrics are simple to explain, reproducible and tightly bound to the goals of the company. Good metrics will serve both as a north star to guide the company to greater heights and a canary in the coal mine to warn of danger. But companies need to hire the right people who know how to build those metrics. Those people tend to be deeply embedded in the data and with the product that they work on. They work closely with their product managers, designers and engineers to understand what makes the product tick.

The next time you hire a data specialist, ask them “how should I measure my business?” Then ask what the advantages and disadvantages are of using their chosen metrics. If they can’t answer those questions, then they probably shouldn’t be the one you count on to make the guide posts for your business.

Bio: Drew Moxon is the lead data scientist and head of analytics at NCSOFT's San Mateo studio. His background is in economics, experimentation, behavioral analytics and metric design.

Original. Reposted with permission.

Related:

• Avoiding Tunnel Vision in Peer Comparisons
• Lift Analysis – A Data Scientist’s Secret Weapon
• 20 Lessons From Building Machine Learning Systems