The Machine Learning Problem of The Next Decade
How can businesses integrate imperfect machine-learning algorithms into their workflow?
Driving toward accuracy
The obvious way to use an 80% accurate algorithm is to use it in 80% of cases where it's correct. This is easier said than done. A self-driving car that can read the road with 100% accuracy would basically mean the end of human drivers, parking lots and traffic jams, and could fundamentally change the ways cities work. But what do we do with a self-driving car with 99% accuracy? Without careful deployment, that car could kill its passengers 1% of the time. And we are going to have 99% accurate self-driving cars long before we have 100% accurate driving algorithms.
How accurate are self-driving cars today?
Google recently released data on its self-driving cars, tracking the number of miles between "disengagements," or situations where a human had to take over. Google errs on the side of caution and reports that only around 15% of the disengagements would have resulted in "contact" -- or an accident -- if control hadn't been handed over to a human driver. But that’s still on the order of 30,000 miles between potential crashes, while human drivers go on the order of 1 million miles between potential crashes and 100 million miles between fatal crashes.
Will Google and others blow through the gap between automated drivers and human drivers, or will they get stuck on a fixed accuracy level, as happened in CrowdFlower's Kaggle competition?
Google's cars will be useful either way, because even an imperfect machine-learning algorithm can usually give you a very good assessment of its accuracy on any particular prediction.
Google's cars are fantastic at knowing that something funny is going on and handing the control back to a driver. If they couldn't do this, they would still be too unsafe to ever operate.
This doesn't just go for cars: Financial algorithms that priced mortgage-backed securities operated with a very bad estimate of their own certainty, and this caused the subprime mortgage crisis. Even ATMs that automatically read the checks you deposit will still hand over funny-looking or suspicious checks to a human for a second opinion.
The design of the handoff to a human is critically important to making this system work. A good model needs to express how uncertain it is and why it has uncertainty. Human operators need to learn to interpret the information and use it to make themselves more efficient.
Good interfaces depend on the application. Medical diagnosis systems work best in the real world when they guide a doctor toward information that will help them make an informed decision. Fraud detection algorithms need to give humans good visualizations to understand what's important in massive data sets.
With Microsoft's Azure ML and IBM's investment in Watson, making models is easier than ever. Companies no longer need a Google-size R&D budget to make machine learning applicable to their business. These models aren't perfect, but they're useful. The new challenge for businesses is how to integrate an imperfect machine-learning algorithm into their existing workflow.
Original Post
Bio: Lukas Biewald is the co-founder and CEO of CrowdFlower. As a former data scientist, Lukas was frustrated by the amount of time he had to spend cleaning and labeling data instead of actually using it to solve business problems, leading him to co-found CrowdFlower in 2009. Today, the CrowdFlower platform connects over 5 million data enrichers in almost every country who work around the clock to provide companies with clean and actionable data.
Related:
