The Machine Learning Problem of The Next Decade

How can businesses integrate imperfect machine-learning algorithms into their workflow?

By Lucas Biewald, CrowdFlower

machine-learning-problemA few months ago, my company, CrowdFlower, ran a machine learning competition on Kaggle. It perfectly highlighted the biggest opportunity (and challenge) with machine learning: What do you do with an 80% accurate algorithm?

We uploaded data collected on our platform and Kaggle sent it out to over 1,000 data scientists, who competed to see who could build the best search model.

The simplest approach gave a baseline accuracy of 32%. Within hours a team beat that with a 35% accurate model. By the next morning, one team already had a 53% accurate model.
Extrapolating the first four days to our 60-day contest, you might expect the winning accuracy to get close to 100%.

But in fact, this is what happened:
The winning entry -- submitted by Chenglong Chen -- was just 6% more accurate than the best model submitted a week into the contest.

And it wasn’t for a lack of trying! As the Kaggle competition went on, more and more teams entered and existing teams refined and resubmitted their entries:
Given that over 1,000 smart data scientists worked on this task, it's fair to say that 71% accuracy on this task is very close to the best possible accuracy with today's technology.

What does this mean for the future of machine learning?

These results are familiar to anyone who has ever worked on an A.I. project. For the first couple of weeks performance improves steadily, and then you hit a wall. Maybe you have a breakthrough or two, but there's no way to put a plan or a process around breakthroughs.

Every engineering project has delays and issues, but machine-learning projects are harder to manage than any other. In the first week you might go from zero to 80% accuracy. The next 20% might take you another week, a month or a lifetime -- it's impossible to tell.

How do you make an 80% accurate model useful? Until we're replaced by robots, this is going to be the machine learning challenge of the next decade.

Continued on next page ...