Prev
From:
Businesses will learn harsh but valuable truths if they subject new ideas to controlled experiments, says Microsoft's Ronny Kohavi.
MIT Technology Review, February 22, 2011, By Erica Naone
Pick your page: Is the top or bottom version of this home page for a previous version of Microsoft Office more likely to produce sales? The bottom version might seem appealingly uncluttered. But when subjected to tests with actual Web users, it performed 64 percent worse than the other. Credit: Microsoft
You know that great idea you have for improving your business? Ronny Kohavi, an architect at Microsoft's Online Services division, says there's good reason to suspect it's actually lousy. Studies of the software industry indicate that when ideas people thought would succeed are evaluated through controlled experiments, less than 50 percent actually work out.
"This is very humbling," Kohavi says. Still, he observes, it's hard to convince people to use data to evaluate an idea rather than relying on their intuition.
Kohavi's been a true believer for years, but most businesses aren't using these principles. He says that when he started at Microsoft in 2005 and founded an internal experimentation platform, he couldn't find any internal groups that were doing controlled testing before launching changes to Microsoft's websites.
Kohavi knew the value of testing from his time as the director of data mining and personalization at Amazon, where his team had evaluated an idea called behavior-based search. The idea: when users searched for keywords, they would be presented not only with results that matched the keywords but also with items that other people actually bought after searching for those words. The search team didn't like the idea, because it would mean recommending products that seemed unrelated to what the user had searched for.
Read more.