Publications

By Mike Moran, 01/29/2008, WebProNews

...

I was contacted recently by Ronny Kohavi, Microsoft's General Manager of Experimentation Platform, who uses the market research term "controlled experiments" for what they do. Ronny came to Microsoft from Amazon, where he served as Director of Data Mining and Personalization, and he's written four of the top papers on machine learning according to CiteSeer. He was gracious enough to answer several questions for you to explain how experimental marketing works in real life.

Me: What was the purpose of the solitaire

vs. poker test

for the Game Downloads page in Windows Marketplace ?

RK: One of the simplest ways to improve clickthrough rates is to change images. It's usually trivial to change images, and when someone is designing an image, they typically have several alternatives. Instead of betting on one, try the top three (and encourage diversity of ideas). What is surprising to people about this example is that many predict the winning image incorrectly and, more annoyingly to some, that the difference is so large (61%). It's very common for the technical audience to prefer the Poker image and confuse themselves with the target audience.

Me: Can you could explain the advantages of controlled experiments?

RK: When I was director of data mining and personalization at Amazon, the two most successful innovations by my team were not on any road map the year before, and were initially ranked so low by myself and the team that one was given as a ramp-up project to a new employee, and the other given to an intern. The projects generated hundreds of millions of dollars in incremental revenue. Such an observation is very humbling and highlights the biggest advantage of controlled experiments: we can try a lot of things quickly, and let users guide us.

Read more.