Features

Gregory Piatetsky-Shapiro, Q1: What are the interesting challenges in your current job at Microsoft?

Ronny Kohavi

Kohavi: To place my answer into the right context, I formed a team here that has built an Experimentation Platform, which allows multiple product teams to experiment with ideas by running controlled experiments, or A/B tests.

There are multiple challenges, which I'll split into technical and cultural.

The technical challenges are the ones everyone expects: write a highly-available system that scales, is easy to use, satisfies the user requirements, consists of high-quality code that is easy to maintain, etc. These are by no means easy to meet, but they're better understood and we can plan for them. The devil is in the details in many cases. For example, randomization of users in controlled experiments is not as trivial as people think. Most people don't think of the monotonic ramp-up property, which we described in our recent KDD conference paper (http://exp-platform.com/hippo.aspx).

The cultural challenges, or adaptive challenges as Linsky and Heifetz refer to them in their great book Leadership on the Line (http://www.amazon.com/dp/1578514371) are much harder. Microsoft has a history of shipping shrink-wrapped products (e.g., Office), where great processes have been defined and tuned over the years. However, as more interactions with end-users are happening on the web through services, there is a great opportunity to take advantage of the ability to learn from user behavior.

Even though data-driven decision-making seems like a no-brainer in theory, in practice it is hard for organizations to adopt to a new way of doing things. Upton Sinclair's quotation comes to mind here:

"It is difficult to get a man to understand something when his salary depends upon his not understanding it,"

and we see it multiple times. Some people believe that experimentation threatens their job as decision makers. Program Managers at Microsoft are used to defining the next set of features to develop. Proposing several alternatives and admitting you don't know which is best is hard. Editors and designers-they get paid to select a great design. We know from experience that many ideas fail, and failures may hurt image and professional standing.

Changing the culture from plan well and execute well to do it wrong quickly, and iterate is not easy. A great book encouraging the latter approach is Mike Moran's book (http://www.amazon.com/dp/0132255960).

To help with the cultural challenge, we created physical stress HiPPOs that say "Listen to the Customer." The HiPPO itself stands for the Highest Paid Person's Opinion, and we distributed hundreds of them across Microsoft so that people can squeeze them when decisions are made without HiPPOs. To the HiPPOs themselves, we say: when someone comes to you with a question about whether to use blue or red for the color, don't fall into the trap of sharing your personal opinion. Say: "Experiment. Try both!"

Here is more information about the challenges (PDF).

Interview continues here ...