Interview: Amy Gershkoff, Director of Customer Analytics & Insights, eBay on How to Design Custom In-House BI Tools

Dr. Amy Gershkoff is Director of Customer Analytics & Insights at eBay. Prior to joining eBay, she was the Chief Data Scientist for WPP, Data Alliance, where she worked across WPP’s more than 350 operating companies to create integrated data solutions. Previously, she served as Director of Media Planning at Obama for America, where she was the architect of Obama’s advertising strategy. Gershkoff also co-founded Changing Targets Media, for which she was named one of the nation’s “40 under 40” leading entrepreneurs. She has been featured in the Washington Post as one of the nation’s most prominent innovators and one of the Top 50 Women to Watch in Tech. She has been a commentator on NPR, Bloomberg, ABC, and for various print media outlets and holds a Ph.D. from Princeton University.

Here is my interview with her:

Anmol Rajpurohit: Q1. While sharing your experience from the Obama Campaign, you mentioned that "best-in-class BI tool is key". What was your approach towards building such a BI tool? Which commercial or open-source tools did you use?

Amy Gershkoff: I have seven core principles that I use when I design business intelligence tools:

1. Leverage data from multiple sources and departments
2. Reorganize data around key business questions
3. Simplify the data to display only the most relevant metrics
4. Visualize the data to make trends easy to see
5. Benchmark results against key performance indicators
6. Enable users from across the organization to draw insights from the data
7. Create a platform that facilitates integrated measurement

 
Generally, I find that off-the-shelf business intelligence tools do not meet the needs of clients who want to derive custom insights from their data. Therefore, for medium-to-large organizations with access to strong technical talent, I usually recommend building custom, in-house solutions. On the campaign, I had a great product team as part of my organization, so we built a custom, in-house BI tool.

AR: Q2. Did you face situations where Analytics did provide some useful information, but not the complete information, for example, good correlation but no reliable results for causation? In such cases, what approach did you took? What was your experience with A/B testing?

AG:

Anytime you’re measuring results, it’s critical to understand what you’re not capturing, that is, what are the unobserved variables or biases in your analysis.

For example, on the Obama campaign, at one point we saw a significant lift in polling numbers from a certain part of a particular state. We noticed the polling numbers spiked right after we had significantly increased our advertising spend there, so most people assumed the polling lift was caused by the advertising increase. However, I noticed that there were many other parts of that same state where we had also increased our advertising spend and had not seen similar lift in poll numbers. Upon closer inspection, we could see that the President had visited that part of the state quite recently, where he participated in a number of events including a large and well-publicized rally, and hadn’t visited anywhere else in the state. Thus, it appeared the lift in the polling numbers was caused at least in some significant part by the President’s visit rather than the advertising.

A/B testing often gives people a false sense of security: if the test is set up correctly, people assume any observed difference in outcome can be attributed to a difference in treatment between group A and group B. However, the key part of the statement above is “if the test is set up correctly,” meaning, if the samples selected for treatment A and treatment B are balanced.

Many people simply pull two random samples, assume they will be balanced, and run the A/B test. A much more robust approach is to employ sample balancing checks to ensure the samples have comparable means on key variables that predict the outcome. This ensures that you can actually attribute any differences in observed outcomes to differences in treatment.

AR: Q3. In the current Big Data landscape, what factors do you think will help differentiate the future Analytics leaders?

AG: The most successful analytics professionals combine strong technical acumen with deep business strategy skills. Unfortunately, such individuals have become increasingly rare, partially as a consequence of how the academy is organized: data scientists and statisticians receive strong technical training but little training on business strategy. By contrast, MBAs may take a course on “Big Data,” but rarely receive deep technical training in data science. The most successful analysts have strong business strategy skills that they can use as a lens to view technical problems, but these individuals are few and far between.

If you can’t find the rare person with both strong business acumen and strong data science skills, one strategy I have used is to pair MBAs and data scientists up to problem solve together. This has allowed the data scientist to benefit from the business strategist’s thinking, while also enabling the strategist to benefit from the technical firepower of the data scientist. This has proven to be a great way to ensure that cutting-edge technical ideas are infused with business thinking.

AR: Q4. Which of the current trends in Big Data arena are of great interest to you?

AG: For virtually all companies, the focus should not be on Big Data: it should be on Smart Analytics. Most companies focus on frenetically collecting as much data as possible, including often paying large sums for third party data, when their focus should first and foremost be on making the best possible use of the data they already have.

In my previous role as a consultant, I saw many companies who had failed to connect datasets across departments even within the same company. For instance, one client had the data about their website’s performance siloed in their IT department, the data about their earned social media in their communications department, and their data on their advertising in their marketing department. Obviously, these three datasets are very much related, and when I helped this client “connect the dots,” they were able to mine the website traffic data to uncover significant insights about the performance of their marketing and communications campaigns that could improve both. Many companies can make significant gains by simply joining internal datasets and mining the resulting data for insights.

AR: Q5. What key qualities do you look for when interviewing for Data Science related positions on your team?

AG: When interviewing data scientists, I look for 5 key characteristics:

1. Know what you don’t know: The data science field has exploded in recent years; today, no one could possibly be an expert on every facet of the field. Thus, I look for people who know their expertise and also know the limits of their expertise, who know when they would need to do further research or bring in another expert to be successful versus when their own knowledge is sufficient.
2. Intellectual curiosity: Because data science is such a vast field, and no one will be an expert in all aspects, I look for people who are intellectually curious and eager to learn.
3. Creative thinking: I search for data scientists who can think creatively about how to solve a new problem they have never seen before.
4. Business strategy thinking: For a solution to a data science problem to be optimal, it needs to not only be technically correct but also be actionable from a strategic perspective. For any data science problem, it’s critical to ensure that the solutions have business thinking deeply baked into the fabric of the analysis and that the results make sense from a strategic perspective.
5. Collaborative character: To scope, analyze, solve, scale, and implement a data science solution can require in some cases a dozen people, including data scientists, engineers, statisticians, strategists, product owners, marketing, and often many others; thus, a collaborative nature is critical for success.

 
AR: Q6. On a personal note, are there any good books that you have been reading lately, and would like to recommend?

AG: Lately I’ve been re-reading some of the Dale Carnegie leadership books such as Leader to Leader and Leadership Mastery. I first read these almost a decade ago, but I find that you get something new out of these books at every stage of your career.

Related:

• Interview with Rayid Ghani, Chief Scientist Obama 2012 Campaign
• The First Law of Data Science: Do Umbrellas Cause Rain?
• Interview: George Corugedo, CTO, RedPoint on Big Data Trends and Important Skills
• KDnuggets Exclusive: Interview with Rayid Ghani, Chief Scientist Obama 2012 Campaign