Interview: Daqing Zhao, on Building Effective Data Models for Marketing

We discuss the challenges in identifying the fair price of ad media, recommendations for building effective models for online marketing, unique challenges of Mobile channel, selection of Big Data tools, and more.

Daqing ZhaoDaqing Zhao has over 20 years of experience in analyzing and taking actions on very large data. Trained in data analysis and simulations on molecular systems, he gained extensive expertise in customer centric marketing, optimizing for all stages of customer acquisition, conversion and retention. He has worked on segmentation and predictive modeling for banner ads, web logs, search keywords, emails, transactions, call center, and customer life time values.

Daqing is Director of Advanced Analytics at, leading the predictive analytics, test and experimentation and data science teams. He previously held senior management and technical leadership positions at, the University of Phoenix, Tribal Fusion, Yahoo, Digital Impact, and Bank of America. He also worked on client analytics projects for Intel, HP, Wells Fargo Bank, SBC, Dell, T-Mobile, MSN Search and Travel, Intrawest, PayPal,, MasterCard and others.

Daqing received his Ph.D. from Stanford University, specialized in scientific data processing, simulations and optimizations.

First part of interview.

Here is second and last part of my interview with him:

Anmol Rajpurohit: Q6. Why is it so hard to accurately identify the fair price of advertising media in order to maximize return-on-investment? Why is this so important for online retailers?

Daqing Zhao: The reasons can be many things, such as data availability and scarcity and scalability of data analytics on millions of keywords, billions of ad pages and large numbers of customers. The data we get from media vendors are limited. We need time to figure out how to use that data. The market is changing all the time, due to advertiser behaviors as well as consumer behaviors.SEM

Let me give a simplified example. If we average say two ad words in SEM (Search Engine Marketing). Let's say keyword A has a value per click of $5 and keyword B at $10. The average of the two is $7.5. As a group of the two, in order to be profitable, we cannot bid over $7.5. What happens if we bid just below $7.5 on both keywords? We lose traffic on keyword B and we lose profit on keyword A. I call this a double loss, on both traffic and profit.

In addition, in an auction market, say there are two advertisers competing for these two keywords. Company 1 knows the values of the clicks of keyword A and B separately, and Company 2 does not. What happens? Company 1 will bid lower on A and get less bad traffic and bid higher on keyword B, to get more the better traffic. Company 2 on the other hand, not knowing the values in detail, will bid the same for both. Because the good traffic from keyword B has gone to the higher bidder, Company 2 will get more unprofitable traffic from A. So a company with better analytics can cherry pick in the market, putting other companies without good analytics at a severe disadvantage. This is especially the case if profit margin is very low.

AR: Q7. Your chapter "Frontiers of Big Data Business Analytics: Patterns and Cases in Online Marketing" in the book "Big Data and Business Analytics" provides great insights through interesting case studies. What are your top recommendations for building effective models for online marketing?

DZ: Big Data & Business AnalyticsThe most important thing is to make sure the model is relevant to the business objectives. We need to make sure that the insights and results can benefit the customers and the business in some way. We can build models for a lot of things, but many of them may not be relevant or actionable.

I think also that it is important to assess what data can and cannot be collected, and also the cost and benefits of collecting each type of data. If we can find the smoking gun, predictive modeling algorithms will not be very important.

One may have many ways to do clustering, but if I add or remove some dimensions, the results may have much larger impact than various algorithms may produce.

From my experience, as we analyze the data more and more, we continually find data issues. Every wrong day can be wrong in its own way. So in order to ensure data quality, we cannot just Data-Qualitycollect data or only see some reports such as dashboards. Some data issues may not be apparent in those simple reports. For example, in search engine marketing, we have a very long tail search keyword set with very sparse traffic counts data which can be very volatile in number. If we miss the capture of 10 percent of the conversion counts, it may not be obvious from simple reports. But if we use that information to bid in the media auction market, our ROI would be wrong. If we didn’t model and take action on the data, we might never have found the data issue. The more we analyze and use the data, the cleaner we data will be.

We need efficient model building environment, where analysts can build, refine, deploy, compare, and analyze the results of many models easily. In traditional statistics, data sets are small in size and high in value, and an analyst can afford to spend a long to research on a model. Now data are cheap and models may degrade in weeks or shorter, and the market is changing all the time, we need to be able to scale our models.

It takes time for an organization to get used to the data driven decision process. It takes a long time to cumulate creative assets, and train business decision makers to digest data insights and act on them. A misconception is that data analysis makes online making easier. The truth is that often it is harder for the marketing managers, because now there is visibility on performance, and the insights can be constraining on media spend decisions. Spending money wisely is much harder than spending money.

AR: Q8. Do you see any significant difference in customer behavior across Web and Mobile channel? Are there any Analytics challenges unique to the Mobile channel?

DZ: Yes. Mobile customers are different from desktop users. There are demographic factors, in adoption and usage of Mobile Channelsmart phones. There is also this decentralization of human interaction to the Internet. We used to have desktop as the only device to get online. Now we have not only desktops, but also tablets and smart phones, often not shared as much as desktops. We each may have multiple, many types of devices, each used in a different context. Our interactions with the Internet overall are higher, but for each device, we may exhibit less activities. It is very important to understand customer behavior across the devices. Any one device is just part of the puzzle. Here, we have challenges not only for attributions but also for behaviors.

Another challenge for smart phone is about locations. It is hard not only to collect data on location and context but also to analyze and refine the targeting algorithms. When we get data from a partner, we still need to understand what we get, what the match rates mean, etc. I see most efforts are around simple rules like if the customer is near where and then do what. This type of targeting is not scalable and less effective.

AR: Q9. What are the Big Data tools you commonly use in your organization? How do you select an appropriate Analytics tool, given so many choices in the market?

DZ: We use the principle of separation of concerns. It is far better to let software companies and open source communities build and refine the lower level data science infrastructure. We definitely have to own the design of the solutions specific to our data, our customers, and our products and businesses. We may work with consultants on some areas in the middle. We use commercial and open source tools, and we also have internal data technical teams as well as analytics teams. There are some types of work, due to for example, privacy protection and business knowledge, we have to do internally and cumulate expertise in house.

We have Hadoop, and Hbase, and are in efforts to test solutions using Spark and H2O. We also have SAS, and use R, Mahout, SAP/KXEN, and Tableau for visualization, etc. We also work with research groups at SAS, IBM and others on some solutions. We evaluate products to suit our solution needs and everything is dynamic. In the end, we optimize the overall benefits and cost of ownership.

AR: Q10. What is your favorite interview question when hiring a data scientist or data analyst for your team? For a data scientist role, what skills would you categorize as "must-have" and which ones as "good-to-have" ?

DZ: We don't have specific questions for interviews. I like asking description of a project the candidate worked on recently and from there to see his or her technical and business skills. In addition to the “must-haves” such as basic math Skillsskills, computer skills, and SQL skills, and the ability to communicate ideas and insights, we like candidates who have handled and not intimidated by large data sets and complex analysis strategies. We also like candidates who enjoy learning new things and can think outside of the box. Continuous lifelong learning is not easy, but there are people who want to learn new things and are eager to face new challenges. This is a very important trait working in a new, fast changing field.

In hiring talents in data science, often the technical side is not the most difficult part, but it is the common sense that is hard if possible to train. Many may be good at numbers, but the really good ones see the patterns and solutions behind the numbers.

walter-isaacson-steve-jobs AR: Q11. On a personal note, what book did you read recently and would strongly recommend?

DZ: One book I read a while ago was Walter Isaacson's book on Steve Jobs, but the book keeps coming back to my mind. I am amazed at the incredible inventor, his fascinating way of thinking and an interesting personality. Jobs has qualities that differentiate from many of the pure technical people and pure business people, and the book offers interesting insights about the dynamics of people with different backgrounds. There are a lot of things for us to learn, being in a new, ever changing field, combining technical innovations and impact on the social side.