KDnuggets Exclusive: Part 2 of the interview with Quentin Clark, CVP, Microsoft Data Platform Group

We discuss Microsoft decision to embrace Hadoop as the standard, collaboration with Hortonworks, and advice to newbies in Data Science.

Here is Part 1 of the interview.
Quentin Clark, Microsoft
Quentin recently delivered a keynote at Strata 2014 on “Bringing Big Data to One Billion People”. He mentioned that Microsoft’s vision is to change the world through making Big Data accessible to a billion people. He shared a few user stories demonstrating real-life impact and talked about the endless opportunities Big Data provides for the progress of mankind.

For reference: here are the slides and video of his keynote at Strata 2014

Anmol Rajpurohit: 4. Can you please talk about the decision to embrace Hadoop as the standard and collaboration with Hortonworks? What factors led to the decision to support an open source project, that too in the data platform space where Microsoft already has some great products and services?

Hadoop Logo QC: What’s important about Hadoop is that it is a standard. Just as SQL became a standard so many years ago for relational data, Hadoop became the standard for managing and processing new kinds of data. Microsoft recognizes this, and the decision to invest in it was fairly straightforward. It’s true that as an open source project, we had to create a different approach in how we are investing to ensure our customers get the best that standard has to offer.

Hortonworks Logo The partnership with Hortonworks is a key part of that approach. It has allowed us to participate in the Hadoop community much more effectively than simply taking the code and working on it on our own. We have put in thousands of engineering hours and tens of thousands of lines of code now into the Apache project. This certainly has made Hadoop great for Windows, but more importantly we are contributing to projects like Tez, Stinger and Hive.

That work has allowed Microsoft to offer Windows Azure HDInsight – a 100% Apache Hadoop service in the cloud. And it’s empowered our partner Hortonworks to offer the Hortonworks Data Platform (HDP) – a 100% Apache Hadoop product on Windows and on Linux. Windows Azure HDInsight combines the best of Hadoop open source technology with the security, elasticity and manageability that enterprises require. We have built Windows Azure HDInsight to integrate with Excel and Power BI for Office 365, allowing people to easily connect to data through HDInsight, then refine and do business analytics in a turnkey fashion.

AR: 5. Data Science is currently being considered one of the most attractive careers. What advice would you give to people interested in a long term career in Data Science?

Get InvolvedQC: Get involved. Like becoming a software developer, one of the most important things in skill development for the Data Scientist is simply experience. Tools, languages, systems and the such will always evolve. What is enduring is the techniques and intuitions people build as they get exposed to more challenges and domains.

AR: 6. Which book (or article) did you read recently and liked? What keeps you busy when you are away from work?

How We Decide by Jonah Lehrer QC: I have been reading this book How We Decide by Jonah Lehrer- it’s an interesting view into how individuals approach making choices. What’s been really a lot of fun for me over the last few years is just how much the nature of my job has changed. The addition of cloud products – Power BI in Office 365 and our data platform offerings in Windows Azure – has changed our work substantially. SQL Server, even as the most deployed relational database out there, is not really a direct business. But our cloud services are – and when creating products like Power BI and Azure SQL Database we have to think about and develop for things like viral adoption, attracting individual customers to the service, using telemetry every day to optimize experiences, etc.

My two boys keep me quite busy outside work of course. My wife and I try to maximize our time together with them enjoying all that the area has to offer – Seattle is a great city, and the pacific northwest has unbeatable skiing and hiking.

In case you missed, here is Part 1 of the interview.