Prev
Gregory PS, Editor: I hope you like this posting of DJ Patil as much as I do ! There is also a free report on Data Jujitsu you can download.
by DJ Patil , @dpatil | July 17, 2012
aving worked in academia, government and industry, I've had a unique opportunity to build products in each sector. Much of this product development has been around building data products. Just as methods for general product development have steadily improved, so have the ideas for developing data products. Thanks to large investments in the general area of data science, many major innovations (e.g., Hadoop, Voldemort, Cassandra, HBase, Pig, Hive, etc.) have made data products easier to build. Nonetheless, data products are unique in that they are often extremely difficult, and seemingly intractable for small teams with limited funds. Yet, they get solved every day.
How? Are the people who solve them superhuman data scientists who can come up with better ideas in five minutes than most people can in a lifetime? Are they magicians of applied math who can cobble together millions of lines of code for high-performance machine learning in a few hours? No. Many of them are incredibly smart, but meeting big problems head-on usually isn't the winning approach. There's a method to solving data problems that avoids the big, heavyweight solution, and instead, concentrates building something quickly and iterating. Smart data scientists don't just solve big, hard problems; they also have an instinct for making big problems small.
We call this Data Jujitsu: the art of using multiple data elements in clever ways to solve iterative problems that, when combined, solve a data problem that might otherwise be intractable. It's related to Wikipedia's definition of the ancient martial art of jujitsu: "the art or technique of manipulating the opponent's force against himself rather than confronting it with one's own force."
How do we apply this idea to data? What is a data problem's "weight," and how do we use that weight against itself? These are the questions that we'll work through in the subsequent sections.
To start, for me, a good definition of a data product is a product that facilitates an end goal through the use of data. It's tempting to think of a data product purely as a data problem. After all, there's nothing more fun than throwing a lot of technical expertise and fancy algorithmic work at a difficult problem. That's what we've been trained to do; it's why we got into this game in the first place. But in my experience, meeting the problem head-on is a recipe for disaster. Building a great data product is extremely challenging, and the problem will always become more complex, perhaps intractable, as you try to solve it.
Read more.