Hilary Mason Wants To Get You Started With Big Data

I spent part of this week with Hilary Mason, one of the smartest people that I know in Big Data. She works as the Chief Scientist for Bit.ly and has a wealth of skills at her fingertips that bridge computer science and mathematics. Plus, she is used to facing largely male audiences and just being the smartest person in the room. She was speaking at The Strange Loop conference in St. Louis this week, which should definitely be on your radar for next year if you are interested in this topic or want to broaden your programming skills.

Mason outlined in a series of workshops the tools you need to get started with manipulating Big Data and understanding the basics of machine learning, something she does everyday as she sifts through each one of those shortened URLs that we all create furiously.

The first step is setting up a proper environment, and for Mason it is a Linux machine with a variety of tools on it that you can find on her Github page linked above. She is a Python programmer, and so this reflects that interest. She uses Python with JSONview's Chrome extension, NLTK, numpy, Pycluster, hcluster, and mathplotlib. You can use most of these tools on other OSs too.

Second, you need to obtain a few test data sets that you can start to manipulate. Even if you aren't drinking out of the Bit.ly data fire hose, there are ways to get access to lots of great data around the Internet. Mason mentioned a few places, including:

• The New York Times. Each and every article that is posted on the main NYT Web site going back dozens of years has oodles of metadata and tags galore. Just view the source on any news article to see how the human editors have classified it. You'll need to start off by registering for an API key here and then select "Article Search API"
• Mason has put together several dozen different bundles of research-quality data sets at this link here
• Pete Skomoroch's various sample datasets that he has collected, and there is always the epic reference
• Data Source Handbook by Pete Warden

Third, you need to start thinking about how to make your data sets smaller. ...

Then comes the fun part, exploring your data. ...

Read more.