Blogs and tweets could predict the future

In the time it takes you to read this sentence, more than a thousand tweets will have been twittered and dozens of blogs posted. Much of their content will be ephemeral fluff: personal gripes and tittle-tattle interesting to no one but the parties concerned. Yet despite this, it is possible to use that torrent of information to make predictions about social and economic trends that affect us all.

Interest in the idea of analysing web data to make predictions took off around a year ago, when researchers at Google used the frequency of certain search terms to forecast the sales of homes, cars and other products.

In their landmark study, Hal Varian, Google's chief economist, and his colleague Hyunyoung Choi showed how the volume of searches for certain products, such as types of car, rose and fell in line with monthly sales. Google keeps extensive records of what is being searched for, and that information is available almost instantaneously. That could make Varian and Choi's method a far quicker way of gauging purchasing behaviour than traditional sales forecasts, which are often made by looking back at purchasing patterns.

Now other sources, such as blog posts and tweets, are being mined too, and the variety of subject matter they address might mean that phenomena other than purchasing patterns can be explored. "The possibilities are enormous," says Joseph Engelberg, a finance researcher at the University of North Carolina at Chapel Hill.

Tweets may prove useful to political pollsters, for example. Bryan Routledge and his colleagues at Carnegie Mellon University in Pittsburgh, Pennsylvania, ran a sentiment analysis on tweets posted in the run-up to the 2008 US presidential election relating to candidates Barack Obama and John McCain. They used the results to try to assess voting intentions as the election neared.

The researchers found that this Twitter rating tracked more formal opinion polls closely. And while they were not able to improve on the accuracy of those polls, the work did show that Twitter could provide a cheaper, quicker alternative, says Routledge.

Read more.