US intelligence tests crowd-sourcing against experts

Can large groups be better predictors of terrorism and international events than analysts in government spy services?

LA Times, By Ken Dilanian, August 21, 2012

IARPA ... DARPA's sister agency - IARPA (the Intelligence Advanced Research Projects Agency), which funds experimental projects for the U.S. intelligence community - is running a four-year, $50-million program that pays people willing to predict major world events, including wars and terrorist strikes. Unlike the earlier scheme, participants can't profit from their predictions.

Now in its second year, the so-called crowd-sourcing project involves competing corporate and university teams, including UC Irvine. Each team includes more than a dozen social scientists and as many as 2,000 participants, who can answer hundreds of questions each if they want.

The study, known as Aggregative Contingent Estimation, is designed to see whether the 17 agencies in the U.S. intelligence community can aggregate the judgment of its thousands of analysts - rather than rely on the expertise of just a few - to issue more accurate warnings to policy makers before and during major global events.

Participants in the project give their best guesses on whether the Free Syrian Army will gain control of Aleppo, for example, or whether Kim Jong Un will resign as the leader of North Korea before April 1, 2013, according to project websites.

Initial results are promising, according to Philip Tetlock, a University of Pennsylvania professor who leads one of the teams. His 2005 book, "Expert Political Judgment: How Good Is It? How Can We Know?" showed that political pundits often fared worse in predictions than the proverbial dart-throwing monkey.

"The idea that the average of a group of forecasters is more accurate than any one forecaster goes back 100 years," Tetlock said. In 1906, British scientist Francis Galton determined that the average of 787 guesses about the weight of an ox at a county fair was startlingly accurate.

Read more.