The Fourth Paradigm: Data-Intensive Scientific Discovery

a collection of essays written by Microsoft's and outside scientists, as a tribute to Jim Grey

A Deluge of Data Shapes a New Era in Computing

New York Times, By JOHN MARKOFF, December 14, 2009

In a speech given just a few weeks before he was lost at sea off the California coast in January 2007, Jim Gray, a database software pioneer and a Microsoft researcher, sketched out an argument that computing was fundamentally transforming the practice of science.

4th Paradigm Dr. Gray called the shift a "fourth paradigm." The first three paradigms were experimental, theoretical and, more recently, computational science. He explained this paradigm as an evolving era in which an "exaflood" of observational data was threatening to overwhelm scientists. The only way to cope with it, he argued, was a new generation of scientific computing tools to manage, visualize and analyze the data flood.

In essence, computational power created computational science, which produced the overwhelming flow of data, which now requires a computing change. It is a positive feedback loop in which the data stream becomes the data flood and sculptures a new computing landscape.

In computing circles, Dr. Gray's crusade was described as, "It's the data, stupid." It was a point of view that caused him to break ranks with the supercomputing nobility, who for decades focused on building machines that calculated at picosecond intervals.

Now, as a testimony to his passion and vision, colleagues at Microsoft Research, the company's laboratory that is focused on science and computer science, have published a tribute to Dr. Gray's perspective in "The Fourth Paradigm: Data-Intensive Scientific Discovery." It is a collection of essays written by Microsoft's scientists and outside scientists, some of whose research is being financed by the software publisher.

The essays focus on research on the earth and environment, health and well-being, scientific infrastructure and the way in which computers and networks are transforming scholarly communication. The essays also chronicle a new generation of scientific instruments that are increasingly part sensor, part computer, and which are capable of producing and capturing vast floods of data.

Read full story.