Academic Research as Unfiltered Water
Contents of our research journals are like unfiltered water. Decision and policy makers should stand forewarned before they use the contents of academic journals to guide them in their work. Fortunately, most of them maintain a healthy skeptical attitude to our findings and prescriptions until they have more evidence on their efficacy.
This is not to minimize the labor of editors and referees who spend months scrutinizing scholarly work for shortcomings before allowing it to be published in journals. This work may be necessary but rarely sufficient to prepare new ideas and research findings to be put to actual use. The editorial process is already quite arduous, and making it longer will add more delay before new ideas can be disseminated for broader discussion and evaluation.
Scholarly scrutiny is based on scholarly knowledge with its own limitations. There is much about the world that even the most knowledgeable person does not know. Full consequences of most, if not all, research reports cannot be known until they go through a much longer process of filtration at multiple levels. This can take years, even decades. Only a small fraction of new ideas survive this lengthy scrutiny and experience through trials in the field. When they do, we have greater confidence that the results of putting them into practice have a lower chance of yielding a surprise.
Research findings published in academic journals rarely have had the opportunity to be filtered by experience, robustness, and common sense. Researchers, public relations offices of universities and corporations, as well as the mass media have all the incentives to pronounce on the practical implications of new findings from academic research way before these findings are ready for the prime time. It is only an aggressive decision maker who jumps at such announcements without allowing for the fact that research journals are forums for proposing new ideas that show some initial promise; some good ideas are mixed in with a lot of bad ones in this offering. Temptation to get a jump on the competition, being photographed for the newspapers and testifying in Congress is high for those willing to take the risk of being proved wrong.
The recent brouhaha about the Reinhard and Rogoff research paper is hardly a unique example. Computational errors are not that uncommon. But given time, they have a better chance of being caught and rectified before they do much damage. The story may be remembered more for the consequences of the alacrity in incorporating unfiltered academic research into public policy.