Some activities of economists are valued more than others in the academic pecking order. Research trumps teaching. Journal articles often have more impact than books (unless you are as well known as Mankiw or Krugman and can corner the 'principles' textbook market). As John Quiggin remarks:
..books are a high-effort, low-payoff exercise for economists, unless you have something you really need to say at book length.
But surely lowest of the low activity must be attempting to replicate others work? If you get the same result, journals certainly won't want to publish it - and even if you don't, there's no guarantee they will (some journals won't touch them). Yet we all know that published results are often not as robust as authors like to portray them. Results can vary widely depending on the period chosen, the dummies used, the regression method, treatment of outliers etc. And documentation is often surprisingly sloppy.
If one were to venture down the replication path, though, what are your chances? Not that high, according to a new paper B. D. McCullough, Kerry Anne McGeary and Teresa D. Harrison from Drexel University: Do Economics Journal Archives Promote Replicable Research? (Hat tip: Craig Newmark).
All the long-standing archives at economics journals do not facilitate the reproduction of published results. The data-only archives at Journal of Business and Economic Statistics and Economic Journal fail in part because most authors do not contribute data. Results published in the FRB St. Louis Review can rarely be reproduced using the data/code in the journal archive. Recently-created archives at top journals should avoid the mistakes of their predecessors. We categorize reasons for archives’ failures and identify successful policies.
The authors attempted to replicate the results from 138 empirical articles in the Federal Reserve Bank of St. Louis Review. Of the 138 articles, 29 employed software they did not have or used proprietary data, and were excluded. Of the remaining 117, they were able to replicate only 9 of them: a lousy 8 per cent. In 2001 the National Research Council wrote that:
The ability to replicate a study is typically the gold standard by which the reliability of scientific claims are judged.
Judging by these results economics would not even meet a bronze standard, let alone gold. Many economists think that most articles are replicable, and hence the issue is not a matter for concern. They are wrong.
McCullough et al argue that the incentives for authors to comply with replication policies (or journal archives) are weak; many don't comply, and there are seldom any sanctions against them. The honour system of most journals simply don't work. They put the case for mandatory archiving of both data and program code:
Data and/or code archives are supposed to make it easier for one researcher to build on the work of another. A necessary condition for this is that the archived material enables the replication of the published results. “Data only” archives do not meet this condition, since the article cannot possibly describe everything that was done to the data. Neither do data/code archives meet this condition if data and code are not archived or if the data and code are not those that actually produced the published results.
It is the duty of the journal editors to ensure that authors supply material to the archive, and it is the duty of the author to ensure that the archived materials will accurately reproduce the published results. ...Generally, it is the duty of the editors to provide authors with incentives to comply with the journal archiving policy, and the failures of these archives must be laid at the feet of their editorial boards.
Note: Also worth reading is an earlier paper by Richard G. Anderson, William H. Greene, Bruce D. McCullough, and H. D. Vinod: The Role of Data & Program Code Archives in the Future of Economic Research. Amongst other things, it discussses the supply of and demand for replication, the current 'strategic information problem', and summarises much of the earlier research. The authors found that most economic journals don't even have a policy on replication:
So, what is the “replicability” policy of most current professional journals? Most journals have no such policy. A few journals have “replication policies” whereby authors “must” provide data and code upon request, and even fewer journals have mandatory data/code archives, where the researcher must deposit his data and code prior to publication. There is no journal, however, that can provide tangible evidence that its published results are, generally speaking, replicable. To date, every systematic attempt to investigate this question has concluded that replicable economic research is the exception and not the rule.
Noting some recent progress, Anderson et al conclude with a plea:
For empirical economics to at last become a science, the profession must embrace the scientific method by archiving and sharing data and program code.
UPDATE: I should have mentioned a related weblog discussion on statistical significance and data mining: see Kevin Drum's post Lies, damn lies and... and Andrew Gelman's comment: Something fishy in political science p-values Of course, this problem of publication bias goes well beyond political science (Hat tip: Mark Thoma).