Does empirical economics follow the scientific method?

Some activities of economists are valued more than others in the academic pecking order. Research trumps teaching. Journal articles often have more impact than books (unless you are as well known as Mankiw or Krugman and can corner the 'principles' textbook market). As John Quiggin remarks:

..books are a high-effort, low-payoff exercise for economists, unless you have something you really need to say at book length.

But surely lowest of the low activity must be attempting to replicate others work? If you get the same result, journals certainly won't want to publish it - and even if you don't, there's no guarantee they will (some journals won't touch them). Yet we all know that published results are often not as robust as authors like to portray them. Results can vary widely depending on the period chosen, the dummies used, the regression method, treatment of outliers etc. And documentation is often surprisingly sloppy.

If one were to venture down the replication path, though, what are your chances? Not that high, according to a new paper B. D. McCullough, Kerry Anne McGeary and Teresa D. Harrison from Drexel University: Do Economics Journal Archives Promote Replicable Research? (Hat tip: Craig Newmark).

All the long-standing archives at economics journals do not facilitate the reproduction of published results. The data-only archives at Journal of Business and Economic Statistics and Economic Journal fail in part because most authors do not contribute data. Results published in the FRB St. Louis Review can rarely be reproduced using the data/code in the journal archive. Recently-created archives at top journals should avoid the mistakes of their predecessors. We categorize reasons for archives’ failures and identify successful policies.

The authors attempted to replicate the results from 138 empirical articles in the Federal Reserve Bank of St. Louis Review. Of the 138 articles, 29 employed software they did not have or used proprietary data, and were excluded. Of the remaining 117, they were able to replicate only 9 of them: a lousy 8 per cent. In 2001 the National Research Council wrote that:

The ability to replicate a study is typically the gold standard by which the reliability of scientific claims are judged.

Judging by these results economics would not even meet a bronze standard, let alone gold. Many economists think that most articles are replicable, and hence the issue is not a matter for concern. They are wrong.

McCullough et al argue that the incentives for authors to comply with replication policies (or journal archives) are weak; many don't comply, and there are seldom any sanctions against them. The honour system of most journals simply don't work. They put the case for mandatory archiving of both data and program code:

Data and/or code archives are supposed to make it easier for one researcher to build on the work of another. A necessary condition for this is that the archived material enables the replication of the published results. “Data only” archives do not meet this condition, since the article cannot possibly describe everything that was done to the data. Neither do data/code archives meet this condition if data and code are not archived or if the data and code are not those that actually produced the published results.

It is the duty of the journal editors to ensure that authors supply material to the archive, and it is the duty of the author to ensure that the archived materials will accurately reproduce the published results. ...Generally, it is the duty of the editors to provide authors with incentives to comply with the journal archiving policy, and the failures of these archives must be laid at the feet of their editorial boards.

Note: Also worth reading is an earlier paper by Richard G. Anderson, William H. Greene, Bruce D. McCullough, and H. D. Vinod: The Role of Data & Program Code Archives in the Future of Economic Research. Amongst other things, it discussses the supply of and demand for replication, the current 'strategic information problem', and summarises much of the earlier research. The authors found that most economic journals don't even have a policy on replication:

So, what is the “replicability” policy of most current professional journals? Most journals have no such policy. A few journals have “replication policies” whereby authors “must” provide data and code upon request, and even fewer journals have mandatory data/code archives, where the researcher must deposit his data and code prior to publication. There is no journal, however, that can provide tangible evidence that its published results are, generally speaking, replicable. To date, every systematic attempt to investigate this question has concluded that replicable economic research is the exception and not the rule.

Noting some recent progress, Anderson et al conclude with a plea:

For empirical economics to at last become a science, the profession must embrace the scientific method by archiving and sharing data and program code.

UPDATE: I should have mentioned a related weblog discussion on statistical significance and data mining: see Kevin Drum's post Lies, damn lies and... and Andrew Gelman's comment: Something fishy in political science p-values Of course, this problem of publication bias goes well beyond political science (Hat tip: Mark Thoma).

Comments

[John Quiggin]

I always encourage Honours and PhD students to start with a replication exercise, if only to impress on them how difficult it is.

[A. PERLA]

Are we talking about "economist replicants", as in the Blade Runner film?

Now, THAT is an interesting thought! Built-in data gathering and regression analysis programs.

Wow! (; ^ )

[AJE]

In our Econometrics class we had to do a replication. I replicated a replication, and couldn't verify either!

[Laurent GUERBY]

Well, if one looks at the current inflation measure "debate" on economics blog (and of the past decades...), it's clear that if BLS raw price data was available the "debate" would become a bit more like science and a bit less faith-based or anecdote-based.

First rule of economics: never publish your data. People who don't trust you are just people who don't understand economics. Enough said!

Hope that attitude will change at some point, there are very interesting and very hard problems in economics, more reasons to (re)use what has been learned in hard sciences.

[A. PERLA]

Guerby : "it's clear that if BLS raw price data was available the "debate" would become a bit more like science and a bit less faith-based or anecdote-based."

Oh, how so? You track the data and then try to explain it. The result is still subjective because objective explanations remain interpretative ... and therefore subjective. (Why do you think that all the economists in the world, set head to foot, still couldn't reach a conclusion. Joke! ;^)

Economics is first and foremost an empirical science, and empirical evidence, I suggest, since it is observational, remains interpretive (as we try to bend it into a "theory".)

A graph of raw price data will stimulate as many opinions explaining its behaviour as there are observers. A regression may give a trend line that history has shown highly variable depending upon the period in question. Look at what happened to the "Philips curve" in another post, over a period of a few decades ...

[A. PERLA]

Post scriptum: And, as regards econometric forecasting models ... I've wasted company money on a good many. The survey in the Economist is for free and does just as well at a macro level ...

[Laurent GUERBY]

A. PERLA, first it's obvious there's no magic inflation number representing honestly the situation of every single person, so there will always be a subjective judgement to make so I'm in agreement with you here :).

Then the question is for each one interested in having some case-specific inflation measure, how do you obtain one? Simple answer: publish the data collected using taxpayer money and let the market provide appropriate inflation numbers.

For example some countries do reevaluate pensions or state welfare based on CPI. But is it optimal to reevaluate old people pensions with the same number as young single mother welfare? Their cost are probably widely different as is their effective inflation. Same thing for geographic and lifestyle differences.

Do we agree?

[Rob Elliott]

Excellent post - we have linked to it from our blog "http://globalisation-and-the-environment.blogspot.com/". This issue will only increase in importance and it is good to see that the top economics journals are moving in that direction. A recent talk/paper by David Card also touches on this issue when he discusses the future of labour economics.

[Scott Peterson]

I think that your closing quote would best be restated by saying that "For economics to at last become a science, the profession must embrace the scientific method by archiving and sharing data and program code"; note that I left out the word "empirical". For economics to be referred to as a science, period, no theory can be accepted unless research backing up said theory is found to be replicable.

[Arthur Eckart]

There's good and bad science. Empirical economics has often proven conventional wisdom or intuitive thinking was wrong. However, it has also been (mis)used for biased outcomes.