I just got pointed to this letter written by the President of the Alfred P. Sloan Foundation Paul Joskow on “Research Integrity and Reproducibility” and it is definitely worth a read.
It is available as part of the Sloan Foundation 2014 Annual report. It starts on pXiii. It is also available via Research Gate. And I have extracted the letter from the Annual Report and posted it here.
The letter provides a bit of background (including discussing the claims that Science is Broken and refutations of that claim). Joskow goes on to state ”
“I agree that the recent flurry of studies that raise questions about the credibility of results reported in important scientific papers should not lead to the conclusion that “science is broken.” However, this is not to say that there is no room for improvement”
I generally agree with that statement. He then outlines some questions related to this topic and to the issue of reproducibility:
- Why are there seemingly so many published research papers whose empirical results cannot be readily reproduced by other scientists?
- Why do there seem to be so many high profile retractions of published research papers?
- How can we change scientific practice, institutions, incentives, or norms in ways that lead to more reproducible, more reliable research?
And then he declares, “The Alfred P. Sloan Foundation has a keen interest in these questions.” which is good to see (although I did kind of know this already).
The rest of the latter then focuses on two particular topics
In what follows I will discuss how the structure of scientific careers and the incentives facing scientists and journals create barriers to conducting fully reproducible research. I will then discuss the opportunities grantmakers have to reduce or remove those barriers and what the Sloan Foundation is doing to make the published scientific literature more reproducible, reliable, and credible.
Among the topics he covers in the 1st topic:
- What Is Reproducibility?
- Failures of Reproducibility, Replicability, and Generalizability
- Incentives (with a funny footnote: Come on, I’m an economist. You think there wasn’t going to be a section on incentives?).
And then what I found most interesting is the part on “Opportunities for Grantmakers”
- Insist on high methodological standards
- Support the development of new tools that lower the costs of making data and methods available
- Fund replication projects
- Foster the development and adoption of norms and institutions devoted to good replication practices
- Train scientists directly
- Support data science professionals
- Explore ways to facilitate the repeated scientific analysis of private and proprietary data
Overall I think this is definitely worth a read for those working in any area of science or science funding or related activities.
Disclosures:
- I receive funding from the Alfred P. Sloan Foundation for the microBEnet project
- I really like the Alfred P. Sloan Foundation.
- The program officer for my microBEnet project Paula Olsiewski is the one who pointed me to this letter.