---
title: "The Reproducibility Debate in Science"
author: |
  | John Maindonald
  | Centre for Mathematics and Its Applications
  | Australian National University
date: "25 November 2015"
output: ioslides_presentation
subtitle: Statisticians need to get involved
bibliography: osc.bib
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = FALSE, fig.retina=NULL)
```

## Focus

Focus is on comparative studies, e.g., compare speed 
of walking between group that has been primed to think
about the disabilities of aged with unprimed group.

* Null hypothesis: Priming does not slow walking speed  
* Alternative: Does slow walking speed, i.e., there is an effect  

A small $p$-value (typically $p \leq \alpha$, with $\alpha = 0.05$) is 
commonly taken to justify rejection of the Null, implying a real effect.   

* For medical research, focus is on pre-clinical studies
* Limited relevance to co-operative multi-disciplinary studies
* Carry-over from animal or in vitro models to humans is a 
separate discussion.

## Aside: Fisher on _p_-values

<p style="color:blue">If one in twenty does not seem high enough 
odds, we may, if we prefer it, draw the line at one in fifty 
(the 2 per cent point), or one in a hundred (the 1 per cent point). 
... <strong>A scientific fact should be regarded as experimentally 
established only if a properly designed experiment rarely 
fails to give this level (0.05 or 0.02) of significance.</strong>^1^
</p>

<p style="color:blue"> ... we may say that a phenomenon is 
experimentally demonstrable when we know how to conduct 
an experiment which will rarely fail to give us 
a statistically significant result.$^2$</p>  
<br>

<div style="font-size:smaller" "font-size:smaller">
1: @r17_fisher_1926  
2: @r18_fisher_1937
</div>

## Sources of evidence
* Ioannidis (2005)^1^ --- '... Most Published ... Findings Are False' 
    + This paper put reproducibility issues "on the map"
* Direct evidence --- results do not reproduce
    + Examples shortly, best evidence is in psychology
    + Most worrying evidence is in cancer studies
* Warning signals, from examination of papers
    + 'lack of +ve & -ve controls', faulty stats,   
  'inappropriate use of key reagents', 'failure to repeat', ...
* Results may be unreplicible (check paper to see this)
    + Key information may be omitted or wrong 

<div style="font-size:smaller">
1: @r19_ioannidis_2005, 'Why Most Published Research Findings Are False' 
</div>

## Selected Evidence

* Amgen: Reproduced 6 only of 53 'landmark' cancer studies.$^1$
    + @r2_begley_2013 notes issues with the studies that failed
* Bayer: Main results from 19 of 65 'seminal' drug studies
    + NB, journal impact factor was not a good predictor!$^2$  
* fMRI studies: 57 of 134 papers (42%) had $\geq$ 1 case lacking
check on separate test image. 
Another 14%, unclear ...$^4$  
* The Reproducibility:Psychology Project (~40% replicated)
    + Summary of results: 28 Aug 2015 issue of Science

<div style="font-size:smaller" "font-size:smaller">
1: @r23_begley_ellis_2012, 'Raise standards ...'; NB also @r2_begley_2013  
2: @r9_prinz_schlange_asadullah_2011, 'Believe it or not ... drug targets'  
3: @r8_kriegeskorte_simmons_bellgowan_baker_2009, '... dangers of double dipping'  
4: @r1_osc_EstRep2015, 'Estimating the reproducibility of psychological science'
</div>

<div class="notes">
Begley's 6 red flags  

- Blinding?
- Were basic expts repeated?
- Are all results given?  
  (Cross-check images)
- +ve & -ve controls?
- Were reagents validated?
- Flawed statistical analysis
</div>

##
```{r repPsych, echo=FALSE, message=FALSE}
df <- read.csv('repPsych.csv')
ord <- order(df$T_r.O)
df <- df[ord,]

library(mgcv, quietly=TRUE)
library(latticeExtra, quietly=TRUE)
gph <- xyplot(T_r.R ~ T_r.O, data=df, groups=Discipline,
     xlab = list("Effect size r (original)", cex=1.25), alpha=0.75,
     ylab = list("Effect size r (replication)", cex=1.25),
     xlim = c(-.2, 1), ylim = c(-.3, 1),
     xaxs="i", yaxs="i", aspect = 1, 
     auto.key=list(text=c(levels(df$Discipline),levels(df$Journal)),
                   points=TRUE, space="right"
                   ),
     scales=list(cex=1.0, tck=0.5),
     par.settings=simpleTheme(pch=c(16,16,c(3,5,4)), cex=1.5,
                              col=c(trellis.par.get()$superpose.symbol$col[1:3], rep("black",2))),
     main="Psychology: Open Science Collaboration Results"
)
gph2 <- xyplot(T_r.R ~ T_r.O, data=df, groups=Journal,
     xlim = c(-.2, 1), ylim = c(-.3, 1),
     xaxs="i", yaxs="i", 
     par.settings=simpleTheme(pch=c(3,5,4), col="white", lwd=2, cex=1.25)
)
# Fitted curve
RO.gam <- gam(T_r.R ~ Discipline+s(T_r.O, by=Discipline), 
              family=scat, data=df)
df[, c('fit','se')] <- predict(RO.gam, se=TRUE)
df <- within(df, {lower=fit-se
                  upper=fit+se})
gph+as.layer(gph2)+
  layer(data=subset(df, Discipline=="Social"), 
          col2 <- trellis.par.get()$superpose.symbol$col[2],
          llines(T_r.O, fit, col=col2),
          llines(T_r.O, lower, col=col2, lty=2),
          llines(T_r.O, upper, col=col2, lty=2),
        panel.abline(h=0, col="gray"),
        panel.abline(v=0, col="gray"),
        panel.abline(a=0, b=1, col="gray"))
```

## Collins & Tabak^1^ --- Factors include ... 

<div style="color:blue">
<ul style="list-style-type:none">
<li> poor training . . . in experimental design</li>  
<li> making provocative statements rather than presenting technical details</li> 
<li> Crucial experimental design elements that are too frequently ignored include blinding, randomization, replication, sample-size calculation and the effect of sex differences</li>  
<li> some scientists reputedly use a 'secret sauce' to make their experiments work — and withhold details . . . or describe them only vaguely . . .</li>
</ul>
</div>

Also: Deviations from stated protocol; errors in data; selective use of data;
selection effects

<div style="font-size:smaller">
1. @r15_collins_tabak_2014, '... NIH plans to enhance reproducibility'
</div>

## What are the issues?
* Faulty design/reporting/execution (Begley, Collins & Tabak) 
    + Repeating the same mistakes will not help
    + Some results are in principle unreplicable
* Selection effects -- mostly there is no effect
    + More enlightened use of p-values will help
    + More crucially, the total process is not transparent
    
We do not know, certainly not with any certainty, where the balance 
lies between these two different types of issue.  In this area, science
lacks a scientific understanding of its own processes.

## Is the criticism overblown?

* _The scientific process does finally identify the chaff_  
    _This contrasts with, e.g., alternative medicine._
    
Sure, but the process is far too protracted & tortuous.    
Too many methodological failures go undetected.  
Rewards systems encourage work that is poor quality.

* _One can never achieve 100% certainty_  

Sure, but we can do a lot better than at present.
Science should not be ignorant of its own processes.

* _Present processes are pretty much OK_ (implied, not said)

We have the technology needed to do a vastly better job, but are not using it!

## Journal & refereeing failures
* 'Publish' all; not just _p_ <= $\alpha$ ($\alpha$ = 0.05 or 0.01, or ...)
    + _p_ <= 0.01 rather than _p_ <= 0.05 is not an answer
* Referees & readers do not have the information needed
    + Know exactly what was done; check data, code
* Referees are at or beyond the limits of their expertise
    + Statistical analysis is an especial difficulty
* The system is not making good use of modern technology
    + plus, it interacts with rewards systems in malign ways
* Savvy critics are a huge untapped resource that is wasted
    + Other experts, in the area or in relevant areas
    + As, e.g., population projection, let the market choose.
    
    
__Open Science__'s response: make all processes transparent 
    
## Commentary in _Science_ (June 26 2015)
### 1. Self-correction in Science at work^1^

* Publish replications (PPS now has a section for this)
* Highlight & reward completeness of information 
* Encourage publishing well (not often), ...
* Create a culture that is willing to admit mistakes, ...
* School scientists in research ethics

### 2. Promoting an open research culture^2^
* Transparency & Openness Promotion (TOP) Guidelines

<div style="font-size:smaller" "font-size:smaller">
1: @r22_alberts_others_2015; 'Self-correction in science at work'  
2: @r21_nosek_others_2015; 'Promoting an open research culture'
</div>

## TOP's 8 standards --- 4 levels of each^1^

* <span>(1) Citation standards (data, code, materials)</span>
* (2)-(5) Transparency wrt data, analytic methods (code), research materials, design and analysis
* (6)-(7) Preregistration of studies, analysis plans   
* <span>(8) Replication</span> 

### Levels for <strong>(8) Replication</strong>, as an example

* level 0: Discourages
* level 1: Encourages
* level 2: Encourages, & conducts blind review of results
* level 3: Encourages, with a protocol
<br>

<div style="font-size:smaller" "font-size:smaller">
1: @r21_nosek_others_2015; 'Promoting an open research culture'
</div>

## Scholarship: Beyond the paper

<p style="color:blue">
"Now we are witnessing the transition to yet another scholarly communication system — one that will harness the technology of the Web to vastly improve dissemination.  ... The Web opens the workshop windows to disseminate scholarship as it happens, erasing the artificial distinction between process and product. ...</p>

<p style="color:blue">
Today's publication silos will be replaced by a set of decentralized, interoperable services that are built on a core infrastructure of open data and evolving standards — like the Web itself .... This 'decoupled journal' publishes promiscuously, then subjects products to rigorous review through the aggregated judgements of expert communities, supporting both rapid, fine-grained filtering and consistent, meaningful evaluation."
</p>

<div style="font-size:smaller">
Jason Priem:
Nature 495, 437–440 (28 March 2013) doi:10.1038/495437a
</div>

## An Open Source Model for Science

* Open Source Malaria --- think “Linux for Malaria Research"^1^  
  This follows a successful Schistosomiasis project.
  
* The Validation Science Exchange's Reproducibility Initiative^2^  
* Cancer Studies --- 50 "most impactful" from 2010-2012^3^
<br>

<br>

<div style="font-size:smaller">
1: @r32_todd_others_2015 (Matt Todd & others), @r33_osbr_2015  
2: @r34_iorns_others_2015 (Iorns & others)
3: @r25_errington_iorns_gunn_tan_lomax_nosek_2014; @r5_kaiser_2015, in June 26 2015 _Science_    
</div>

## Slides
Slides for this talk (pdf + R Markdown sources) will be posted at:
http://maths-people.anu.edu.au/~johnm/stats-issues/

## References


