---
title: "Reproducibility in Science --- Rethink, or Crisis?"
subtitle: A search for a more scientific science
author: |
  | John Maindonald
  | Centre for Mathematics and Its Applications
  | Australian National University
date: "`r format(Sys.time(), '%d %B, %Y')`"
output:
  ioslides_presentation
bibliography: osc.bib
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = FALSE, fig.retina=NULL)
```

## Focus

Focus is on comparative studies, e.g., 
compare a cocktail thought to slow aging 
in mice with a placebo   

* Null hypothesis: Cocktail does not slow aging  
* Alternative: Does slow aging, i.e., there is an effect  

A small $p$-value (typically $p \leq \alpha$, with $\alpha = 0.05$) is 
commonly taken to justify rejection of the Null, implying a real effect.   

* For medical research, focus is on pre-clinical studies
* Limited relevance to large co-operative multi-disciplinary studies
* Carry-over from animal or in vitro models to humans is a 
separate discussion.

## Overview
* Evidence of a problem --- 
    + Direct evidence --- results do not reproduce
    + Warning signals --- 'lack of +ve & -ve controls', faulty stats,   
  'inappropriate use of key reagents', 'failure to repeat', ...
* Use/misuse of $p$-values --- maybe mostly no effect, but ...
    + always, expect $\ge$ 5% of studies to show p $\le$ 0.05
* Initiatives (NB: articles in 'Science' for June 26 2015)
    + Pointers to a radical reshaping of research & publication?
* What needs to happen?

## Matching Ideal to Reality {.smaller}
If only!
 :   <div style="color:blue">The glorious endeavour 
 that we know today as science has grown out of the 
 murk of sorcery, religious ritual, and cooking.
 But while witches, priests and chefs were developing 
 taller and taller hats, scientists worked out a 
 method for determining the validity of their 
 results: they learned to ask <strong>"Are they 
 reproducible?"</strong>^1^</div> 
<br>

Reality
 :   <div style="color:blue">Scientists like to think of science as self-correcting.
 To an alarming degree, it is not.$^2$</div>  
<br>

<div style="font-size:smaller" "font-size:smaller">    
1: @r28_scherr_1983   
2: @r29_economist_editorial_2013, 'Unreliable research. Trouble at the lab',
October 19
</div>

## Selected Evidence

* Amgen: Reproduced 6 only of 53 'landmark' cancer studies.$^1$
    + @r2_begley_2013 notes issues with the studies that failed
* Bayer: Main results from 19 of 65 'seminal' drug studies
    + NB, journal impact factor was not a good predictor!$^2$  
* Failed attempts to reproduce widely cited priming studies$^3$  
  (Thinking about Grandpa makes one walk slowly?)
* fMRI studies: 57 of 134 papers (42%) had $\geq$ 1 case lacking
check on separate test image. 
Another 14%, unclear ...$^4$  

<div style="font-size:smaller" "font-size:smaller">
1: @r23_begley_ellis_2012, 'Raise standards ...'; NB also @r2_begley_2013  
2: @r9_prinz_schlange_asadullah_2011, 'Believe it or not ... drug targets'  
3: @r27_yong_2012, 'Replication studies: Bad copy'  
4: @r8_kriegeskorte_simmons_bellgowan_baker_2009, '... dangers of double dipping'
</div>

<div class="notes">
Begley's 6 red flags  

- Blinding?
- Were basic expts repeated?
- Are all results given?  
  (Cross-check images)
- +ve & -ve controls?
- Were reagents validated?
- Flawed statistical analysis
</div>

## Collins & Tabak lay it on!$^1$ 

<div style="color:blue">
__Factors include__  

<ul style="list-style-type:none">
<li> poor training . . . in experimental design</li>  
<li> making provocative statements rather than presenting technical details</li> 
<li> Crucial experimental design elements that are too frequently ignored include blinding, randomization, replication, sample-size calculation and the effect of sex differences</li>  
<li> some scientists reputedly use a 'secret sauce' to make their experiments work — and withhold details . . . or describe them only vaguely . . .</li>
</ul>
</div>

Also: Deviations from stated protocol; errors in data; selective use of data;
selection effects

<div style="font-size:smaller">
1. @r15_collins_tabak_2014, '... NIH plans to enhance reproducibility'
</div>

## $p$ is a relative measure of evidence
### If no real effects, expect $\alpha$ = 5% of cases to show effect

* What then if the probability of a real effect is very small?
    + Might explain the Bayer 19/65 result; unlikely for 6/53

### For what follows, note that $\alpha$ and power are tightly linked
* Power = Probability that a real effect will be detected
* Decrease $\alpha$ and, for a given design, power decreases.
* Increasing power (better or bigger expt) shifts the distribution of
$p$-values --- more fall under threshold (0.05, or ...)
* If there are no real effects, power is irrelevant!

## Fisher on $p$-values

<p style="color:blue">If one in twenty does not seem high enough 
odds, we may, if we prefer it, draw the line at one in fifty 
(the 2 per cent point), or one in a hundred (the 1 per cent point). 
... <strong>A scientific fact should be regarded as experimentally 
established only if a properly designed experiment rarely 
fails to give this level (0.05 or 0.02) of significance.</strong>^1^
</p>

<p style="color:blue"> ... we may say that a phenomenon is 
experimentally demonstrable when we know how to conduct 
an experiment which will rarely fail to give us 
a statistically significant result.$^2$</p>  
<br>

<div style="font-size:smaller" "font-size:smaller">
1: @r17_fisher_1926  
2: @r18_fisher_1937
</div>

## "... no ... fixed level" --- If only!

<p style="color:blue"> $\ldots$ the calculation is 
absurdly academic, for $\ldots$ <strong>no scientific worker 
has a fixed level of significance at which $\ldots$ 
in all circumstances, he rejects hypotheses; he rather 
gives his mind to each particular case in the light 
of his evidence and his ideas.</strong> Further, the 
calculation is based solely on a hypothesis, which 
$\ldots$ is often not believed to be true at all, so 
that the actual probability of erroneous decision
... may be much less than the frequency specifying the 
level of significance.</p>  

###
Fisher was not especially consistent. See
<span style="font-size:smaller">http://www.jerrydallal.com/lhsp/p05.htm</span>

###
<br>

<div style="font-size:smaller">
1: @r16_fisher_1956
</div>

## Thought Experiment (950 attempts) {.small}

Ratio of real to no relationship: 50:900 (1:18)  
$\alpha$ (Prob[detect if No]) = 0.05; Power (Prob[detect if Yes]) = 0.5  

_From 950 total_ | _900 True No_ | _50 True Yes_ | Total = 950
------------------| -------------- | -------------- | -----
Yes result    | 900 $\times$ 0.05 = 45 | 50 $\times$ 0.5 = 25  | 70
_True_:_Total_ ratio |              | | $\dfrac{25}{70}$ = 0.36  

<br>

For 'Yes', $p$ is from a distribution in the range $p \le 0.05$; thus $p = 0.01$, $p = 0.005$, and $p = 0.05$ all add just 1 to the count.  

The _True_:_Total_ ratio has the name Positive Predictive Value (PPV).  
False Discovery Rate (FDR) = 1 - PPV

## The Ioannidis results --- Observations$^1$

1) Large studies are more likely to yield true results.
2) With a smaller effect size, findings are less likely to be true.
3) Many relationships, less testing => low PPV (high FDR).
4) Flexibility in designs, definitions, outcomes \& analysis
reduce chances that results will be true.
5) Financial \& other interests \& prejudices reduce the 
likelihood that results will be true.
6) "Truth" is less likely in fields that are "hot" 
(justifiably?)

All the above have the proviso "other factors being equal".
<br>

###

<br>

<div style="font-size:smaller">
1: More details, including a model for the effect of bias; see   
@r19_ioannidis_2005, 'Why Most Published Research Findings Are False' 
</div>

## A Defendable Use of $p$-values
$p$-values are relative measures of the weight of evidence  

__Repeated__ $p \le 0.05$ results establish a result  

* How many repeats?  It surely depends on the prior odds.  
* Each new $p \le 0.05$ (or whatever) makes the null less likely!   
<br>

### Scenarios that might explain PPV = 37% 
(e.g., take Bayer 19/65 as 80% of $\tilde{} \dfrac{24}{65} \simeq 37$%
with real effect)   

* A ~1:27 prior odds for an effect, with power=0.8 
* Or ~1:17 prior odds, with power 0.5

## $p \le 0.05$ vs $p = 0.05$; prior odds 1:17

For a given $p$,  posterior odds = $\dfrac{-1}{e\, p\, log(p)}$ $\times$ prior odds^1^

$p$-value | Xplier | $\frac{1}{17} \times$ Xplier  | Prob (PPV)
----------| ------------ | -----------------------| -------
0.05      | 2.5          | $\dfrac{2.5}{17}$ | $\dfrac{2.5}{2.5+18}$ = 0.13
0.01      | 8.0          | $\dfrac{8}{17}$ | 0.32
0.001     | 53.2         | $\dfrac{53.2}{17}$ | 0.76

<br>
For $p \le 0.05$, power=0.5, Xply by 10.0 --- much as for
$p \simeq 0.0075$   
NB that $p \le 0.05$ is much stronger evidence than $p = 0.05$   

<div style="font-size:smaller">
1: Based on @r6_sellke_bayarri_berger_2001; cf also @r13_nuzzo_2014  
Presents a (slightly?) optimistic picture
</div>


## Tukey on Data Analysis Foundations$^1$   
**Distinguish Model Development from Inference:**  

* Models aim to give real world descriptions  
  NB: Simulation extends the range of useful models  
* Inference (or Generalization)  
  Use diverse challenges to build (or destroy!)
  confidence in inferences.
  
Accept inferences that have survived diverse challenges.  

* Challenges to all aspects, not just the statistics

<br>

<div style="font-size:smaller">
1: @r4_tukey_1997
</div>

<div class="notes">  
***Tukey's warns:*** 

* Do not assume "that we always know what in fact we never know -- the exact 
probability structure ..."  
* No data set is large enough to tell us how it should be analysed.  
[JM: Adequacy of contextual information is a further issue.]
</div>

## Types of Challenge
1. For experiments, critique/criticise the design.  
2. Look for biases in processes that generated the data.  
3. Look for inadequacies in laboratory procedure.  
4. Examine any available model diagnostics (NB plots).   
5. Check model on test data (Test data chosen how?)
6. Repeat the exercise (a training/test approach)  
    + But how robust is the replication?

Also, critique the science underlying interpretation of results.
 
## Current Practice relies mostly on:

* A hope that researchers will keep challenges 1 -- 4 in mind  
     + Are they equipped to think about such challenges?
* Challenges from referees.  
     + Statistical issues require statistically savvy referees
     + Do referees have all necessary details (Typically, no!)  
* Some high profile papers attract post-publication critique  
     + How can the system use/accommodate this?  

__Current practice wastes effort__ as other
researchers follow false leads, or go down what had
been identified (but not advertised) as blind alleys.

## Commentary in _Science_ (June 26 2015)
### 1. Self-correction in Science at work^1^

* Publish replications (PPS now has a section for this)
* Highlight & reward completeness of information 
* Publish well (not often), ...
* Create a culture that is willing to admit mistakes, ...
* School scientists in research ethics

### 2. Promoting an open research culture^2^
* Transparency & Openness Promotion (TOP) Guidelines

<div style="font-size:smaller" "font-size:smaller">
1: @r22_alberts_others_2015; 'Self-correction in science at work'  
2: @r21_nosek_others_2015; 'Promoting an open research culture'
</div>

## TOP's 8 standards --- 4 levels of each^1^

* <span>(1) Citation standards (data, code, materials)</span>
* (2)-(5) Transparency wrt data, analytic methods (code), research materials, design and analysis
* (6)-(7) Preregistration of studies, analysis plans   
* <span>(8) Replication</span> 

### Levels for <strong>(8) Replication</strong>, as an example

* level 0: Discourages
* level 1: Encourages
* level 2: Encourages, & conducts blind review of results
* level 3: Encourages, with a protocol
<br>

<div style="font-size:smaller" "font-size:smaller">
1: @r21_nosek_others_2015; 'Promoting an open research culture'
</div>

## Center for Open Science Projects$^1$
* Many Labs --- reproduce 13 classical psych studies$^2$
    + Of  13 studies --- 10: successful, 1: weakly, 2: no!  
    + Plots show scatter across the 36 participating teams  
* Reproducibility: Psychology --- 100 studies (3 journals, 2008)$^3$ 
    + 1 replicate only of each study
    + Subjectively, 39% replicated original results$^3$ 
* Cancer Studies --- 50 "most impactful" from 2010-2012$^4$
<br>

###

<br>

<div style="font-size:smaller">
1: @r31_cohoon_2015; 'COS (Center for Open Science)'; <a>https://cos.io/</a>    
2: @r7_klein_others_2014; 'Many Labs'   
3: @r1_osc_EstRep2015; 'Estimating the replicability ...'   
4: @r25_errington_iorns_gunn_tan_lomax_nosek_2014; 'An open investigation ... cancer biology research'
</div>

## 
```{r effect-size, echo=FALSE, message=FALSE, warning=FALSE, out.width="1000px"}
repos <- "http://cran.csiro.au"
if(!require(httr)){install.packages('httr', repos=repos)}
library(httr)
if(!require(latticeExtra)){install.packages('httr', repos=repos)}
library(latticeExtra)
if(!exists('MASTER')){
  # Read in Tilburg data
  if(!file.exists('rpp_data.csv'))
    info <- GET('https://osf.io/fgjvw/?action=download', 
                write_disk('rpp_data.csv', 
                           overwrite = TRUE)) 
  MASTER <- read.csv("rpp_data.csv")[1:167, ]
}
colnames(MASTER)[1] <- "ID" # Change first column name to ID to be able to load .csv file
useHere <- !is.na(MASTER$T_r..O.) & !is.na(MASTER$T_r..R.)
df <- MASTER[useHere, c('Journal..O.','Discipline..O.', 'Power..R.','T_r..O.','T_r..R.')]
df <- droplevels(df)
ord <- order(df$T_r..O.)
df <- df[ord,]
df$Power <- as.numeric(as.character(df$Power))
names(df)[1:3] <- c('Journal','Discipline', 'Power')

library(mgcv, quietly=TRUE)
library(latticeExtra, quietly=TRUE)
gph <- xyplot(T_r..R. ~ T_r..O., data=df, groups=Discipline,
     xlab = list("Effect size r (original)", cex=1.25), alpha=0.75,
     ylab = list("Effect size r (replication)", cex=1.25),
     xlim = c(-.2, 1), ylim = c(-.3, 1),
     xaxs="i", yaxs="i", aspect = 1, 
     auto.key=list(text=c(levels(df$Discipline),levels(df$Journal)),
                   points=TRUE, space="right"
                   ),
     scales=list(cex=1.0, tck=0.5),
     par.settings=simpleTheme(pch=c(16,16,c(3,5,4)), cex=1.5,
                              col=c(trellis.par.get()$superpose.symbol$col[1:3], rep("black",2))),
     main="Psychology: Open Science Collaboration Results"
)
gph2 <- xyplot(T_r..R. ~ T_r..O., data=df, groups=Journal,
     xlim = c(-.2, 1), ylim = c(-.3, 1),
     xaxs="i", yaxs="i", 
     par.settings=simpleTheme(pch=c(3,5,4), col="white", lwd=2, cex=1.25)
)
# Fitted curve
RO.gam <- gam(T_r..R. ~ Discipline+s(T_r..O., by=Discipline), 
              family=scat, data=df)
df[, c('fit','se')] <- predict(RO.gam, se=TRUE)
df <- within(df, {lower=fit-se
                  upper=fit+se})
gph+as.layer(gph2)+
  layer(data=subset(df, Discipline=="Social"), 
          col2 <- trellis.par.get()$superpose.symbol$col[2],
          llines(T_r..O., fit, col=col2),
          llines(T_r..O., lower, col=col2, lty=2),
          llines(T_r..O., upper, col=col2, lty=2),
        panel.abline(h=0, col="gray"),
        panel.abline(v=0, col="gray"),
        panel.abline(a=0, b=1, col="gray"))
```

## Initiatives --- Cancer and Other {.smaller}
__Cancer studies__  

* The Reproducibility Project --- Cancer Biology (noted above)$^1$ 
    + Replicate 50 "most impactful" studies from 2010-2012.  
    + Substantial progress, no published reports yet  
    + Raw datasets and data analyses will be publicly available  

__Applying the Open Source Model to Science__

* Open Source Malaria --- think “Linux for Malaria Research"$^2$  
  This follows a successful Schistosomiasis project.
  
__General__    

* The Validation Science Exchange's Reproducibility Initiative$^3$  
<br>

<div style="font-size:smaller">
1: @r25_errington_iorns_gunn_tan_lomax_nosek_2014; @r5_kaiser_2015, in June 26 2015 _Science_    
2: @r32_todd_others_2015, @r33_osbr_2015  
3: @r34_iorns_others_2015  
</div>

## Challenges for Statistics Education

* Get design issues back on the agenda  
* Use the Nature checklist$^1$ to monitor learning
    + Tick each item off as its import is mastered.  
* Drive home: $p$-value $\neq$ probability of a real effect!
* Analyses must be robust against any reasonable challenge  
    + Challenge science, design, execution, use of diagnostics.  
* This is an era of post-publication review.  Involve students. 

###
<br>

<div style="font-size:smaller">
1: http://www.nature.com/authors/policies/checklist.pdf
</div>

## Further Comment (1)
* A finding of 'no detectable effect' is important evidence     
    + It is essential context for 'successes'    
    + It may warn later experimenters against a blind alley.  
* The publishing model needs to change.  But how?  
    + Publish everything somehow, including "failures".  
    + Report on a plan of research, not the individual study.  
    + Post-publication review has a large & useful role.  
    + Bloggers may sometimes fill in for deficiencies in
scientific processes!  
* Cooperate in larger more definitive studies; look for a
dose-effect pattern. 

## Further Comment (2)

* Warnings for science funding & management systems:  
    + Avoid measures that corrupt what they presume to measure.  
    + Funding regimes are prone to reward a "muddle through ourselves" 
    mentality, with a devaluing of key skills (esp statistics?)  
* High quality work requires high level skills from all relevant 
specialisms.  Focus on people skills, not page counts!  
* In major areas, (social science, public health, ...)
new technologies offer rich new data sources and opportunities 
for new types of research.^1,2^  
<br>

<div style="font-size:smaller">
1: c.f. @r14_king_2013 'Restructuring the Social Sciences'  
2: @r24_pentland 'Social Physics'
</div>

## In conclusion
### Acknowledgements 

Dr Terry Neeman (ANU Statistical Consulting Unit)
and ANU SCU clients --- plied me with several of the key papers.
<br>

###

### Slides
Slides for this talk (pdf + R Markdown sources) can be found at:   
http://maths-people.anu.edu.au/~johnm/stats-issues/

## References


