---
title: "Reproducibility in Science - Rethink, or Crisis?"
author: "John Maindonald, MSI, ANU"
date: "revised Oct 23 2015"
output:
  beamer_presentation:
    fonttheme: structurebold
  ioslides_presentation: default
subtitle: How believable is published laboratory science?
---

## All is Not Well at the Lab {.smaller}
Myth?
 :   "The glorious endeavour that we know today as science has grown out 
of the murk of sorcery, religious ritual, and cooking. But while witches, 
priests and chefs were developing taller and taller hats, scientists 
worked out a method for determining the validity of their results: 
they learned to ask "Are they reproducible?"\footnote{Scherr 1983, in "The Best of the Journal 
of Irreproducible Results".}  
<br>

Reality
 :   "Scientists like to think of science as self-correcting.  To an alarming degree, it is 
 not."\footnote{Economist article, Oct 19 2013}

## Selected Evidence

* 9 attempts to reproduce a 1998 priming study failed 
  (Does thinking about a professor improve test performance?)
* Amgen scientists: reproduced results from 6 only of 53 'landmark' cancer studies.
(but no details are available)
* Bayer scientists: main results in 19 of 65 'seminal' drug studies
    + NB, journal impact factor was not a good predictor!   
    + Limited summary information, scant detail  
* Of 134 fMRI papers examined by Kriegeskorte et al (2000), 42% (57) had 
at least one case where results were checked on the images used to find
an effect. Another 14%, ?  
   
\alert{$p < 0.05$ has implied a low ($<$ 25\%) probability of a real effect!}   
\alert{Many/most were fishing experiments, plus other factors also?}
   
See also references in:
Landis et al (2012): "A call for transparent reporting to optimize the predictive value of preclinical research."  Nature 490: 187--191.

## The Wide Reach of Study Quality Issues
* Collins & Tabak (2014): __"Policy: NIH plans to enhance reproducibility"__. Nature 505, 612–613. {\small doi:10.1038/505612a}
* Yong (2012): __"Nobel laureate challenges psychologists to clean up their act"__. {\small Nature doi:10.1038/nature.2012.11535}  
* Chen (2013): __“Hidden depths: Brain science is drowning in uncertainty.”__ New Scientist, issue 2939, 17 October.
* Everitt, J I (2015): __"The Future of Preclinical Animal Models in Pharmaceutical Discovery and Development"__.  Toxicologic Pathology 43:70-77. {\small doi:10.1177/0192623314555162}
* Leek et al (2010): __"Tackling the widespread and critical impact of batch effects 
in high-throughput data."__  Nat. Rev. Genet. 2010 Oct;11(10):733-739.  
* Suter & Cormier (2014): __"The Problem of Biased Data & Potential Solutions for Health & Environmental Assessments"__.  Human and Ecological Risk Assessment, to appear. {\small doi:10.1080/10807039.2014.974499}

## Collins & Tabak (Nature, 27 Jan 2014) lay it on!
"__Factors include__  

<ul style="list-style-type:none"; color="blue">
<li>poor training . . . in experimental design</li>  
<li> making provocative statements rather than presenting technical details</li> 
<li> Crucial experimental design elements that are too frequently ignored include blinding, randomization, replication, sample-size calculation and the effect of sex differences</li>  
<li> some scientists reputedly use a 'secret sauce' to make their experiments work — and withhold details . . . or describe them only vaguely . . .</li>
</ul>
"

NB also: Mismatch between stated and actual protocol; selective use of data; data mistakes.

## A Thought Experiment (1100 relationships probed) 

Ratio of no to true relationship; here, \textcolor{brown}{1000}:\textcolor{blue}{100}  
Power (Prob[detect true relationship]);  \textcolor{blue}{here, 0.8}  
Type 1 error rate (\alert{set in advance});  \textcolor{blue}{here, 0.05}  

 _From 1100 total_ | _1000 True No_ | _100 True Yes_    | Total = 1100    
---------------| ------------- | ---- |   
Yes result     | 1000 $\times 0.05$ = \textcolor{brown}{50} | 100 $\times$ 0.8 = \textcolor{blue}{80}  | 130   
Fraction of total |  | \textcolor{blue}{PPV\footnote{PPV = \textcolor{blue}{P}ositive \textcolor{blue}{P}redictive \textcolor{blue}{V}alue; FDR (False Discovery Rate) = 1 - PPV} = $\frac{80}{130} = 0.615$}    

  
For the 'Yes' results, $p$ is from a distribution in the range $p \le 0.05$; thus $p = 0.01$, $p = 0.005$, and $p = 0.05$ all add just 1 to the count.  

$p \le 0.05$ may be roughly equivalent to an individual $p \simeq 0.0038$

## More detailed analysis\footnote{Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124} 

Ionnidis gives formulae that account for:  

* $u$ = Proportion of claimed results that in truth reflect bias  
* $n$ = # of independent teams addressing the same questions  
  
Note that:  

* Increasing the power shifts the distribution of observed
$p$-values so that more fall under the threshold. 
* Power is a function of $\alpha$.  Decrease $\alpha$ and,
for a given experiment, power decreases.

## The Ionnidis results --- Observations
  
\begin{enumerate}
\item Large studies are more likely to yield true results (NB, caveats)   
\item The smaller the effect size, the less likely that findings are true   
\item Many relationships, less testing => low PPV  
\item Flexibility in designs, definitions, outcomes \& analysis ...  
\item Financial \& other interests \& prejudices ... 
\item "Truth"" is less likely in fields that are "hot" (justifiably?)
\end{enumerate}

## $p \le 0.05$ versus $p = 0.05$ (with prior odds 1:10)\footnote{For a semi-popular
exposition ("$P$ values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume."), see:\\
Nuzzo, R. Scientific method: Statistical errors.    Nature 506, 150–152 (13 February 2014) doi:10.1038/506150a
(uses result from Sellke et al 2001)}

With prior odds = 1:10 (e.g.), what does $p$ = 0.05, \ldots, imply?

Slightly (?) optimistically, multiply prior odds by by $\frac{-1}{e\, p\, log(p)}$

$p$-value | Xplier | $\frac{1}{10} \times$ Xplier  | Prob
----------| ------------ | -----------------------| -------
0.05      | 2.5          | 2.5:10 | $\frac{2.5}{2.5+10}$ = 0.2
0.01      | 8.0          | 8:10 | 0.44
0.001     | 53.2         | 53.2:10 | 0.84

Xplier as for $p \le 0.05$ with power=0.8,
requires $p \simeq 0.0038$

$p = 0.05$ is weaker evidence than a lump all $p \le 0.05$ together.  
(but ideally, look at the individual $p$-value.)


## Fisher on $p$-values

\textcolor{blue}{If one in twenty does not seem high enough
odds, we may, if we prefer it, draw the line at one in fifty
(the 2 per cent point), or one in a hundred (the 1 per cent point).
. . . A scientific fact should be regarded as experimentally
established only if a properly designed experiment rarely
fails togive this level  [0.05 or 0.02] of significance.}
[The arrangement of field experiments.
J Minist Agric GB 33: 503–513, 1926.]

\textcolor{blue}{. . . we may say that a phenomenon is
experimentally demonstrable when we know how to conduct
an experiment which will rarely fail to give us
a statistically significant result.}  
[The Design of Experiments, 2nd Edition, 1937, p.16]

So then, repeated $p \le 0.05$ results are needed to establish
a result? [How many repeats?  It surely depends on the prior odds.]

Hence a way to shape $p$-value discourse to make it
defendable.  \alert{Each new $p \le 0.05$ (or whatever)
makes the null less likely!}

## Fisher in a very accommodating mode

\textcolor{blue}{\ldots the calculation is absurdly academic,
for \ldots no scientific worker has a fixed level of
significance at which \ldots in all circumstances,
he rejects hypotheses; he rather gives his mind to each
particular case in the light of his evidence and his ideas.
Further, the calculation is based solely on a hypothesis,
which \ldots is often not believed to be true at all, so that
the actual probability of erroneous decision, supposing such
a phrase to have any meaning, may be much less than the
frequency specifying the level of significance.}
[Fisher RA. Statistical Methods and Scientific Inference.
London: Oliver and Boyd/Longman Group, 1956.]

Fisher was not especially consistent.\footnote{See
http://www.jerrydallal.com/lhsp/p05.htm}

## Data Analysis Foundations (cf Tukey)
**Distinguish Model Development from Inference:**

* Models aim to give real world descriptions
  NB: Simulation extends the range of useful models
* Inference (or Generalization)
  Use diverse challenges to build (or destroy!)
  confidence in model-based inferences.

**Tukey's warnings:**

* Do not assume "that we always know what in fact we never know -- the exact
probability structure ..."
* No data set is large enough [and has sufficient background information] to tell
us how it should be analysed.

Tukey: __More honest foundations for data analysis.__ Journal of Statistical Planning and Inference, vol. 57, no. 1, pp. 21-28, 1997


## Types of Challenge
1. For experiments, is the design open to criticism?
2. Look for biases in processes that generated the data
3. Look for inadequacies in laboratory procedure
4. Model diagnostics (NB plots) & other critiques
5. Check model on test data (Test data chosen how?)
6. Repeat the exercise (the ultimate in training/test?)
    + But how robust is the replication result?

Add also, more general criticisms of the science that underlies
interpretation or results.

## Current Practice relies mostly on:

* A hope that researchers will keep challenges 1 -- 4 in mind
     + Are they equipped to think about such challenges?
* Challenges from referees.
     + Statistical issues require statistically expert referees
     + Do referees have all necessary details (Typically, no!)
* Some high profile papers attract post-publication critique
     + How can the system use/accommodate this?

__Current practice results in huge wasted effort__ as other
researchers follow false leads, or go down what others had
earlier identified (but not advertised) as blind alleys.

Hence, increasingly, a demand to __"Repeat the study"__

## The Science of Doing Science
* There is a science of the doing of science.  Statistics is
a large part of it.  So also is human psychology.
* Since 2012, these issues have gained increasing momentum
* Independent replication has become the surest way to
show that results are replicable.  Initiatives are afoot to
build it into scientific processes.
    + The checks must however be done to a high standard
    + How useful is a further $p < 0.05$ result?
    + Independendent replication is an 'easy' add-on
* Genuinely scientific approach: Understand the system
    + All results should be reported, 'successes' & failures
      This provides a needed context for judging 'successes'
    + This is not an easy add-on to current procedures
    + New Researchers --- do not believe all you read!

## Reproducibility Initiatives --- Psychology
* The Center for Open Science Many Labs project
    + reproduce 13 classical psych studies
    + 10 successful, 1 weakly reproduced, 2 not reproduced
    + Plots show the scatter across the 36 participating teams
      \textnormal{\footnotesize \color{blue} https://osf.io/ebmf8/}
* Replication Project: Psychology\footnote{https://osf.io/ezcuj/wiki/home/}
    + 270 authors participated in replications --- 100 studies
    + Studies were sampled from 2008 issues of 3 psych journals
    + Initial results: 39 reproduced + 24 "moderately similar" +
37 failures\footnote{http://www.nature.com/news/first-results-from-psychology-s-largest-reproducibility-test-1.17433}

## Initiatives --- Cancer and Other
__Cancer studies__
* The Center for Open Science has a $1.3 million grant from
the Laura and John Arnold Foundation, to fund replication
of the 50 "most impactful" cancer biology studies from 2010-2012.
    + Substantial progress, no published reports yet
    + Raw datasets and data analyses will be publicly available
    + Individual replication reports, plus final summary report
    + See as an example \textnormal{\color{blue} https://osf.io/tcauf/}

__General__
* The Reproducibility Initiative:
\textnormal{\color{blue} http://validation.scienceexchange.com/\#}

## Funding Agencies and Journals
* Collins & Tabak: ". . . NIH plans to enhance reproducibility", Nature, 27 Jan 2014
* Nature: __Statistics & general methods__ checklist
\footnote{http://www.nature.com/authors/policies/checklist.pdf}  
Match experimental designs against criteria --- demands for randomisation,
replication, local control (or blocking), etc. --- that parallel Fisher's criteria for
field experiments
* REMARK: REporting recommendations for tumor MARKer prognostic studies
\footnote{http://www.nature.com/nrclinonc/journal/v2/n8/full/ncponc0252.html}
* ARRIVE guidelines for animal experimentation:\footnote{http://www.nc3rs.org.uk/arrive-guidelines}
(maybe, really, for a different list.  NB that ethics committees
need to be aware of the statistical issues for efforts to reduce numbers
of animals used.)

## Challenges for Statistics Education

* Get design issues back on the agenda
* Hammer hard: $P(H_0\; |\; p <= 0.05) \neq Pr(p <= 0.05\; |\; H_0)$
* What does $p <= 0.05$ really imply?
    + Not much, unless the prior "no effect" proportion is known
    + A prize for the best short demonstration (paper or film)?
* Analyses must be robust against any reasonable challenge
    + Challenge science, design, execution, apply model diagnostics
* Familiarize students with the Nature style of checklist
    + Its items are a checklist for their learning.  Tick them off
    as each idea is mastered
* This is an era of post-publication review.  Get students involved.

The case for better understanding and better standards is now widely
recognised.

## Further Comment (1)

* In a context where 29% of positives are true positives
    + ~40:1 prior odds for the NULL, with power=0.8, might explain the ~29%
    + How much else is at play -- bad design, faulty analyses, ...
* A finding of 'no detectable effect' is important evidence
    + It provides essential context for a $p <= 0.05$ or $p <= 0.01$
    + It helps warn later experimenters against blind alleys
* The publishing model needs to change.  But how?
    + Publish everything, one way or another, including "failures".
    + Report on a plan of research, not the individual study.
    + Post-publication review has a large & useful role.
    + Bloggers may sometimes fill in for deficiencies in
scientific processes!
* Foster cooperation in large more definitive studies; ask how effect may change with dose.

## Further Comment (2)

* Warnings for science funding & management systems
    + Avoid measures that corrupt what they presume to measure.
    + Funding regimes are prone to reward a "muddle through ourselves"
    mentality, with a devaluing of key skills (esp statistics?)
* High quality work requires high level skills from all relevant
specialisms.  Focus on the people skills, not the page counts!
* In social science and areas such as public health, new technologies offer
rich new data sources and opportunities for new types of research.
[cf Gary King (2014): “Restructuring the Social Sciences: Reflections from Harvard's Institute for Quantitative Social Science .” PS: Political Science & Politics, 47, 165-172. doi:10.1017/S1049096513001534.]

## Ionnidis (2005) on what might be done
* Target large-scale evidence, where pre-study probability is high
    + Watch however for bias, as much in large studies as small
* ... many teams, ... what is the totality of the evidence?
    + upfront registration.
    + develop & adhere to a protocol
* Do not chase statistical significance
    + Assess, as far as possible, the pre-study odds (R)
    + Check findings that are judged relatively established
* Most new discoveries: from research with low pre-study odds
    + A single study will often give a very partial picture
    + Look for insight on the extent of pre-study data dredging?

Context: __Careful study design + accurate & full reporting.__

## References & Selected Further Papers
Begley and Ellis (2012): __Raise standards for preclinical cancer research.__  Nature, vol 483, pp 531-533.

Klein et al (2014). __Investigating Variation in Replicability. A ‘‘Many Labs’’ Replication Project.__
Social Psychology 2014; Vol. 45(3):142–152.

Kriegeskorte, Simmons, Bellgowan & Baker (2010):  __Circular analysis in systems neuroscience: the dangers of double dipping.__  Nature Neuroscience 12, 535-540.

Prinz, Schlange and Asadullah (2011). __Believe it or not: how
much can we rely on published data on potential drug targets?__
Nature Reviews Drug Discovery 10, p712.  

Sellke et al. __Calibration of $p$ Values for Testing Precise Null 
Hypotheses__. American Statistician, Vol 55: pp.62-71, 2001

## Further Papers
Couzin-Frankel (2013): __"When Mice Mislead."__  Science 342: 922-925.
{\small doi:10.1126/science.342.6161.922}

Pfeiffer, Bertram, Ioannidis (2011). __Quantifying Selective Reporting and the Proteus Phenomenon for Multiple Datasets with Similar Bias.__ PLoS ONE 6(3): e18362. doi:10.1371/journal.pone.0018362

## Acknowledgements \footnote{
Slides for this talk (pdf + R Markdown source) can be found at:
\textnormal{\color{blue}} \footnotesize
http://maths-people.anu.edu.au/~johnm/stats-issues/}

Dr Terry Neeman (ANU Statistical Consulting Unit)
and ANU SCU clients --- plied me with several of the key papers
that I referenced.

## More general issues -- Herricanes vs Himmicanes

\textnormal{\color{blue} Claim: Female hurricanes are deadlier than male hurricanes}

The suggestion is that they are taken less seriously?.\footnote{Jung et al
\url{http://www.pnas.org/cgi/doi/10.1073/pnas.1402786111}}

$\log$(E[\texttt{deaths}]) was regressed on damage in 2013 US\$
(graph that follows has 2014 US\$),
barometric pressure at landfall, femaleness of the name, \&
interactions.
\vspace*{9pt}

\textnormal{\color{blue} More realistically, transform the damage variable:}

With a $\log$ transformation, the femaleness effect vanishes.

Even better, use the Yeo \& Johnson (2000) extension of Box-Cox power transform methodology.\footnote{
"A new family of power transformations to improve normality or symmetry".
Biometrika, 87, 954-959.  From R, type \texttt{help(yjPower, package='car')}
}  Use transforms thus:
$\lambda = -0.19$ for \texttt{deaths+1};
$\lambda = 0.14$ for the damage variable.}
\vspace*{6pt}

A storm of reaction on the blogosphere was mostly himmicane.

## Hurricanes -- Deaths vs Damage in 2014 \$US

```{r doplot2, echo=FALSE, fig.width=6, fig.height=4.5}
hurricNamed <- DAAG::hurricNamed
library(latticeExtra, quietly=TRUE)
library(car, quietly=TRUE)
library(MASS)
ytxt <- c(1,3,10, 30, 1000)
xtxt <- c(1,100,10000, 1000000 )
funx <- function(x)x^0.14
funy <- function(y)-(y+1)^-0.19
gph <- xyplot(funy(deaths) ~ funx(NDAM2014),
       groups= mf,
       data=hurricNamed,
       scales=list(y=list(at=funy(ytxt), labels=paste(ytxt)),
                   x=list(at=funx(xtxt), labels=paste(xtxt))),
       xlab =expression("Damage (millions of 2014 US$);  transform yjPower("*lambda==0.14*")"),
       ylab=expression("Deaths;  transform yjPower("*lambda==-0.19*")"),
       auto.key=list(columns=2),
       par.settings=simpleTheme(col=c("red","blue"),
                                pch=16))
gph2 <- gph + layer(panel.text(x[c(13,84)], y[c(13,84)],
                       labels=hurricNamed[c(13,84), "Name"], pos=3,col="gray30", cex=0.8),
                    panel.text(x[c(13,84)], y[c(13,84)],
                       labels=hurricNamed[c(13,84), "Year"], pos=1, col="gray30", cex=0.8))
ab <- coef(lm(funy(deaths) ~ funx(NDAM2014) + mf, data=hurricNamed))
gph3 <- gph2+layer(panel.abline(ab[1], b=ab[2], col="red", alpha=0.4),
           panel.abline(ab[1]+ab[3], b=ab[2], col="blue", lty=2, alpha=0.4))
hurricJ <- hurricNamed[-(93:94),]
curve.nb <- glm.nb(deaths~NDAM2014+mf, data=hurricJ)
x <- exp(log(with(hurricJ, pretty(funx(NDAM2014), 40)))/0.14)
df<- expand.grid(mf=factor(c('f','m')), NDAM2014=x)
df[,'hat'] <- funy(predict(curve.nb, newdata=df, type='response')-1)
gph4 <- gph3 + latticeExtra::layer(data=df,
           panel.lines(funx(NDAM2014[mf=='f']), hat[mf=='f'], col='red',
                       lwd=2, lty=2, alpha=0.5),
           panel.lines(funx(NDAM2014[mf=='m']),
           hat[mf=='m'], col='blue', lwd=2, lty=2, alpha=0.5))
gph4 + layer(lims <- current.panel.limits(),
             xleft <- lims$xlim[1] + 0.01*diff(lims$xlim),
             ytop <- lims$ylim[2] - 0.125*diff(lims$ylim),
             panel.text(xleft, ytop, adj=0,
                        paste(" Dashed curves are for regressions\n",
                              "of log(E[deaths]) on 'Damage',\n",
                              "separately for males & females.\n",
                              "(Katrina & Audrey were excluded.)"), cex=0.8, col="gray30"))
```

  






