What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results

Omar Al-Ubaydli & John A. List & Dana L. Suskind, 2017. "What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results," American Economic Review, American Economic Association, vol. 107(5), pages 282-286, May.


  • Unscrupulous researchers might cherry pick certain results or data, or interpret ambiguous findings in favor of significant results, for example by not sharing the results of initial trials (Babcock and Loewenstein 1997).
  • Publication bias, ofren characterized by editors favoring studies that report significant results, worsens these problems by providing researchers with an added incentive to conduct suspect inference (Young, Ionnidis, and Al-Ubaydli 2008).
  • false positives lead to vast amounts of wasted public resources
  • There are also a wide variety of best practices that should be adopted by journal editors to combat publication bias, such as guaranteering journal space for replication studies and for studies that yield statistically insignificant results... (Yound, Ionnidis, and Al-Ubaydli 2008)
  • There exists a lively debate over the relative merits of the aforementioned data types (naturally-pccurring data/field experimental data/laboratory experimental data) in forming the basis of more general inference (Levitt/List 2007, Al-Ubaydli/List 2015, Deaton/Cartwright 2016)
  • adverse heterogeneity... the participants' attributes make them systematically predisposed to exhibiting a stronger relationship than in the population at large.
  •  Most of the experimental studies published in the economics literature are administered by the principal investigators, or their lieutenants,... They have a strong incentive to comply with whatever protocol they are investigating, as they seek to maximize the scientific value of their projected discoveries,... When such insights are scaled up, however, it is no longer practically possible ... Many medical trials do not anticipatie such scaling problems (Bonell, Hargreaves, and Rees 2006).
  • Problems stemming from inadvertently chaotic implementation of the original protocol are compounded by those relating to conflicts of interest, especially when rolling out revolutionary ideas, as these often challenge the power and established practives of incumbent organizations.
  • Speaking to policymakers has been a major goal of economists for centuries... the scholar must backward induct when setting up the original research plan to ensure swift transference of programs to scale.