Batch Adaptive Designs to Improve Efficiency in Social Science Experiments

(2022)

(with Nicole Pashley and Dominic Valentino)

Social science experiments are vital for assessing causal effects, but they can be costly, which in turn makes them inaccessible to many researchers or may lead to small, sub-optimal sample sizes. In this letter, we show how a particular experimental design—the Neyman allocation—can lead to more efficient experiments that allow researchers to achieve the same level of statistical power as traditional designs but with a significantly smaller number of units. This design relies on unknown variances, however, and so previous work has proposed what we call the batch adaptive Neyman allocation (BANA) design that uses an initial pilot study of units to approximate the optimal Neyman treatment allocation in the second larger batch. We extend this design to the common multiarm experimental setting of political science, derive an unbiased estimator of the average treatment effect for the design, and show how to perform inference that is valid in finite samples. Simulations verify that the BANA design can lead to significantly more efficient experiments when the variance of the outcome differs across treatment conditions. Finally, we show the potential sample size savings that researchers could see in practice by reviewing the heteroskedaticicty of several recent experimental studies.