Finite Sample Bias from Instrumental Variables Analysis in Randomized Trials

| Howard Bloom, Pei Zhu, Fatih Unlu

This paper is the first step in a study of instrumental variables analysis with randomized trials to estimate the effects of settings on individuals. The goal of the study is to examine the strengths and weaknesses of the approach and present them in ways that are broadly accessible to applied quantitative social scientists. This paper begins with the methodological limitations of conventional ways to study causal relationships, such as cross-sectional regression, longitudinal regression, and latent variables analysis. It then examines finite sample bias for the simplest application of the alternative instrumental variable approach of a single-setting characteristic and individual outcome, and studies how “clustering” — when units of analysis (for example, students) are randomized or treated in groups (for example, by school) — affects instrument strength and finite sample bias. The last part of the paper extends the discussion to situations with multiple instruments for a single mediator and outcome where multiple instruments are constructed from information on treatment status for multiple studies, sites, or subgroups (aka — strata). Specifically it addresses questions such as, “What happens when the treatment effect on the mediator is the same for all strata?” and “By how much must treatment effects on a mediator vary across strata in order for multiple instruments to reduce finite sample bias?” It also demonstrates that clustering does not affect answers to these questions.