Using Covariates to Improve Precision
Empirical Guidance for Studies That Randomize Schools to Measure the Impacts of Educational Interventions
This paper examines how controlling statistically for baseline covariates (especially pretests) improves the precision of studies that randomize schools to measure the impacts of educational interventions on student achievement. Part I of the paper introduces the concepts, issues, and options involved. Parts II and III present empirical findings that illustrate how precision is influenced by a wide range of different covariates. These findings were based on multiple years of individual data for student test scores in reading and math from five urban school districts. They represent grades three and five for elementary schools, grade eight for middle schools, and grade ten for high schools. Part IV of the paper compares its results to those of previous research, presents an approach for quantifying uncertainty about its results, and considers what further research is needed. Findings indicate that: (1) pretests can reduce the number of randomized schools required for a given level of precision to about one-half of what would be needed otherwise for elementary schools, one-fifth for middle schools, and one-tenth for high schools; (2) aggregate school-level pretests are as effective in this regard as are individual student-level pretests; (3) the precision-enhancing power of pretests declines somewhat, but not much, as the number of years between the pretest and post-tests increases; (4) the precision-enhancing power of pretests for multiple baseline years is only slightly greater than that for a single baseline year; and (5) the precision-enhancing power of pretests is substantial, even when the pretest differs from the post-test.