In a randomized controlled trial, measuring treatment contrast – the difference in services received by a program group and those in a counterfactual condition – is critical for understanding what a program’s effects suggest about the best ways to improve services. This paper explains why treatment contrast is important and offers guidance about how to measure it.
A two-stage study design can test a complex set of interventions, individually and in combination. Reflections on Methodology shows how this approach was used for a pair of programs, the first administered in preschools and the second implemented as a kindergarten follow-up for individual students.
Multisite randomized trials allow researchers to study both the average impact of an intervention and how the impact varies across settings, which can help guide decisions in policy, practice, and science. This Reflections on Methodology post distills some key considerations for research design and for reporting and interpreting such variation.
Social network analysis models the structure of relationships using “nodes” (such as organizations) and “edges” (or ties, such as contracts). This Reflections on Methodology post highlights what the method can analyze — strength and complexity of connections, an organization’s positional power — in the context of a community development study in Chicago.
The proliferation of school choice systems offers researchers opportunities to study the effects of education reforms on a large scale, rigorously but relatively quickly. In the first of two posts on the subject, Reflections on Methodology discusses how to ensure that a school assignment process is truly random.
This working paper (forthcoming in July 2018 as a chapter in The ANNALS of the American Academy of Political and Social Science) updates the existing pipeline paradigm for evidence building with a cyclical paradigm that encompasses evidence building, implementation, and adaptation.
An important tenet of building reliable evidence is that study findings can be both reproduced and replicated and that the methods and data used stand up under scrutiny. This post in the Reflections on Methodology series outlines several ways to ensure credibility in research design and practice.
Assessing an intervention’s effects on multiple outcomes increases the risk of false positives. Procedures that make adjustments to address this risk can reduce power, or the probability of detecting effects that do exist. MDRC’s Reflections on Methodology discusses how to estimate power when making adjustments as well as alternative definitions of power.
To improve outcomes among high-interest borrowers, policymakers need to understand what is driving usage. This second post in MDRC’s Reflections on Methodology series discusses how a data discovery process revealed clusters of borrowers who differed greatly in the kinds of loans and lenders they used and in their loan outcomes.
A Literature Review
Examining the scholarly literature published since a seminal review in 2000, this working paper discusses the principles that underlie project-based learning, how it has been used in K-12 settings, the challenges teachers have confronted in implementing it, and what is known about its effectiveness in improving students’ learning outcomes.