 |
Research Methodology |
|
| |
|
 |
|
| |
|
 |
Whether and How to Use State Tests to Measure Student Achievement in a Multi-State Randomized Experiment
An Empirical Assessment Based on Four Recent Evaluations
|
| |
|
|
U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance
2011. Marie-Andrée Somers, Pei Zhu, and Edmond Wong.
This reference report, prepared for the National Center for Education Evaluation and Regional Assistance of the Institute of Education Sciences (IES), uses data from four recent IES-funded experimental design studies that measured student achievement using both state tests and a study-administered test.
|
|
| |
|
|
Designing and Analyzing Studies That Randomize Schools to Estimate Intervention Effects on Student Academic Outcomes Without Classroom-Level Information
Working Paper
|
| |
|
|
2011. Pei Zhu, Robin Jacob, Howard Bloom, and Zeyu Xu.
This paper provides practical guidance for researchers who are designing and analyzing studies that randomize schools — which comprise three levels of clustering (students in classrooms in schools) — to measure intervention effects on student academic outcomes when information on the middle level (classrooms) is missing.
|
|
| |
|
|
Nine Lessons About Doing Evaluation Research
Howard Bloom’s Remarks on Accepting the Peter H. Rossi Award
|
| |
|
|
2010. Howard S. Bloom.
In a speech before the Association for Public Policy Analysis and Management Conference on November 5, 2010, Howard Bloom, MDRC’s Chief Social Scientist, accepted the Peter H. Rossi Award for Contributions to the Theory or Practice of Program Evaluation.
|
|
| |
|
|
When Is the Story in the Subgroups?
Strategies for Interpreting and Reporting Intervention Effects on Subgroups
Working Paper
|
| |
|
|
2010. Howard S. Bloom and Charles Michalopoulos.
This revised paper examines strategies for interpreting and reporting estimates of intervention effects for subgroups of a study sample. Specifically, the paper considers: why and how subgroup findings are important for applied research, the importance of prespecifying subgroups before analyses are conducted, and the importance of using existing theory and prior research to distinguish between subgroups for which study findings are confirmatory, as opposed to exploratory.
|
|
| |
|
|
Finite Sample Bias from Instrumental Variables Analysis in Randomized Trials
Working Paper
|
| |
|
|
2010. Howard S. Bloom, Pei Zhu, and Fatih Unlu
This paper is the first step in a study of instrumental variables analysis with randomized trials to estimate the effects of settings on individuals. The goal of the study is to examine the strengths and weaknesses of the approach and present them in ways that are broadly accessible to applied quantitative social scientists.
|
|
| |
|
|
The Implications of Teacher Selection and Teacher Effects in Some Education Experiments
Working Paper
|
| |
|
|
2010. Michael J. Weiss.
In some experimental evaluations of classroom- or school-level interventions, random assignment is conducted at the student level and the program is delivered at the higher level. This paper clarifies the correct causal interpretation of “program impacts” when this study design is used and discusses the implications and limitations of this research design. A real example is used to demonstrate the paper’s key points.
|
|
| |
|
|
New Empirical Evidence for the Design of Group Randomized Trials in Education
|
| |
|
|
2009. Robin Jacob, Pei Zhu, and Howard S. Bloom.
This paper provides practical guidance for researchers who are designing studies that randomize groups to measure the impacts of educational interventions.
|
|
| |
|
|
Modern Regression Discontinuity Analysis
Working Paper
|
| |
|
|
2009. Howard S. Bloom.
This paper provides a detailed discussion of the theory and practice of modern regression discontinuity. It describes how regression discontinuity analysis can provide valid and reliable estimates of general causal effects and of the specific effects of a particular treatment on outcomes for particular persons or groups.
|
|
| |
|
|
Remarks on Accepting the Peter H. Rossi Award
|
| |
|
|
2008. Judith M. Gueron.
In a speech before the Association for Public Policy Analysis and Management Conference on November 7, 2008, Judith M. Gueron, President Emerita and Scholar in Residence at MDRC, accepted the Peter H. Rossi Award for Contributions to the Theory or Practice of Program Evaluation.
|
|
| |
|
|
Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions
Working Paper
|
| |
|
|
2008. Howard S. Bloom, Carolyn J. Hill, Alison Rebeck Black, and Mark W. Lipsey.
This MDRC working paper on research methodology explores two complementary approaches to developing empirical benchmarks for achievement effect sizes in educational interventions.
|
|
| |
|
|
Empirical Issues in the Design of Group-Randomized Studies to Measure the Effects of Interventions for Children
Working Paper
|
| |
|
|
2008. Howard Bloom, Pei Zhu, Robin Jacob, Stephen Raudenbush, Andres Martinez, and Fen Lin.
This MDRC working paper on research methodology provides practical guidance for researchers who are designing studies that randomize groups to measure the impacts of interventions on children.
|
|
| |
|
|
Empirical Benchmarks for Interpreting Effect Sizes in Research
Working Paper
|
| |
|
|
2007. Carolyn J. Hill, Howard S. Bloom, Alison Rebeck Black, and Mark W. Lipsey.
No universal guideline exists for judging the practical importance of a standardized effect size, a measure of the magnitude of an intervention's effects. This working paper argues that effect sizes should be interpreted using empirical benchmarks — and presents three types in the context of education research.
|
|
| |
|
|
The Core Analytics of Randomized Experiments for Social Research
Working Paper
|
| |
|
|
2006. Howard S. Bloom.
This MDRC research methodology working paper examines the core analytic elements of randomized experiments for social research. Its goal is to provide a compact discussion of the design and analysis of randomized experiments for measuring the impact of social or educational interventions.
|
|
| |
|
|
Making Random Assignment Happen
Evidence from the UK Employment Retention and Advancement (ERA) Demonstration
|
| |
|
|
UK Department for Work and Pensions.
2006. Robert Walker, Lesley Hoggart, and Gayle Hamilton, with Susan Blank.
The largest ever random assignment test of a social policy in Britain is being applied in a demonstration of the Employment Retention and Advancement (ERA) program. This report, written by MDRC and British colleagues as part of a consortium of social policy research firms and produced for the UK Department for Work and Pensions, examines how well random assignment worked.
|
|
| |
|
|
Using Covariates to Improve Precision
Empirical Guidance for Studies That Randomize Schools to Measure the Impacts of Educational Interventions
|
| |
|
|
2005. Howard S. Bloom, Lashawn Richburg-Hayes, and Alison Rebeck Black.
This paper examines how controlling statistically for baseline covariates (especially pretests) improves the precision of studies that randomize schools to measure the impacts of educational interventions on student achievement.
|
|
| |
|
|
Conducting Classroom Observations in First Things First Schools
Working Paper
|
| |
|
|
2004. Angela Estacion, Teresa McMahon, Janet Quint, with Bernice Melamud and LaFleur Stephens.
Relying on 427 classroom observations conducted over a three-year period, this study traces changes in teachers’ instructional practices in the First Things First schools.
|
|
| |
|
|
Sample Design for an Evaluation of the Reading First Program
|
| |
|
|
2003. Howard S. Bloom.
This paper illustrates how to design an experimental sample for measuring the effects of educational programs when whole schools are randomized to a program and control group. It addresses such issues as what number of schools should be randomized, how many students per school are needed, and what is the best mix of program and control schools.
|
|
| |
|
|
Exploring the Feasibility and Quality of Matched Neighborhood Research Designs
|
| |
|
|
2003. David C. Seith, Nandita Verma, Howard S. Bloom, George C. Galster.
|
|
| |
|
|
Intensive Qualitative Research
Challenges, Best Uses, and Opportunities
|
| |
|
|
2003. Alissa Gardenhire and Laura Nelson.
|
|
| |
|
|
"Help, I'm Getting Buried in Field Notes!"
A Manual for Qualitative Data Management and Analysis
|
| |
|
|
2003. Rebecca Widom with Herbert Collado.
|
|
| |
|
|
Using Instrumental Variables Analysis to Learn More from Social Policy Experiments
|
| |
|
|
2002. Lisa A. Gennetian, Johannes M. Bos, Pamela A. Morris.
|
|
| |
|
|
Using Place-Based Random Assignment and Comparative Interrupted Time-Series Analysis to Evaluate the Jobs-Plus Employment Program for Public Housing Residents
|
| |
|
|
2002. Howard S. Bloom, James A. Riccio.
|
|
| |
|
|
Can Nonexperimental Comparison Group Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs?
|
| |
|
|
2002. Howard S. Bloom, Charles Michalopoulos, Carolyn J. Hill, Ying Lei.
|
|
| |
|
|
Measuring the Impacts of Whole-School Reforms
Methodological Lessons from an Evaluation of Accelerated Schools
|
| |
|
|
2001. Howard S. Bloom.
|
|
| |
|
|
Extending the Reach of Randomized Social Experiments
New Directions in Evaluations of American Welfare-to-Work and Employment Initiatives
|
| |
|
|
2001. James A Riccio, Howard S. Bloom.
|
|
| |
|
|
A Meta-Analysis of Government Sponsored Training Programs
|
| |
|
|
University of Maryland Baltimore County.
2001. David H. Greenberg, Charles Michalopoulos, Philip K. Robins.
|
|
| |
|
|
Modeling the Performance of Welfare-to-Work Programs
The Effects of Program Management and Services, Economic Environment, and Client Characteristics
|
| |
|
|
2001. Howard S. Bloom, Carolyn J. Hill, James Riccio.
|
|
| |
|
|
A Regression-Based Strategy for Defining Subgroups in a Social Experiment
|
| |
|
|
2001. James J. Kemple, Jason C. Snipes with Howard Bloom.
|
|
| |
|
|
Assessing the Impact of Welfare Reform on Urban Communities
The Urban Change Project and Methodological Considerations
|
| |
|
|
2000. Charles Michalopoulos, Johannes M. Bos, Robert Lalonde, Nandita Verma.
|
|
| |
|
|
The Politics of Random Assignment
Implementing Studies and Impacting Policy
|
| |
|
|
2000. Judith M. Gueron.
|
|
| |
|
|
Building a Convincing Test of a Public Housing Employment Program Using Non-Experimental Methods
Planning for the Jobs-Plus Demonstration
|
| |
|
|
1999. Howard Bloom.
|
|
| |
|
|
Estimating Program Impacts on Student Achievement Using "Short" Interrupted Time Series
|
| |
|
|
1999. Howard S. Bloom.
|
|
| |
|
|
Using Cluster Random Assignment to Measure Program Impacts
Statistical Implications for the Evaluation of Education Programs
|
| |
|
|
1999. Howard S. Bloom, Johannes M. Bos, Suk-Won Lee.
|
|
