Projects: Evaluation of Academic Instruction in After-School Programs

Project

Overview

STATUS: Completed

Basics

Issue: P-12 Education

Focus Area: Afterschool/Out-of-School Time

Population: Children and Schools

Focus Population: Struggling Students, Elementary and Middle School Students

Locations: Alabama, California, Connecticut, Florida, Georgia, Kansas, Louisiana, Michigan, New Mexico, New York, Pennsylvania, Texas, Wisconsin

Intervention: Reading Instruction

Project Overview

Many low-income children in the early grades need after-school care. And many of these children score well below their more advantaged peers on standardized tests of reading and math. The confluence of these circumstances suggests that it may be possible to strengthen the academic component of after-school programs (now often confined to providing help with homework) to better enable children participating in these programs to develop solid literacy and numeracy skills.

Whether or not imparting a stronger instructional focus to after-school programs will bring about the desired goals is uncertain. The programs operate at the end of a school day, when children (and teachers) may be tired, less able to concentrate, and expecting a change of pace, so that activities and instructional materials must be engaging. Participation in after-school programs is voluntary, and program attendance is likely to be less consistent than during the school day, making continuity of instruction a challenge. Staff members in after-school programs typically include not only certified teachers but also paraprofessionals and sometimes volunteers, and time for instructor training and daily preparation is generally limited. Finally, while in-school curricula have the advantages of being tied to state standards and (in some cases) being scientifically-based, it is not clear that they can be adapted to an after-school setting. For all these reasons, the U.S. Department of Education selected MDRC to mount a rigorous test of whether children in grades 2-5 who receive enhanced academic instruction in after-school programs have better academic outcomes than their peers who get more typical after-school academic support.

FEATURED CONTENT

Report

The Final Report from the Evaluation of the Enhanced Academic Instruction in After-School Programs
This report presents two-year implementation and impact findings on two supplemental academic instruction approaches developed for after-school settings -- one for math and one for reading. It addresses whether one-year impacts are different in the second year of program operations and whether students benefit from being offered two years of enhanced after-school academic instruction.
- Overview
- Full PDF
Report

The Evaluation of Enhanced Academic Instruction in After-School Programs

Findings After the First Year of Implementation
This report presents one-year implementation and impact findings on two supplemental academic instruction approaches developed for after-school settings — one for math and one for reading. Compared with regular after-school programming, the supplemental math program had impacts on student SAT 10 test scores and the supplemental reading program did not — although the reading program had some effect on reading fluency.
- Overview
- Full PDF

Agenda, Scope, <br> & Goals

Agenda, Scope and Goals

Before launching a full-scale test of the effectiveness of enhanced academic instruction, it was important to be sure that the math and reading curricula that were evaluated were strong, that they could be adapted for use in after-school programs, and that they could be implemented successfully. MDRC’s evaluation therefore included both a pilot phase and an implementation phase. The pilot phase involved selecting curriculum developers to modify existing reading and math materials for the after-school setting; developing additional instructional materials as needed; implementing the new materials on a limited scale during the 2004-2005 academic year to assess their feasibility and promise; and refining the curricula in light of this experience.

Upon successful completion of the pilot phase, a full-scale random assignment study of the impacts produced by the two curricula was mounted in a larger number of programs during the 2005-2006 school year, and continued through the 2006-2007 school year. The programs received technical assistance and support to help ensure solid implementation. The questions this test was designed to address include:

Does enhanced instruction in after-school programs that use reading and math curricula adapted from the regular school day significantly boost reading and math proficiency, as measured by test scores?
What are the effects of the after-school reading and math interventions on other in-school academic behavior outcomes, such as homework quality and completion and classroom behavior (for example, engagement)?
Do the interventions’ impacts differ for students of different grade levels and academic proficiency? Which students benefit the most from participation?
What is the impact of the enhanced after-school program once the program has been in operation for a year, when after-school teachers may have more experience?
What is the impact on student academic performance after two years of program implementation?

Design, Sites, & <br> Data Sources

Design, Sites & Data Sources

During the pilot phase, four after-school centers tested the reading materials, and another four centers tested the math materials. These centers were located in Atlanta, Georgia; Anne Arundel County, Maryland; Bridgeport, Connecticut; and Norristown, Pennsylvania.

The design for the random assignment study called for a sample of approximately 50 after-school centers in the first implementation year. Twenty-five of these centers used the reading curriculum, and the remaining 25 employed the math curriculum. Within each center, students were randomly assigned to receive the enhanced program or to receive the kind of academic support usually provided by the program. During the second implementation year, 27 of the 50 after-school centers continued to participate in the study.

During the full study phase, data were drawn from a variety of sources. Intake forms and achievement tests administered at intake are used to describe the sample; to compare students in the sample with others in the district, in after-school programs, and in urban schools more generally; and to define subgroups of students on the basis of their pre-random assignment characteristics. Attendance data allow us to examine the intensity of participation in the services under study, while field research, document review, and monitoring reports enable us to understand the nature of the services provided and the fidelity with which the enhanced instruction is delivered. A follow-up survey is being used to describe participation by sample members in other after-school activities that are intended to improve their academic and youth development outcomes. Finally, follow-up achievement tests and surveys are used to estimate differences in student outcomes between the program and control groups.

Funders & <br />Partners

Funders

U.S. Department of Education

Partners

MDRC Staff

Fred Doolittle
Project Director
Vice President, Director, K - 12 Education Policy Area
Doolittle joined MDRC in 1986, initially specializing in studies of employment and training programs for economically disadvantaged out-of-school youth and adults. As MDRC expanded its work to include education, he focused on evaluations of elementary and secondary school reforms.
More
Marie-Andrée Somers
Senior Associate, K-12 Education
Somers is a Senior Research Associate who specializes in quantitative impact evaluation in the field of education. She is currently co-directing an evaluation of the BELL summer program, funded by the Edna McConnell Clark Foundation’s Social Innovation Fund project.
More
Pei Zhu
Senior Associate, K-12 Education
Zhu is an economist in MDRC’s K-12 Education policy area whose current work focuses on experimental and quasi-experimental impact analyses, evaluation design, and related methodological issues.
More
Rebecca Unterman
Research Associate, K-12 Education
Unterman is a coauthor of reports from MDRC’s evaluation of small high schools in New York City and from the Evaluation of Academic Instruction in After-School Programs.
More