|
Policy Framework
Many low-income children in the early grades need after-school care. And many of these children score well below their more advantaged peers on standardized tests of reading and math. The confluence of these circumstances suggests that it may be possible to strengthen the academic component of after-school programs (now often confined to providing help with homework) to better enable children participating in these programs to develop solid literacy and numeracy skills.
Whether or not imparting a stronger instructional focus to after-school programs will bring about the desired goals is uncertain. The programs operate at the end of a school day, when children (and teachers) may be tired, less able to concentrate, and expecting a change of pace, so that activities and instructional materials must be engaging. Participation in after-school programs is voluntary, and program attendance is likely to be less consistent than during the school day, making continuity of instruction a challenge. Staff members in after-school programs typically include not only certified teachers but also paraprofessionals and sometimes volunteers, and time for instructor training and daily preparation is generally limited. Finally, while in-school curricula have the advantages of being tied to state standards and (in some cases) being scientifically-based, it is not clear that they can be adapted to an after-school setting. For all these reasons, the U.S. Department of Education selected MDRC to mount a rigorous test of whether children in grades 2-5 who receive enhanced academic instruction in after-school programs have better academic outcomes than their peers who get more typical after-school academic support.
Agenda, Scope, and Goals
Before launching a full-scale test of the effectiveness of enhanced academic instruction, it was important to be sure that the math and reading curricula that were evaluated were strong, that they could be adapted for use in after-school programs, and that they could be implemented successfully. MDRC’s evaluation therefore included both a pilot phase and an implementation phase. The pilot phase involved selecting curriculum developers to modify existing reading and math materials for the after-school setting; developing additional instructional materials as needed; implementing the new materials on a limited scale during the 2004-2005 academic year to assess their feasibility and promise; and refining the curricula in light of this experience.
Upon successful completion of the pilot phase, a full-scale random assignment study of the impacts produced by the two curricula was mounted in a larger number of programs during the 2005-2006 school year. The programs received technical assistance and support to help ensure solid implementation. The questions this test was designed to address include:
- Does enhanced instruction in after-school programs that use reading and math curricula adapted from the regular school day significantly boost reading and math proficiency, as measured by test scores and overall academic achievement (for example, grades)?
- What are the effects of the after-school reading and math interventions on other in-school outcomes, such as proficiency in other subject matters, homework quality and completion, classroom behavior (for example, engagement), and grade retention?
- What effect does participation in the intervention have on overall academic effort? Do participants increase, decrease, or hold constant the amount of time they spend on academic work outside the classroom?
- Do the interventions’ impacts differ for students of different genders, ages, grade levels, ethnicities, and academic proficiency? Which students benefit the most from participation?
Design, Sites, and Data Sources
During the pilot phase, four after-school centers tested the reading materials, and another four centers tested the math materials. These centers were located in Atlanta, Georgia; Anne Arundel County, Maryland; Bridgeport, Connecticut; and Norristown, Pennsylvania.
The design for the random assignment study called for a sample of approximately 50 after-school centers. Twenty-five of these centers used the reading curriculum, and the remaining 25 employed the math curriculum. Within each center, students were randomly assigned to receive the enhanced program or to receive the kind of academic support usually provided by the program.
During the full study phase, data were drawn from a variety of sources. Intake forms and achievement tests administered at intake are used to describe the sample; to compare students in the sample with others in the district, in after-school programs, and in urban schools more generally; and to define subgroups of students on the basis of their pre-random assignment characteristics. Attendance data allow us to examine the intensity of participation in the services under study, while field research, document review, and monitoring reports enable us to understand the nature of the services provided and the fidelity with which the enhanced instruction is delivered. A follow-up survey is being used to describe participation by sample members in other after-school activities that are intended to improve their academic and youth development outcomes. Finally, follow-up achievement tests and surveys are used to estimate differences in student outcomes between the program and control groups.
What's Next
A report offering findings after one year of implementation was released in June 2008. The project is continuing into a second year of implementation, data collection, and analysis. In this final phase of the project, the analysis will examine whether impacts produced in the second year of implementation are greater than those produced in the initial year and what the impacts are of offering students access to the program for two school years. A final report will be released in 2009.
|
|
