Middle schools (grades 5 or 6 through grade 8) stand at a crucial intersection in American public education systems. They are charged with the responsibility of building on the basic literacy and numeracy skills that students learn in elementary school and with helping students master those skills if they have fallen behind. They must be prepared to nurture the physical, social, and intellectual growth that students undergo in their early adolescent years. Moreover, they are expected to provide students with the habits of mind and behavior that they will need to make healthy transitions to high school and young adulthood.

Yet too many middle schools are failing. In particular, schools that serve high-poverty student populations face significant obstacles that can keep them from providing adequate opportunities for teaching and learning. Many students enter middle school, for example, with poor preparation in reading, writing, and mathematics, and the schools never succeed in narrowing the gap as students approach high school. The size and complexity of many middle schools make them ill-equipped to deal with discipline problems that interfere with effective teaching and learning. Patterns of poor attendance and weak study habits often begin for students during the middle grades and become established behavior patterns. Nonselective middle schools in the urban district that is the focus of this report exemplify these problems.1 During the 1990s, for example, more than 75 percent of the seventh- and eighth-grade students in these schools had reading and math skills below grade level, and more than 25 percent could be considered chronic absentees (students with attendance rates of 80 percent or lower).

Given these challenges, many students leave middle school to enter high school without the foundation of skills and work habits that they need for academic success. Throughout the 1990s, for example, almost all eighth-grade students in the district were promoted to the ninth grade on time. Yet only about 60 percent of those students were promoted to the tenth grade a year later, and less than 50 percent were on schedule to graduate four years after starting high school

The Talent Development Middle School model is a comprehensive reform initiative designed to help transform the structure and curriculum of large middle schools in urban districts, with the aim of improving students’ levels of achievement and raising teachers’ and students’ expectations. The model includes a systematic reorganization of each school into small learning communities, organized around interdisciplinary teacher teams that share the same students and have common planning time. It infuses the curriculum with academic courses in English, language arts, mathematics, science, and U.S. history that are based on nationally recognized standards. Teachers receive professional development on the use of the curriculum and accompanying instructional practice, and each school employs the services of curriculum coaches to help support teachers on an ongoing basis. The model provides catch-up opportunities during the school day to students who are struggling with mathematics or reading.

Talent Development was created by practitioners and researchers at the Center for Research on the Education of Students Placed At Risk (CRESPAR), based at The Johns Hopkins University; the model operates in 21 middle schools nationwide. MDRC is conducting an independent, third-party evaluation of Talent Development, funded by the U.S. Department of Education’s Institute of Education Sciences, through CRESPAR. The evaluation and CRESPAR’s efforts to expand the use of Talent Development are part of the U.S. Department of Education’s Comprehensive School Reform Demonstration (CSRD) program. An earlier report by MDRC looked at the Talent Development High School model.2

This report focuses on preliminary findings for the impact of Talent Development on six nonselective, comprehensive middle schools that implemented the model in a northeastern, urban school district. (This district is the locus of Talent Development’s initial and most extensive scaling-up effort.) The six schools began working with Talent Development between 1996-1997 and 1998-1999 and are referred to throughout the report as “early-implementing schools.” As of the 2001-2002 school year, all six of the early-implementing schools had at least three years of implementation experience; four of these schools had four years of implementation experience; and two of those four schools had five years of implementation experience. Within each school, introduction of the core Talent Development components was usually a three- or four-year process. (The report also includes limited analysis for the first year of implementation in another five middle schools that began implementing the model in the district more recently.)

In addition to assessing Talent Development’s impacts on seventh- and eighth-grade students’ achievement and attendance, the report describes the context in which the Talent Development middle schools in the district operate, explains the model’s core components, and outlines the initial implementation of those components in the district.

A Rigorous Approach to Assessing Impacts

There are few rigorous studies of the effectiveness of comprehensive middle school reform interventions. Because such reforms affect an entire school, the challenge in evaluating their impact lies in identifying a group of similar students in similar schools that did not experience the intervention being tested and can provide a reliable “comparison” to tell evaluators what would have happened if business continued as usual. This type of comparison is important because many factors cause student outcomes (such as test scores, attendance rates, and grade promotion) to differ from school to school and from year to year. Rigorous comparison should determine how much variation was caused by a particular intervention — that is, what the impact of the intervention was over and above what would have occurred without the intervention. While a random assignment design — considered “the gold standard” of research evaluation — was not feasible for this study, the analytic approach used here includes a combination of two particularly strong quasi-experimental evaluation methods: an interrupted time series analysis and a comparison schools technique. The study relies on the strengths of each method to address the limitations that each one might have alone.

In this report, two interrupted time series analyses are performed. The first compares student performance in Talent Development schools with the performance of similar students in the same schools prior to Talent Development’s implementation. The difference between performance levels in the two groups is referred to as a “deviation from the baseline.” Many factors — some related to Talent Development and some not — may have contributed to the deviations from the baseline that emerged through this evaluation. In order to account for the factors that were unrelated to Talent Development, a second interrupted time series analysis was conducted for non-Talent Development schools, a group of comparison schools in the same district that have characteristics similar to those of the Talent Development schools. The difference between the deviations from the baseline in the Talent Development schools and the deviations from the baseline in the non-Talent Development schools represents the estimated impact of Talent Development.3

Key Impact Findings

The eighth grade marks the culmination of students’ middle school experiences and the start of a critical transition period, and eighth-graders’ engagement and performance levels are key indicators of their readiness for the challenges of transitioning successfully to high school. For these reasons, the impact analysis in this report focuses first on outcomes for eighth-grade students. We also focus there because the model’s estimated impact on the engagement and performance of eighth-grade students represents Talent Development’s cumulative effect on the middle school experience. In other words, in the first year of Talent Development’s implementation, eighth-grade students will have experienced only one year of the model; in the third year, some eighth-graders will have been exposed to the model in the sixth, seventh, and eighth grades.

The findings for the study cover a five-year follow-up period. Because all six early-implementing schools had been using Talent Development for at least three of the follow-up years, the three-year follow-up results are the most reliable. The four-year follow-up results are based on the experiences of four of the early-implementing schools, and the five-year follow-up results are based on the experiences of just two of the schools. This means that, even though some of the impacts in Years 4 and 5 are promising, they may not be statistically significant because they are based on a smaller sample of schools. Years 4 and 5 of the follow-up period are important to the story of Talent Development, however. Implementation may need two or more years to gain enough traction to produce significant results, and eighth-graders in these later years may have benefited from cumulative years of exposure to Talent Development.

• Talent Development had a positive impact on math achievement for eighth-grade students, which emerged in the third year of implementation and then strengthened during the next two years in the schools for which data are available.

Eighth-grade math achievement in both Talent Development and non-Talent Development schools improved during the follow-up period, relative to a common baseline period. Improvements in the Talent Development schools began to outpace those in the comparison schools in Year 3. For example, during a three-year baseline period, eighth-grade students in the Talent Development schools scored at about the 23rd Normal Curve Equivalent (NCE) on the state standards assessment.4 Students in the comparison schools scored at about the 24th NCE during the same period. In Year 3, average math scores improved to the 29th NCE in the Talent Development schools (a deviation of 6 NCE points), compared with the 28th NCE in the comparison schools (a deviation of 4 NCE points). The difference of 2 points in deviations from the baseline is statistically significant and represents the impact of Talent Development. Among the schools for which data are available, this impact on math NCE scores grew to 4 points in both Years 4 and 5.

More meaningful, perhaps, is Talent Development’s impact on reducing the percentage of eighth-grade students who scored in the bottom quartile for the state as a whole. In the pre-Talent Development period, about 83 percent of students in Talent Development schools and 81 percent of students in non-Talent Development comparison schools fell into this category. By Year 3, this percentage dropped to 72 percent in the Talent Development schools (an 11 percentage point deviation) and to 74 percent in the comparison schools (a 7 percentage point deviation). The difference of 4 points in deviations from the baselines represents the impact of Talent Development. Although this impact is not statistically significant, it is part of a clear trend that grew over time to produce impacts of 12 percentage points in Year 4 and 11 percentage points in Year 5 among the schools for which data are available.5

• Talent Development produced an inconsistent pattern of impacts on eighth-grade reading achievement over the follow-up period.

Talent Development’s impact on reading achievement was much less consistent than its impact on math achievement. The analysis found statistically significant and positive impact estimates for eighth-grade reading achievement in the second year of Talent Development implementation — an improvement of almost 3 NCEs on average reading achievement and a reduction of nearly 6 percentage points in the percentage of eighth-graders scoring in the bottom quartile on the state reading assessment. However, these impacts dropped in Years 3 and 4 and then rose again in Year 5 for the two schools for which data are available. Only the impacts in Year 2 are statistically significant.

• In general, Talent Development schools modestly outpaced their comparison schools on eighth-grade attendance.

During most years of follow-up, average eighth-grade attendance rates improved in both Talent Development and non-Talent Development comparison schools. Statistically significant impact estimates of about 2 percentage points are found in Years 2 and 3. Overall attendance rates averaged between 85 percent and 88 percent in both sets of schools during the five years of follow-up.

Talent Development did not produce a consistent pattern of impacts, positive or negative, on seventh-grade math achievement, reading achievement, or attendance rates during the five-year follow-up period.

During most of the follow-up period, there was no systematic change, relative to the baseline period, in test scores or attendance rates among seventh-graders in the Talent Development schools. In some years, the Talent Development schools exhibited marginal improvements in math achievement, reading achievement, or attendance rates. In other years, there was no change, or even a slight decline in outcomes. These patterns were virtually the same for the non-Talent Development comparison schools. Overall, therefore, Talent Development did not produce impacts, positive or negative, on most of the test-score and attendance measures that were examined for seventh-graders. The one exception to this occurred in math problem-solving for seventh-graders. Here, during the last two years of the follow-up period, improvements in test scores for the Talent Development schools outpaced the more modest improvements in those scores for students in the non-Talent Development schools.

Sizing Up the Results

Taken together, what do these findings suggest about the Talent Development Middle School model? Two important patterns emerge: (1) The most prominent impacts occurred in mathematics achievement among eighth-grade students; and (2) the strength of impacts seems to be associated with the timing and intensity of Talent Development’s implementation.

The most significant impacts were found for math achievement among eighth-grade students and were particularly strong in the later years of implementation. This pattern may reflect a combination of factors: Eighth-graders in these later years may have benefited from cumulative years of exposure to Talent Development, and implementation may need two or more years to gain enough traction to produce significant results. Also, the development of math impacts before impacts in reading seems to be consistent with the nature of the curricular materials and teacher training in math, which focused on grade-specific content units and were readily transferable to classroom practice.

The presence of Talent Development impacts seems to correspond with whether and when key components of the model were implemented. The components of the model were phased in over three years in the six schools that are the focus of this report. The impact analysis shows that improvements in student achievement, at least in math, began to emerge in Years 3 and 4 of implementation.

Although the early impact findings in this report should be considered preliminary — because this study focuses on only six middle schools with three to five years of follow-up data — they are encouraging, particularly for math achievement among eighth-grade students. The magnitude of the impacts reported here is, by traditional research standards, considered to be small to moderate.6However, the impacts are comparable to achievement impacts found in rigorous evaluations of other notable models of comprehensive school reform and to impacts found in the Tennessee class-size experiment.7Finally, it is possible for small-to-moderate effect sizes to have substantial educational significance. For example, if all 38 nonselective middle schools in the district attained the most promising impacts on math achievement described in this report, more than 1,200 eighth-grade students could move out of the bottom quartile in math achievement each year.

Next Steps

Overall, the findings in this report suggest that the Talent Development Middle School model has positive and significant impacts on certain measures, particularly when key components have been adequately implemented. This may offer promise that the model will have positive and significant impacts on other outcomes in the future, but more data collection and analysis need to be done.

A subsequent report from MDRC on the Talent Development Middle School model will track outcomes for two more years of implementation in the six early-implementing schools and for three years of implementation in the five later-implementing schools in an effort to answer remaining questions, including:

• Will the improvements in eighth-grade mathematics be strengthened over time? Will the impacts continue to be sustained in early-implementing schools, and will those impacts eventually accrue in later-implementing schools and for students in other grade levels?



• Will a more consistent pattern of impacts on eighth-grade reading achievement, and on seventh-grade math and reading achievement, emerge in later follow-up years?



• Will improvements in achievement during middle school years translate into students’ greater persistence in high school and their eventual graduation?

The upcoming report, due in 2005, will be produced in the context of a range of comprehensive school reform research sponsored by the U.S. Department of Education. Taken together, this research has the potential to deliver a powerful message to policymakers, researchers, and practitioners about what interventions help to improve student performance and attendance in low-performing secondary schools.

Notes

1 In order to preserve the anonymity of the subjects in this study, the report refers to the participating school district as “the district.” The district includes 38 nonselective schools. The term “nonselective” refers to schools that typically enroll students from a nearby neighborhood and do not require them to meet academic or other performance standards for admission.

2 James J. Kemple and Corinne M. Herlihy, The Talent Development High School Model: Context, Components, and Initial Impacts on Ninth-Grade Students’ Engagement and Performance (New York: MDRC, 2004).

3 It should be noted, however, that even this combination of approaches may not control for all factors that may confound causal inferences about the effects of Talent Development on student performance. For example, the analytic approach may not account for systematic differences in school leadership’s motivation to undertake a school change process and the influence that that may have had on both school functioning and student achievement, even without Talent Development’s components and supports.

4 The Normal Curve Equivalent (NCE) is a way of measuring where a student falls along the normal curve for the statewide population of test-takers. The NCE score ranges from 1 to 99 and has a statewide average of 50, which indicates that a student is performing at grade level. Less than 20 percent of students across the state have NCE scores below 30.

5 It is important to note that, for the first three years of Talent Development implementation, the pattern for the schools with data from Years 4 and 5 is the same as that for the other schools. This suggests that the findings in Years 4 and 5 are not simply driven by changes in the sample of schools included in the analysis that occurred after Year 3.

6 J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed (Hillsdale, NJ: Lawrence Erlbaum Associates, 1998); Howard S. Bloom, Sandra Ham, Laura Melton, and Julieanne O’Brien, Evaluating the Accelerated Schools Approach: A Look at Early Implementation and Impacts on Student Achievement in Eight Elementary Schools (New York: MDRC, 2001); Mark Lipsey, Design Sensitivity: Statistical Power for Experimental Research (Newbury Park, CA: Sage Publications, 1990).

7 Geoffrey D. Borman, Gina M. Hewes, Laura T. Overman, and Shelly Brown, “Comprehensive School Reform and Achievement: A Meta-Analysis,” Review of Educational Research 73 (2): 125-230; Jeremy D. Finn and Charles M. Achilles, “Tennessee’s Class Size Study: Findings, Implications, Misconceptions,” Educational Evaluation and Policy Analysis 21 (2): 97-109.