Project Graduation Really Achieves Dreams (GRAD) is an ambitious education reform initiative designed to improve academic achievement, high school graduation rates, and rates of college attendance for low-income students. It is an unusual reform model in that it intervenes throughout an entire "feeder pattern" of elementary and middle schools that send students into each Project GRAD high school. The initiative recognizes that high schools inherit problems that have arisen earlier in the education pipeline, making it essential to improve both elementary and secondary schools in order to increase the rates of high school graduation, college-going, and college graduation.

Project GRAD schools at all levels build support in the community for school improvement and college attendance, implement a classroom management program, provide students with access to needed social services, and receive special support from local Project GRAD organizations. To help students arrive at middle and high school better prepared academically, Project GRAD elementary schools implement specific reading and math curricula, along with enhanced professional development for teachers. At the high school level, Project GRAD's model assumes that better-prepared students would come from the Project GRAD feeder schools, would benefit from special academic counseling and summer academic enrichment in high school, and would qualify for a scholarship to attend college, which is the "cornerstone" of the Project GRAD reform.

This report describes the effects of Project GRAD on student progress at three high schools in Houston (the initiative's original site) and at high schools in two other school districts (Columbus, Ohio, and Atlanta, Georgia). MDRC — a nonprofit, nonpartisan research organization — conducted a third-party evaluation to determine the effects of Project GRAD by comparing the changes in student outcomes at Project GRAD schools with changes at similar, non-Project GRAD schools in the same districts. (A companion report discusses findings for Project GRAD elementary schools.)¹ In general, Project GRAD student outcomes are tracked from the implementation of the first components of the model at each site until the 2002-2003 school year. The key findings of this report are:

• At Jefferson Davis High School in Houston, the initiative's flagship school, Project GRAD had a statistically significant positive impact on the proportion of students who completed a core academic curriculum on time — that is, received an average grade of 75 out of 100 in their core courses; earned four credits in English, three in math, two in science, and two in social students; and graduated from high school within four years.



• As Project GRAD expanded into two other Houston high schools, these positive effects on students' academic preparation were not evident. Student outcomes at the newer Project GRAD high schools improved, but generally this progress was matched by progress at the comparison high schools.



• Improvements in graduation rates at the three Project GRAD Houston high schools were generally matched by improvements in graduation rates at the comparison schools.



• Looking at early indicators of student success, the initial Project GRAD high schools in Columbus and Atlanta showed improvements in attendance and promotion to tenth grade that appear to have outpaced improvements at the comparison schools, although the differences are only sometimes statistically significant.

What Is Project GRAD and How Was It Implemented?

Project GRAD originated in 1993 at Jefferson Davis High School in Houston, Texas, when business leaders sought to increase the rate of college-going among the school's mostly low-income students by offering scholarships to students who qualified. When this offer did not generate a significant increase in high school graduation and college enrollment, Project GRAD was expanded to include school improvement efforts at the associated middle and elementary schools that fed into the high school.

A complex, multilayered initiative, Project GRAD includes a set of core components for all the schools in a feeder pattern as well as components for the schools at each level, as described below. It is important to note that these components were phased in over time at each site.

Components at Project GRAD High Schools

At the high school level, Project GRAD's strategy relies on the demand push for change from better-prepared students coming from Project GRAD feeder schools and the opportunity pull toward success provided by the promise of the Project GRAD scholarship. Unlike its intervention in the lower grades, Project GRAD at the high school level does not modify the curricula in school, provide substantial professional development for teachers, change the way instruction is offered in the classroom, nor directly address skill deficits of entering students. Instead, it offers two particular components that encourage planning for college, as well as three components that seek to create an environment that is conducive to learning:

• Project GRAD college scholarships are provided to students who have a cumulative 2.5 grade point average, graduate within a four-year time period, complete a recommended college preparatory curriculum, and participate in two summer institutes. Scholarship amounts and criteria vary slightly by site but usually average $1,000 to $1,500 each year during the four years of college. Each Project GRAD high school has a scholarship coordinator who provides counseling, tutoring, and college admission preparation.



• Summer institutes provide an opportunity for qualifying Project GRAD students to experience a college campus-based program taught by college faculty and to enhance their academic skills. The activities vary by site but typically include reading, writing, math, science, enrichment, and remedial activities. The institutes usually consist of four to six hours of instruction and related activities per day for four to six weeks.



• Parental and community involvement to engage parents and the community in the work of the schools, build awareness of the opportunity to attend college, and support the learning of students. At the high school level, annual Walks for Success are conducted, in which principals, teachers, Project GRAD staff, and community leaders visit students' homes to explain the program and encourage parents and students to participate.



• Social services and academic enrichment through one of two programs — Communities In Schools (CIS) or the Campus Family Support (CFS) Plan (developed by Project GRAD) — which bring additional social services, academic activities, and volunteers into Project GRAD schools to address issues that students and their families face and to build commitment to academic success.



• Classroom management programs developed by Consistency Management & Cooperative Discipline (CMCD)^SM that are designed to produce orderly classrooms focused on learning, by promoting student responsibility and self-discipline and positive relationships among students, teachers, and other adults in the school.

Components at Project GRAD Elementary Schools

During the time covered by this study, Project GRAD influenced instruction at elementary schools directly by putting in place reading and mathematics curricula. Most Project GRAD sites used Success for All (SFA), a nationally recognized reading program that focuses on the key elements of reading instruction during concentrated instructional time (90 minutes each day), with the goal of bringing students to grade-level reading by third grade. Math Opportunities, Valuable Experiences, Innovative Teaching (MOVE IT^TM Math) was Project GRAD's recommended math curriculum. It offers elementary school teachers professional development and instructional materials organized around the National Council of Teachers of Mathematics (NCTM) Standards program, involves heavy use of manipulatives to address a wide variety of learning styles, emphasizes daily problem solving, and introduces algebra in the early grades. (Currently, Project GRAD supports whatever reading and math curricula that participating districts adopt.) In addition, Project GRAD elementary schools also offer the three core components focused on parent and community involvement, social services and academic enrichment, and classroom management.

In 1998, Newark, New Jersey, became the first site outside Houston to implement Project GRAD; Columbus, Ohio, and Atlanta, Georgia, followed soon thereafter. Currently, Project GRAD operates in five feeder patterns in Houston and in 12 school districts and 211 schools in eight states across the country, serving more than 131,000 students. To manage and support each Project GRAD initiative, local not-for-profit organizations were established in Houston and the expansion sites.

Five important points should be noted about the implementation of Project GRAD in the three study sites — Houston, Atlanta, and Columbus:

• The schools in this study did implement the core Project GRAD components and followed the general approach set forth in the model. However, the Project GRAD strategy takes time to unfold; in each site, the various components at all schools were rolled out gradually.



• Expansion within the Houston schools and to other school districts stretched the capacity of some program developers to support the model's components and prompted the development of a national organization in 2000 — Project GRAD USA — to sustain implementation efforts and to address implementation issues across sites.



• The model's theory presumes that feeder patterns are sufficiently "self-contained" so that better-prepared students would come to Project GRAD high schools from Project GRAD feeder schools. However, the feeder patterns in the districts studied were more "leaky" than expected, in part as a result of high rates of student mobility and of policies that allowed students to choose the high school they attended.



• This evaluation covers a period in which many efforts were being launched in the districts under study to improve student performance, including reforms specifically focused on high schools. Thus, "business as usual" — as represented by the experiences of the comparison schools — was neither static nor absent reforms.



• The Houston experience provides the clearest opportunity to examine Project GRAD at the high school level, for several reasons: There was longer and stronger implementation; a greater percentage of students entered the high schools with Project GRAD exposure in earlier grades; and — because of Houston's earlier start — there were fewer competing high school reform efforts.

How Was the Evaluation Conducted?

The goal of this evaluation is to understand whether Project GRAD changed the academic outcomes of high school students from what they would have been absent Project GRAD — and, if so, how. To estimate the program's effect on achievement, MDRC used an approach called "comparative interrupted time series analysis," which compares the performance of Project GRAD schools with similar schools that have not implemented the reform. The first step in estimating program impacts with this design is to compare the change at Project GRAD schools in a given student outcome after the schools began implementing Project GRAD with the average outcome during a baseline period, before implementation. This estimate represents how student performance changed in the presence of Project GRAD but does not, by itself, provide a measure of the effect of Project GRAD. The next step is to measure the corresponding change during the same period for similar schools in the same districts that were not implementing Project GRAD. This measurement provides an estimate of how student performance would most likely have changed at the Project GRAD schools had Project GRAD not been implemented. The difference between these two changes is an estimate of the impact of the Project GRAD reform — the effects that can be attributed to Project GRAD.

This evaluation examines a number of student outcomes over varying numbers of years, depending on the site. For Houston, the study tracked attendance rates, test scores, promotion rates, credits earned, graduation rates, and the proportion of students completing a core academic curriculum — for up to ten years at Jefferson Davis High School and its comparison schools and for seven or eight years at the two other Project GRAD high schools and their matched comparisons. For the expansion sites in Atlanta and Columbus, data limitations allowed for only a maximum of three years of follow-up on two outcome measures: attendance rates and promotion rates.

The very nature and complexity of the Project GRAD feeder system intervention, which posits that students would need to be exposed to the program over many years, combined with the limited amount of follow-up in the expansion sites, created a challenging set of conditions for a meaningful evaluation. While the Houston feeder patterns provide a reasonable test of the intervention, the results for Atlanta and Columbus should be treated as more provisional. In addition, because the high school evaluation began concurrently with the implementation of Project GRAD in the expansion site feeder elementary and middle schools, the study was not in a position to capture cumulative effects of students' exposure to the intervention in the earlier grades in those districts. Only in the latter years of follow-up in Houston did this become possible. Finally, findings from the single high schools in Atlanta and Columbus — as well as from Davis High School in Houston — should be interpreted with some caution.

How Did Project GRAD Affect High School Student Outcomes?

The findings from this study — combined with results from the companion report on elementary schools — provide insights into the strengths and limitations that Project GRAD brings to school improvement efforts as well as into the challenges that it still faces. MDRC's elementary school evaluation found that students at Project GRAD schools generally showed as much improvement on high-stakes state achievement tests as students at similar local schools — but without suffering the decline seen in comparison schools on national tests, which measure achievement more broadly. This suggests that Project GRAD can modestly improve the academic achievement of students leaving elementary school — an essential element of its strategy to develop better-prepared students entering high school.

The findings from the high school study tell a more complicated story:

• Focusing on early indicators of high school success (like credits earned and test pass rates in ninth and tenth grades), trends were generally similar and improving at the Project GRAD and comparison high schools in Houston, and any differences in the extent of improvement are not statistically significant.

• At Davis High School in Houston, increases in the percentage of ninth-graders who completed a core academic curriculum on time outpaced increases at the comparison schools. However, this effect did not hold up across the three original Project GRAD Houston high schools as a whole.

• Project GRAD does not appear to have had an independent effect on the percentage of ninth-graders who later graduated from a Houston high school. At both the Project GRAD and the comparison schools, the graduation rate slowly improved over the period of the study, but any differences between schools in the extent of improvement are not statistically significant.

• Neither Atlanta nor Columbus has operated the program enough years to assess its effects on graduation rates or completion of a core academic curriculum, but early indicators suggest that Project GRAD may have had a positive influence on two precursors to graduation.

What Are the Implications of These Findings?

What accounts for this pattern of findings? The evidence points to at least two hypotheses. First, it may be more difficult than expected to quickly improve the academic performance of incoming high school students by intervening in feeder schools — a core tenet of Project GRAD's strategy. The implementation research highlights the fact that the Project GRAD strategy takes considerable time to unfold. In addition, it turns out that the feeder patterns for the high schools in this study were "leaky" — as a result of high rates of mobility and school-choice options, many students in the Project GRAD high schools had not benefited from exposure to the model in elementary or middle school. For example, after ten years of program implementation in the Davis High School feeder pattern, the average pre-high school exposure to Project GRAD for incoming ninth-graders was approximately 4.5 years. In addition, by Year 6 of implementation across the three Houston high school feeder patterns, when ninth-grade students could have had up to five years of Project GRAD exposure in earlier grades, they had had only three years on average. Therefore, a considerable proportion of ninth-grade students in the Project GRAD high schools had not been exposed to Project GRAD in earlier grades, and among those students who had pre-high school Project GRAD exposure, many did not receive the full treatment.

Second, other research has zeroed in on the crucial transitions that students must make as they enter high school, particularly on the central role that completing ninth grade on time plays in a student's eventual completion of high school.² Being "on track" at the end of the first year of high school is a stronger predictor of eventual on-time graduation than a student's entering achievement level is.³ One explanation for the lack of impacts by Project GRAD on most ninth- and tenth-grade measures (and perhaps on graduation rates) could be the initiative's lack of a direct intervention in ninth-grade instruction during the time period covered by the study. On the other hand, while Project GRAD did not produce improvements in graduation rates, its services were able to affect the course-taking of students at Davis High School who were already headed toward graduation. The scholarship requirements, counseling on the best academic preparation for college, and social service supports are likely reasons for the impacts on the academic preparation of graduates at the flagship high school in Houston.

What do these findings mean for the future of Project GRAD? Given Project GRAD's long-view strategy of developing better-prepared students in feeder schools, this evaluation of its high school intervention has come relatively early in its development, particularly in the expansion sites outside its home district of Houston, where high school students would not have had the opportunity to participate in Project GRAD in earlier grades. At the same time, the marketplace in which Project GRAD operates is highly competitive. Local decision-makers, who face intense pressure to improve high schools quickly, are weighing the long-term Project GRAD strategy against other reforms that intervene more directly in the daily educational experience of high schools — many of which have not taken on a comparative study like this to understand their own added value.

One of the key implementation findings of MDRC's study is that Project GRAD is a dynamic organization that has responded to operational lessons and research evidence to modify its strategy over time. In fact, it has already begun to refine its high school approach to address some of the challenges suggested by this evaluation, including the effects of "leaky" feeder patterns, of the relatively slow rollout of the components of the Project GRAD intervention, and of the difficulties of transforming high schools without directly intervening in the classroom.

A recent study identifies "twin pillars" of successful high school reform efforts: personalizing the learning environment and improving instruction.⁴ Project GRAD's high school intervention appears to focus more on the first pillar. To accomplish its goals of improving academic achievement, high school graduation rates, and rates of college attendance for low-income students, Project GRAD may need to make a strategic choice: to intervene directly in classrooms (on its own or in partnership with curricular reformers) or to target high schools where curricular reform is already under way and where Project GRAD's services and scholarship offer would provide added value.

In many ways, Project GRAD's multifaceted strategy was ahead of its time, prefiguring a variety of current reform approaches. The focus on the full span of grades, the connection to postsecondary education, and the need to work above the level of individual schools are now appreciated as important aspects of many district-level reforms. Project GRAD now has the opportunity to build on its strengths, incorporate additional components into its strategy to address its weaknesses, and develop strategic partnerships with other complementary school improvement efforts to create the next generation of its reform model.

Notes

1 Jason C. Snipes, Glee Ivory Holton, and Fred Doolittle, Charting a Path to Graduation: The Effect of Project GRAD on Elementary School Student Outcomes in Four Urban School Districts (New York: MDRC, 2006).

2 Janet Quint, Meeting Five Critical Challenges of High School Reform: Lessons from Research on Three Reform Models (New York: MDRC, 2006); Melissa Roderick, Closing the Aspirations-Attainment Gap: Implications for High School Reform. A Commentary from Chicago (New York: MDRC, 2006).

3 Roderick (2006).

4 Quint (2006).