The Career Academy approach is one of the oldest and most widely established high school reforms in the United States. Career Academies have existed for more than 30 years and have been implemented in more than 1,500 high schools across the country. The durability and broad appeal of the Academy approach can be attributed, in part, to the fact that its core features offer direct responses to a number of problems that have been identified in large comprehensive high schools. Career Academies attempt to create more supportive and personalized learning environments through a school-within-a-school structure. Their curricula combine academic and occupation-related course requirements that aim both to promote applied learning and to satisfy college entrance requirements. Academies establish partnerships with local employers to build sequences of career awareness and work-based learning opportunities for their students.

While the basic organizational features of the approach have remained the same since Career Academies’ inception, the goals and target population have changed. The original Academies were designed primarily to prevent dropping out of high school and to increase preparation for work among students who began high school at high risk of school failure. There is now widespread agreement that Career Academies should seek to prepare students for both work and college, and that they should include a broad cross-section of students, including those who are highly engaged in school.

There has been a great deal of research on the Academy approach. Nevertheless, previous studies have been unable to determine reliably whether differences between Academy students’ high school experiences and outcomes and those of other students result from the Academy itself or from the program’s student targeting or its selection practices. Further, little is known about the relative effectiveness of Academies for different groups within the broad cross-section of students they now serve. There have also been few opportunities to explore the extent to which different contexts and implementation strategies may influence the effectiveness of the Academy approach.

This evaluation has demonstrated the feasibility and benefits of using a large-scale, multi-site random assignment research design to determine the impact of Career Academies on student outcomes. A rarity in education research, this design provides a uniquely rigorous way of comparing the performance of students who had access to an Academy with the performance of a truly comparable group of students who did not have access to the programs. In order to address a number of key policy issues for Academies and related high school reforms, this report focuses on three questions:

• To what extent does the Career Academy approach alter the high school environment in ways that better support students academically and developmentally?

• To what extent does the Career Academy approach change educational, employment, and youth development outcomes for students at greater or lesser risk of school failure?

• How do the manner and context in which Career Academy programs are implemented influence their effects on student outcomes?

This report marks a milestone in the Manpower Demonstration Research Corporation’s (MDRC) 10-year Career Academies Evaluation, which is being supported by the U.S. Departments of Education and Labor and by 17 private foundations and organizations. The report focuses on over 1,700 students who had applied for one of nine Career Academies participating in the evaluation.¹ The participating Academies were able to implement and sustain the basic features of the approach and have adapted to a wide range of local needs and circumstances. They include a range of technical, service-oriented, and business-related career themes and are located in small cities and large urban school districts. Students in the study sample were identified when they were in 8^th or 9^th grade, and this report follows them through the end of their scheduled 12^th-grade year.

Findings in Brief and Policy Implications

Throughout this report, the term impact refers to differences between outcomes for students who were randomly selected to enroll in an Academy and those of students who also applied but were not selected to enroll. Academy applicants were assigned to these groups at random, so there were no systematic differences in the characteristics or school experiences of the applicants initially. Thus, subsequent differences in outcomes for the two groups reflect increases or decreases caused by the Career Academies. Following is a summary of the key findings from the report.

• The Career Academies in this study increased both the level of interpersonal support students experienced during high school and their participation in career awareness and work-based learning activities.

• The Career Academies substantially improved high school outcomes among students at high risk of dropping out. For this group, the Academies reduced dropout rates, improved attendance, increased academic course-taking, and increased the likelihood of earning enough credits to graduate on time.

• Among students least likely to drop out of high school, the Career Academies increased the likelihood of graduating on time. The Academies also increased vocational course-taking for these students without reducing their likelihood of completing a basic core academic curriculum.

• In sites where the Academies produced particularly dramatic enhancements in the interpersonal support that students received from teachers and peers, the Career Academies reduced dropout rates and improved school engagement for both high-risk and medium-risk subgroups (about 75 percent of the students served). Academies that did not enhance these supports actually increased dropout rates and reduced school engagement for some students.

• The Career Academies did not improve standardized math and reading achievement test scores.

• When the findings are averaged across the diverse groups of students in the full study sample, it appears that the Career Academies produced only slight reductions in dropout rates and modest increases in other measures of school engagement. These aggregated findings, however, mask the high degree of variation in effectiveness among different groups of students and across the different program contexts.

The findings that have emerged from the Career Academies Evaluation to date suggest the following implications for policies aimed at improving high schools and helping students prepare for the transition from high school to further education and work.

• Career Academies provide a well-defined approach to creating more supportive high school environments and increasing students’ exposure to career awareness and work-based learning activities.

• Among students who are most at risk of dropping out of high school, Career Academies are an effective means of preventing dropout, increasing school engagement, and helping students acquire the credentials they need to graduate and prepare for post-secondary education.

• Career Academies should continue to serve a heterogeneous population of students. The pervasive positive impacts for students at high risk of dropping out may derive, in part, from exposure to a highly engaged peer group who, on balance, also benefit from exposure to several key dimensions of the Academy experience.

• If Career Academies do not complement their career-related curriculum and work-based learning activities with strong interpersonal and academic supports, they risk reducing school engagement for some students. A highly structured school-within-a-school organization can create a necessary set of conditions for providing these supports.

• Career Academies should build on the effective organizational enhancements they bring to high school reform efforts if they are to improve academic achievement as measured by most standardized tests currently in use. Promising approaches may involve aligning Career Academy curricula with high standards and providing teachers with the incentives and capacity to deliver on such standards.

The above results capture the effects that the Career Academies have had on students through the end of their scheduled 12^th-grade year. The evaluation does not yet include information about the rates at which these students actually graduated from high school and whether the dropouts eventually returned to high school or pursued an alternative credential. The next phase of this evaluation will include this information and will follow the students in the study sample for four additional years as they make the transition from high school to post-secondary education and employment opportunities.

The remainder of this Executive Summary describes the Career Academy approach in greater detail, including its history, and discusses the current policy context and previous research in the Career Academies Evaluation. It then describes the results of the evaluation and their implications for policy and practice.

The Career Academy Approach

The Career Academy approach is distinguished by three core features that offer direct responses to several problems that have been identified in high schools, particularly those serving low-income communities and students at risk of school failure. First, a Career Academy is organized as a school-within-a-school in which students stay with a group of teachers over three or four years in high school. Such arrangements are often referred to as "small learning communities." The aim is to create a more personalized and supportive learning environment for students and teachers. Second, a Career Academy offers students a combination of academic and vocational curricula and uses a career theme to integrate the two. Third, a Career Academy establishes partnerships with local employers in an effort to build connections between school and work and to provide students with a range of career development and work-based learning opportunities. This definition of an Academy is now commonly accepted and was reviewed by a broad range of researchers, policymakers, and practitioners who have worked closely with Career Academies.

The initial Career Academies of the 1970s and 1980s were primarily vocational education programs targeted at students who appeared to be at high risk of dropping out of high school. The central goals of these early programs were to keep students engaged in school, provide them with work-related learning experiences both in the classroom and on the job, and establish clearer pathways between high school and post-secondary employment.

Since the late 1980s, there has been a shift in the primary goals and target population of most Career Academies. In particular, there is now wide agreement that the Career Academy approach should be explicitly distinct from traditional vocational education by seeking to prepare students for both work and college. Vocational education, as defined in federal law and through its historical legacy, has been directed at preparing young people for occupations that do not traditionally require advanced degrees. In line with what has been called "the new vocational education," Career Academies now seek to include a broad range of students and to combine a rigorous academic curriculum with exposure to extensive information about an industry both in the workplace and in the classroom. The career theme is used to integrate curricula and provide exposure to a broad array of careers in a given field and does not typically focus on preparing students for jobs in those areas.

The 1990s have seen extraordinary growth in the number of Career Academies around the country. There are estimated to be more than 1,500 Career Academies nationwide, representing nearly a 15-fold increase in approximately 10 years; many more Academies are in the planning stages. Much of this growth can be traced to the increasing number of national, state, and district Academy support networks. Although most Career Academies share the approach’s basic elements, the Academy model has been adapted to a wide range of local needs and circumstances, resulting in a variety of versions that emphasize different features over others.

The expansion of Career Academy target populations and goals and the rapid growth in the number of Academies have raised several questions about how the Academy approach may be affecting students’ performance in high school. How well does it meet the needs of a much broader range of students than it was initially designed to serve? Is the Academy approach more effective under some conditions than under others? Which features of the Academy model make the most difference for students? MDRC’s Career Academies Evaluation is intended to shed light on these and other questions.

The Current Policy Context

This report is being released at a time when education policymakers and practitioners are pursuing a number of far-reaching strategies for improving American high schools. Many of these strategies include principles embedded in the Career Academy approach, while others include the Career Academy model as an explicit component. Two key policy initiatives are particularly relevant.

First, states, school districts, and employers are now looking for strategies and approaches that can build on the foundation established by the School-to-Work Opportunities Act (STWOA) of 1994. STWOA was aimed at enhancing the relevance and rigor of school- and work-based learning and at creating clearer pathways between high school and post-secondary education and careers. This was to be done primarily through partnerships between schools and local employers. STWOA specifically identifies Career Academies as a "preferred approach" to creating such partnerships and implementing the principles embedded in the legislation. STWOA is scheduled to sunset in 2001 — adding urgency to these efforts and heightening interest in concrete evidence of the potential payoff of Career Academies.

Second, the U.S. Department of Education has committed itself to several initiatives aimed specifically at addressing problems that are unique to high schools. Many of these initiatives are being supported under the Comprehensive School Reform Demonstration developed within the Office of Educational Research and Improvement (OERI) and the New American High Schools established by the Office of Vocational and Adult Education (OVAE). Although most of the strategies that are being developed involve comprehensive reforms of entire high schools, many include key elements of the Academy approach, including the creation of a small school-within-a-school, integration of academic and vocational curricula, and the establishment of partnerships with employers and other organizations in the community.

The findings presented in this report will shed light on the extent to which the Career Academy model, and some of the high school reform approaches embedded in the model, can achieve the goals sought by their proponents.

The Career Academies Evaluation

In 1993, MDRC began an evaluation of the Career Academy approach as it had been defined in previous research and implemented in a broad range of settings across the country. The evaluation’s primary goal is to provide policymakers and educators with reliable evidence about the impact that Career Academies have on students’ success in high school and their transition to further education and the labor market. The evaluation will also offer lessons about how Career Academies operate and are sustained and about the pathways through which Academies affect student engagement and performance in school.

The current report is based on information collected over a six-year period and focuses on nine high schools and their Career Academies.² Each of the Academies had established the basic Career Academy components described earlier: a school-within-a-school organization, an integrated academic/vocational curriculum, and employer partnerships. Moreover, this combination of features was not available elsewhere in the participating high schools.³ These Academies were selected to include school districts and high schools reflecting the typical conditions (large urban centers and small cities) under which Career Academies have been implemented across the country. MDRC was specifically interested in Academies serving a broad range of students, including those who were perceived to be at risk of not succeeding in the regular high school environment. Most of the school districts in the evaluation are large and enroll substantially higher percentages of African-American and Hispanic students than school districts nationally. On average, these school districts have higher dropout rates, higher unemployment rates, and higher percentages of low-income families.

The Career Academies Evaluation is a rarity in the field of education research in that it has demonstrated the feasibility and benefits of implementing a large-scale, multi-site random assignment research design within an ongoing high school program. This was made possible because each of the Career Academies in the study received applications from approximately twice as many students as it was able to serve. This reports focuses on a sample of 1,764 students (referred to in this report as the study sample) who applied for one of the Career Academies selected for the study. Of these, 959 students were randomly assigned to the program group (referred to in this report as the Academy group) and were accepted for admission to the Academies. The remaining 805 students were randomly assigned to a control group (referred to in this report as the non-Academy group) and were not invited to participate in the Academies, although they could choose other options in the high school or school district.

The random assignment process ensured that there were no systematic differences between the two groups of students in terms of their observable and unobservable background characteristics, prior school experiences, and initial motivation and attitudes toward school. Any systematic differences that subsequently emerged between the groups can be attributed with confidence to differences in their access and exposure to the Career Academies.

MDRC obtained data for this report from four sources:

• school transcript records, including information about students’ daily attendance rates, credits earned toward graduation, and course-taking patterns;

• student surveys that asked a wide range of questions about school experiences, employment and work-related experiences, extracurricular activities, preparation for college and post-secondary jobs, and plans for the future;

• standardized math computation and reading comprehension tests administered to 490 students from the study’s sample (from both the Academy and the non-Academy groups) at the end of their 12^th-grade year;⁴ and

• qualitative field research conducted throughout the evaluation to document Academies’ characteristics, local contexts, staff, students, and employer partners.

Students in the study sample were identified at the end of 8^th or 9^th grade. This report follows them for three or four years through the end of their scheduled 12^th-grade year, until just before they would have graduated from high school. The primary focus of the report is on outcomes measured at the end of students’ scheduled 12^th-grade year. Unless otherwise noted, the impact findings discussed in the report are statistically significant, indicating that one may have a high degree of confidence that measured differences in outcomes between the Academy and the non-Academy groups were not a result of chance.

Previously Reported Findings on How Career Academies Changed Students’

High School Experiences

The previous reports from this evaluation examined the extent to which Career Academies changed the high school environment as indicated by differences between Academy and non-Academy students’ experiences during high school.⁵ Following is a brief overview of key findings from these reports.

During their early years in high school, Academy students received more support from their teachers and peers than did their counterparts in non-Academy high school environments. For example, compared with their non-Academy peers, Academy students reported that their teachers had higher expectations of them and that teachers provided them with more individualized attention. Moreover, compared with their non-Academy counterparts, Academy students were more likely to report that their classmates were highly engaged in school and that they had many opportunities to collaborate with their peers on school and work-related projects.

• Career Academies increased students’ participation in career awareness and work-based learning experiences during high school.

Academies also increased students’ exposure to work-related learning experiences in school and in the workplace. Academy students were more likely than their non-Academy peers to be exposed to career-related themes or activities in the classroom. They were also more likely to participate in career-related activities such as job-shadowing or field trips. Finally, Academy students were more likely than non-Academy students to participate in a planned program of work experience and to have high-quality work-based learning experiences during high school.

• The Career Academies in this evaluation demonstrated their capacity to attract large numbers of applicants and to include students with a wide range of demographic and education characteristics.

The growth of the Career Academy movement has been accompanied by questions about whether the programs can and should serve a broad range of students and about which students benefit most from them. Reflecting the shift in goals and target populations of Career Academies nationwide, the programs in this evaluation attracted a mix of students including those at risk of dropping out of high school or failing academically as well as students who had done well in school. Most of the students in the study sample are from minority backgrounds — 56 percent are Hispanic, and 30 percent are African-American — reflecting the racial and ethnic make-up of their communities. Also, more than one-third of the students came from single-parent households, and about one-quarter indicated that their families received public assistance. At the same time, just under half the students reported that both their parents were employed, and about one-third reported that at least one parent had attended college.

• Approximately 88 percent of the students selected for admission to a Career Academy actually enrolled in the programs, and 58 percent of those selected remained in an Academy throughout high school.

Of the students who were initially selected for admission, about 12 percent chose not to enroll, and another 30 percent enrolled in the programs and then left before the end of their 12^th-grade year. It is unclear how much of this attrition could possibly be controlled or avoided by the Career Academies. Student mobility and early dropout are common in most urban school districts, and they were reasons for attrition from the Academies in this evaluation. Just under one-quarter of the students who never enrolled in an Academy or who enrolled and then left reported that they did so because their families moved and they had to transfer to other schools. Another 20 percent reported that they were asked to leave the programs or dropped out of high school altogether. The remaining students (approximately 55 percent of those who were not enrolled in an Academy in the 12^th grade) chose not to enroll or chose to leave the programs. The most common reasons students gave for not enrolling or for not remaining enrolled in an Academy were that they wanted to enroll in another program, they lost interest in the occupational area, or they did not think the Academy would help them get into a good college.

The Impact of Career Academies on Student Outcomes

The central theme that has emerged from the Career Academies Evaluation thus far is that the Academies affected the outcomes for students who were likely to drop out of high school much more than they affected the outcomes for other students. When the results are averaged across the diverse groups of students they serve, it appears that the Academies produced only slight reductions in dropout rates, modest improvements in students’ progress toward high school graduation, and increases in career-related course-taking and involvement in positive youth development activities. These aggregate results mask a high degree of variation in the Career Academies’ potential to make a difference and in the actual differences they made for some students.

To assess this variation in impacts, the study sample was divided into three subgroups based on selected background characteristics and prior school experiences. These characteristics were chosen as indicators of students’ engagement in school at the time they applied for an Academy and as factors associated with the likelihood of their eventually dropping out of school. (See Table ES-1 for a list of the background characteristics used to define these subgroups.) Just over one-quarter of the students were classified as being in the high-risk subgroup and reflected the combination of characteristics associated with the highest probability of dropping out among those in the non-Academy group. Approximately one-quarter of the students in the sample were classified as being in the low-risk subgroup and reflected the combination of characteristics associated with the lowest probability of dropping out among those in the non-Academy group. The remaining students (approximately half the sample) were defined as being in the medium-risk subgroup.

Because each of the characteristics used to define the subgroups was measured before students were randomly assigned to the two main study groups, there are no systematic differences in observed background characteristics between Academy and non-Academy groups within each of the three risk subgroups.⁶ The following sections summarize the impact findings for these subgroups.
Career Academy Impacts for Students in the High-Risk Subgroup

As shown in Table ES-1, students in the high-risk subgroup entered the study with background characteristics and prior school experiences indicating that they were disengaged from school. More than half had failed courses during the 9^th grade, and about one-third could be classified as chronic absentees (having attendance rates lower than 85 percent). Most of these students had low grade point averages (2.0 or lower), and over 40 percent had been held back in a previous grade (as indicated by being overage for their current grade).

Figure ES-1 provides a summary of the impact findings for students in the high-risk subgroup. It shows first that, without access to an Academy, a high percentage of non-Academy students in the high-risk subgroup had become even more disengaged from school. In all, 32 percent of these students dropped out of high school, and only 26 percent had earned sufficient credits to meet the district’s graduation requirements by the end of their scheduled 12^th-grade year.

Figure ES-1 shows that the Career Academies produced substantial improvements in many educational outcomes for students in the high-risk subgroup. In particular, while 32 percent of the non-Academy students in the high-risk subgroup dropped out of high school, 21 percent of the Academy students did so. This 11 percentage point difference represents a one-third reduction in the dropout rate for the non-Academy group. This can be classified as a particularly large reduction in dropout rates. Reductions of this magnitude are rare for school-based interventions.

The Academies also significantly increased average attendance throughout high school for students in the high-risk subgroup (not shown in Figure ES-1). Average attendance rates throughout high school were approximately 76 percent for students in the non-Academy group, compared with 82 percent for students in the Academy group. This amounts to an additional 11 days of school per year over four years.

Moreover, while 26 percent of the high-risk non-Academy group had earned enough credits to meet district graduation requirements, 40 percent of the students in the Academy group did so (an increase of over 50 percent beyond the non-Academy group average). This suggests that, besides improving attendance and preventing students from dropping out, the Academies helped a significant portion of the high-risk subgroup to make up enough of the initial gap in credits earned to meet the district’s graduation requirements three year later.

Also, as indicated by the third set of bars in Figure ES-1, the Academies doubled the percentage of students in the high-risk subgroup who completed a basic core academic curriculum (four English courses, three social studies courses, two math courses, and two science courses). At the same time, students in the Academy group were significantly more likely than their non-Academy counterparts to complete three or more career-related or vocational courses.

The fifth set of bars in Figure ES-1 indicates that the Academies increased the percentage of students in the high-risk subgroup who reported that they had submitted an application to a two-year or four-year college by the end of their 12^th-grade year. In particular, 35 percent of students in the high-risk non-Academy group reported submitting a college application, compared with 51 percent in the Academy group. Academy students in the high-risk subgroup were also more likely to report taking the SATs or ACTs (not shown in the figure).

Finally, the last two sets of bars in Figure ES-1 indicate that Academies did not produce a systematic change in involvement in positive youth development activities or in negative risk-taking behaviors. Positive youth development activities included participation in community volunteer work, receiving recognition for participation in academic or extracurricular activities, and receiving an academic award. Negative risk-taking behaviors included coming to school on drugs, becoming a parent, being expelled from school, and being arrested. Although the differences between the groups shown in Figure ES-1 were not statistically significant, they indicate trends in a positive direction.

Career Academy Impacts for Students in the Low-Risk Subgroup

Figure ES-2 presents a summary of the impact findings for students in the low-risk subgroup. The results for the non-Academy group indicate that, even without access to the Academy intervention, these students appear to be unlikely to disengage from school. For example, as the first set of bars in Figure ES-2 illustrates, only 3 percent of the non-Academy students in the low-risk subgroup dropped out of high school before the end of 12^th grade. Almost the same percentage of Academy students (2 percent) dropped out.

The second set of bars in Figure ES-2 indicates that the Academies increased the percentage of students in the low-risk subgroup who earned sufficient credits to meet their district’s graduation requirement. The figure shows that 86 percent of the Academy students met their districts’ graduation requirement, compared with 75 percent of the students in the non-Academy group.

Also, while approximately equal percentages of Academy and non-Academy students in the low-risk subgroup completed a basic core academic curriculum, the Academies significantly increased the percentage who completed at least three career-related or vocational courses. It should be noted than many students in the low-risk non-Academy group were likely to be enrolled in their high school’s college preparatory programs and courses. Thus, the Academies increased vocational course-taking for the low-risk subgroup while enabling students to complete as many core academic courses as their non-Academy peers.

The fifth set of bars in Figure ES-2 indicates that the Academies reduced the percentage of the low-risk subgroup who reported that they had submitted an application to a two-year or four-year college by the end of their 12^th-grade year. Among these students, 79 percent of the non-Academy group reported submitting a college application, compared with 71 percent of the Academy group. Although not shown in the figure, this occurred despite the fact that Academy and non-Academy students were equally likely to have taken the SATs and ACTs. In addition, over 85 percent of students in both low-risk groups reported that they had conducted at least a modest amount of research on college options during their 12^th-grade year.

Figure ES-2 also shows that Academy and non-Academy students in the low-risk subgroup were equally likely to pursue post-secondary employment opportunities. Further analyses indicated that the Academies do not appear to have induced students to pursue post-secondary employment opportunities instead of either a two-year or four-year college. Further follow-up is needed to determine the effects that the Career Academies may have had on actual college enrollment and employment during the years following high school graduation. This will be explored further in subsequent reports from the Career Academies Evaluation.

Finally, the last two sets of bars in Figure ES-2 show that the Academies did not produce statistically significant changes in the low-risk subgroup’s involvement in positive youth development activities or risk-taking behavior.

Career Academy Impacts for Students in the Medium-Risk Subgroup

• On average, the Career Academies produced little or no change in outcomes for students in the medium-risk subgroup. Results for medium-risk students differed considerably across the participating sites.

The medium-risk subgroup represents approximately 50 percent of the students in the study sample. As shown in Table ES-1, the characteristics of this subgroup do not provide a clear indication of likely school success or disengagement. Figure ES-3 presents a summary of impact findings for students in the medium-risk subgroup. The figure indicates that, on average, the Academies had little or no impact on most outcomes for these students. As discussed below, however, the results for the medium-risk subgroup differed dramatically across the participating sites.

Impact Findings for the Full Sample

• When averaged across the diverse groups of students and sites participating in the evaluation, it appears that the Career Academies produced only modest improvements in students’ engagement and performance during high school.

Figure ES-4 provides a summary of impact findings that are averaged across the full sample of students in the study. It suggests that the Academies produced only slight (and not statistically significant) reductions in dropout rates and in student involvement in negative risk-taking behaviors. On average, the Academies produced modest increases in the percentage of students who earned sufficient credits to meet district graduation requirements and in student involvement in youth development activities. In keeping with one of the central features of the Academy approach, Figure ES-4 indicates a more substantial increase in vocational course-taking. This increase did not come at the expense of students’ being less likely to complete at least a basic core academic curriculum. In general, however, according to the full sample findings, the Career Academies tended to produce small, positive (but not statistically significant) impacts on many student outcomes. As discussed earlier, these aggregate findings mask a great deal of underlying variation that sheds light on the potential strengths and limitations of the Academy approach.

According to standardized achievement tests completed by 490 students in the study sample, the Career Academies did not produce any systematic improvement in students’ math and reading test scores. Although impacts on test scores followed trends found for other outcomes, such as academic course-taking, there was no clear pattern of increases or decreases either on average or among the risk subgroups.

Among students in the high-risk subgroup, average math and reading test scores for the Academy group were somewhat higher than scores for the non-Academy group. While none of the differences was statistically significant, test scores followed this subgroup’s trend of increases in academic course-taking and total credits earned toward graduation. Academy students in the low- and medium-risk subgroups had slightly lower reading test scores than their non-Academy counterparts. This is consistent with the slight (but not statistically significant) reduction in academic course-taking, which was found to be more highly correlated with reading test scores than was non-academic course-taking. There was almost no difference in math test scores between Academy and non-Academy students in the low- and medium-risk subgroups.

Several factors may account for these test score findings. First, qualitative field research information collected for this evaluation indicated that academic curricula and instruction in most of the Career Academies did not differ substantially from those of typical high schools; Academy teachers were required to cover the same basic material as teachers of the same subjects in the rest of the high school. Nor were Academy teachers typically provided with professional development opportunities beyond those offered to their non-Academy counterparts, which focused on standard-setting and instructional strategies in the academic subjects.

Second, there were some important differences between the sample of students who completed the math and reading achievement tests and those who did not. In particular, the magnitude of impacts for the achievement test sample was somewhat smaller and more mixed than the magnitude of impacts described above. For example, among students in the high-risk subgroup who completed the math and reading tests, the Academies produced a somewhat smaller reduction in dropout rates and a somewhat smaller increase in academic course-taking compared with the impacts displayed in Figure ES-1. Among students in the medium-risk subgroup who completed the test, it appears that the Academies actually reduced academic course-taking. In short, the test score sample does not appear to be representative of the full study sample. Nonetheless, there was not a systematic difference in background characteristics between the Academy and non-Academy students in the achievement test sample. Thus, test score impact estimates provide a reliable indication of the Academies’ impact (or lack of impact) on test scores.

Finally, the types of standardized measures of achievement used in this evaluation, and in many school districts, may not adequately capture learning gains that Academy students achieve relative to their non-Academy counterparts. As discussed in an earlier report from this evaluation, Academy teachers were more likely than their non-Academy colleagues to state that they made explicit efforts to plan lessons and activities that cut across academic and non-academic subject areas.⁷ They were also more likely to have students focus on problem-solving activities and to integrate problems and examples from the world of work into their lessons. Academy students were more likely than their non-Academy peers to indicate that they received instruction that included cross-discipline integration and connections between school-based and work-based learning. If the potential benefits of such activities and experiences are of value to schools, they will likely need to be measured through some alternative forms of assessment.

What Factors Help Explain the Pattern of Career Academy Effects?

Figure ES-5 illustrates a conceptual model of the pathways through which the core organizational features of the Career Academy approach are hypothesized to affect student outcomes during high school and beyond. The first column of the figure lists the three core organizational elements of the Career Academy approach: (1) the school-within-a-school, (2) the integrated academic and vocational curricula based on the Academy’s career theme, and (3) the employer partnerships. Three types of supports and learning opportunities (the second column in Figure ES-5) are hypothesized to evolve from the core organizational elements and their interaction: (1) enhanced interpersonal support through the intensive collaboration offered by the school-within-a-school, (2) focused curricula and enriched teaching and learning through the combination of academic and vocational courses, and (3) exposure to career awareness and work-based learning opportunities through the employer partnerships. Together, these supports are intended to increase students’ school engagement and prevent them from dropping out, enhance their performance and help them meet graduation requirements and prepare for post-secondary education and employment, and promote constructive use of non-school hours by increasing developmentally appropriate activities and reducing risk-taking behaviors.

For this report, a variety of analyses were aimed at assessing the relationships between student outcomes and measures of supports and learning opportunities that are likely to arise from the Career Academy’s organizational elements. The findings from these analyses suggest that the strongest associations appear to exist between the interpersonal supports students received early in high school and various measures of their subsequent engagement and performance. The interpersonal supports include students’ perceptions of their teachers’ expectations for them, personalized attention they receive from teachers, the degree to which they see their peers as being engaged in school, and the degree to which they have opportunities to work collaboratively with peers.

Both Academy and non-Academy students who reported that they received particularly high levels of support from their teachers and peers in 9^th or 10^th grade were less likely to drop out of high school, exhibit chronic absenteeism, or engage in risk-taking behaviors than were students who reported lower levels of interpersonal support. They were also more likely to make steady progress toward graduation and to engage in positive youth development activities. One should be cautious about making inferences about causal relationships in this regard. For example, students who achieve positive outcomes may attract strong support from teachers and peers, rather than the other way around. Nevertheless, the patterns of cross-site impacts described below provide further evidence that interpersonal supports are likely to be important antecedents to positive outcomes for students.

To explore the relationship between changes in the school environment that the Academies represent and the impact that Academies have on student outcomes, the evaluation attempted to identify sites in which Academies produced the largest differences in the level of interpersonal support students experienced. Specifically, the individual sites in the evaluation were ranked according to the difference between the percentages of Academy and non-Academy students who reported receiving a high level of support from teachers and peers during 9^th or 10^th grade. For the purposes of this report, the sites with the largest differences are referred to as high-contrast Academies. In the remaining sites, there was little difference in the level of support reported by Academy and non-Academy students; these sites are referred to as low-contrast Academies.

Finally, there are several important similarities between the two groups of sites. Both high-contrast and low-contrast Academies produced substantial increases in students’ exposure to career awareness and development opportunities and their participation in work-based learning activities. It should be noted, however, that within the two groups of sites, some Academies produced much larger increases in students’ exposure to these activities and experiences than others.

• The high-contrast Academies produced a consistent pattern of positive impacts for students in the medium-risk subgroup. On average, the low-contrast Academies increased dropout rates and reduced academic course-taking among these students.

The patterns of impacts for students in the medium-risk subgroup differed dramatically between the high-contrast Academies and the low-contrast Academies. As shown in Figure ES-6, in general the high-contrast Academies produced impacts that were similar but smaller in magnitude to impacts for students in the high-risk subgroup (Figure ES-1): they reduced dropout rates, increased credits earned toward graduation, and increased the percentage of students completing a basic core academic curriculum. Figure ES-6 also indicates that the low-contrast Academies actually increased dropout rates and reduced the percentage of students who completed a basic core curriculum.

While it is not possible to pinpoint the source of differences in impact findings for high- and low-contrast Academies, differences in program implementation may suggest some explanations. For example, qualitative field research information collected for the evaluation indicated that the high-contrast Academies tended to have implemented a tighter school-within-a-school organization compared with the low-contrast sites. The high-contrast Academies typically included a core group of four or five teachers whose responsibilities fell almost exclusively within the Academy. The vast majority of students in high-contrast sites were scheduled together in at least two or three core courses, and very few non-Academy students had to be included in the Academy classes (for example, to ensure adequate enrollments). The high-contrast Academies also tended to be located in a distinct area of the school building or campus. These features of the high-contrast Academies may have nurtured a more personalized learning environment and helped students and teachers feel that they were part of something unique within the school. The tightly organized school-within-a-school may also have served as a foundation for enhancing instructional supports, curriculum integration, and connections between school and work.

The school-within-a-school organization of the low-contrast Academies tended to be more loosely structured and typically included several teachers who had responsibilities both in and outside the Academy. A number of Academy students in low-contrast sites were scheduled in non-Academy sections of core courses, and several of the Academy classes included non-Academy students in order to ensure adequate enrollments. These aspects of program implementation tended to minimize the contrast between the Academy and non-Academy environments. It is difficult to determine how this might account for the apparent reduction in school engagement among the medium-risk subgroup in these sites. It may be that without a tightly organized, highly supportive school-within-a-school environment, the other aspects of the Academy experience (additional vocational courses, career awareness activities, and work-based learning) may have become somewhat of a distraction or burden.

In general, the patterns of impacts for the high-risk and low-risk subgroups were consistent across both groups of sites, with two notable exceptions. First, the low-contrast Academies produced a somewhat larger reduction in dropout rates among the high-risk subgroup. Although the difference in impacts on dropout rates was not statistically significant, this pattern is not consistent with the hypothesis that greater enhancement of interpersonal supports should lead to larger reductions in dropout rates. It is not clear what accounts for the pattern. Second, the low-contrast Academies produced somewhat larger increases in vocational course-taking for both the high-risk and the low-risk subgroups. This may reflect a greater emphasis on vocational course-taking in low-contrast sites and the fact that, on average, relatively few non-Academy students in these sites completed three or more career-related or vocational courses during high school.

Policy Implications and Lessons for Practice

Although the story of the Career Academies’ longer-term effectiveness is not yet complete, the findings to date suggest the following implications and lessons.

• The Career Academies in this study demonstrate the feasibility of implementing a well-defined and effective approach to creating a more supportive high school environment and increasing students’ exposure to career awareness and work-based learning activities.

Large comprehensive high schools (including those participating in this study) have been criticized for being impersonal and for preventing students and teachers from working as teams to create a sense of community and common values. Students in such schools do not have a consistent group of teachers who are accountable for their success, and they see few of the same classmates from course to course. Teachers rarely share the same group of students with a small number of colleagues, and they have few opportunities to coordinate their coursework with teachers in other disciplines. The findings from this evaluation provide evidence that the Career Academies can provide well-defined and effective approaches to addressing such problems.

Another common problem identified in high schools is that students and teachers are isolated from other institutions in the community, particularly employers. Such isolation insulates students from the world of work and misses an opportunity to provide them with learning-oriented exposure to it at a particularly formative point in their development. With few connections among classes or between school and work, many students are inadequately informed about or are unprepared for post-secondary education and employment opportunities. Even with the rise of the school-to-work movement and with the federal School-to-Work Opportunities Act of 1994, there has been a struggle to identify widely implemented strategies that address these concerns. The findings from this evaluation indicate that Career Academies can provide concrete examples of partnerships between schools and employers and can substantially enhance students’ exposure to career development and work-based learning opportunities.

• Career Academies are an effective means of enhancing the school engagement of students who are at high risk of dropping out of high school.

Many of the students served by Career Academies enter high school at a substantial risk of dropping out; many others are likely to become psychologically disengaged from school and to make only limited progress toward graduation. Some of these students have already fallen behind or are disengaged when they enter high school, while others come from home environments that lack the support or resources to facilitate academic persistence and success. Without the intervention of the Academies, about 1 in 3 of these young people will drop out of high school. Previous research has shown that the economic and social costs of not securing a high school diploma are extremely high.

The findings from this evaluation show that the Career Academies substantially reduced dropout rates and substantially improved a variety of measures of school engagement among students in the high-risk subgroup. Not only are effects of this magnitude and pervasiveness rare in the world of education policy interventions, but the long-term payoff, if the effects persist, is likely to be large.

• Career Academies should continue to serve a heterogeneous student population.

Because the largest and most pervasive positive effects in this evaluation were found among students in the high-risk subgroup, it might be argued that the Career Academies should serve only such students. This approach is likely to create a number of problems, however. First, Career Academies have explicitly attempted to move away from targeting students on the basis of their estimated trajectories for school success in order to avoid the tracking and stigma that have been associated with vocational and career-related programs. Second, and perhaps more important, it is likely that exposure to a broad cross-section of students — particularly those who enter the programs highly engaged in school — is an important factor driving the positive effects of Career Academies on the high-risk subgroup. Perhaps the presence of other, highly engaged students in their classrooms helps increase teachers’ attention to and expectations for all students. Excluding engaged students, therefore, would dramatically change the nature of the Academy experience for students at high risk of dropping out.

• Interpersonal supports appear to be necessary conditions for maximizing the positive effects Career Academies have on student engagement. The school-within-a-school organization can provide an effective strategy for enhancing these supports.

The findings indicate that enhancing interpersonal supports may be a key element of school reform initiatives aimed at increasing retention and engagement in school. A highly structured school-within-a-school organization can provide some of the necessary conditions for promoting such supports as personalized attention and high expectations from teachers, high levels of peer engagement, and opportunities for teachers and students to work collaboratively. Career Academies that did not complement their career awareness and work-based learning activities with increased supports (relative to what was already available in the regular school environment) risked having some of their students become disengaged from school.

• Although Career Academies provide a number of supports necessary to keep students engaged in school, these have not been sufficient to enhance achievement, at least as measured by commonly used standardized tests.

The primary added value of Career Academies appears to rest on their enhanced interpersonal supports and increased access to career awareness and work-based learning opportunities. Although these factors may be necessary to keep many students engaged in school, they were not sufficient to improve student achievement. The findings from this evaluation indicate that the Career Academies were quite similar to regular school environments in terms of their academic curricula and typical instructional strategies. From this standpoint, it should not be too surprising that the Academies did not I mprove student achievement as measured by the standardized math and reading tests used in the evaluation. Academies face many of the same challenges that most high schools do in providing teachers and students with appropriate incentives and supports to ensure that they focus on clearly specified learning objectives and that they have the capacity to meet those objectives. The personalized and collaborative nature of the Career Academy’s school-within-a-school organization can serve as a solid foundation on which to build these enhancements.

There is also a question about whether current assessment instruments (including the achievement tests used in this evaluation) adequately capture the distinctive learning gains that Academy students may attain. Such skills may include, for example, the type of work-related competencies outlined in the Secretary’s Commission on Achieving Necessary Skills (SCANS) or the presentation and organizational skills often exhibited in student portfolio assessments. In order to measure such potential benefits of a Career Academy, school officials may want to consider incorporating alternative forms of student and teacher assessment. They may also want to develop forums that recognize efforts by teachers to integrate academic course content with the applied learning and problem-solving approaches of high-quality vocational curricula. Few examples of such assessments and incentives currently exist.

• Longer-term follow-up is needed to ascertain the effects of Career Academies on post-secondary labor market and educational outcomes.

The results in the report summarize the effects that the Career Academies have had on students through the end of the year they were scheduled to be in 12^th grade. The data do not include complete information about actual high school graduation rates or about the critical transition between high school and post-secondary education and work. Further follow-up is needed in order to get a more complete picture of the Academies’ effectiveness and limitations. For example, it will be important to determine whether the reduction in dropout rates among students in the high-risk subgroup translates into higher levels of educational attainment or whether these students simply remain in school longer without earning a diploma or do not go beyond high school. It will also be important to determine whether the Career Academy experience helped or hindered students in the low-risk subgroup, particularly regarding their actual rates of college enrollment and completion. Ultimately, measures of success for Career Academies are likely to depend, in part, on whether the students they attempted to serve are better attached and more successful in the labor market than their non-Academy counterparts.

In order to examine these issues, MDRC’s Career Academies Evaluation will continue through 2003, following the students in the study sample for up to four years after their scheduled graduation from high school. As part of this second phase of the evaluation, MDRC will be administering follow-up surveys to students in the study sample at one year and four years following their scheduled graduation. These surveys will provide information about whether the students’ actually graduated from high school (or received an alternative credential) and about their enrollment and progress through post-secondary education, their labor market experiences, their preparation and planning for the future, and a range of youth development experiences.

Notes:

¹Ten sites were initially selected for the evaluation. One of the initial Career Academies was disbanded after two years in the study and was unable to provide sufficient follow-up data to be included in the impact analysis for this report.

²For a more detailed description of the criteria and process used to select sites for this study, see James J. Kemple and JoAnn Leah Rock, Career Academies: Early Implementation Lessons from a 10-Site Evaluation (New York: MDRC, 1996).

³Although some participating high schools do operate other programs that they classify as Career Academies, information collected for this study indicated that most such programs do not include all the basic components of the Academy approach described earlier. As a result, the participating Career Academy programs represent a clear contrast with the other programs in the high schools.

⁴The test instrument comprised the reading comprehension and math test batteries from the National Educational Longitudinal Survey of 1988 (NELS: 88) Follow-up Study. A total of 490 students from the study sample completed the test, including both high school dropouts and students who remained enrolled in school.

⁵For a more detailed discussion of these findings, see James J. Kemple, Career Academies: Communities of Support for Students and Teachers: Emerging Findings from a 10-Site Evaluation (New York: MDRC, 1997); James J. Kemple, "Selected Dimensions of Applied Learning in Career Academy Classrooms," unpublished MDRC paper, 1997; and James J. Kemple, Susan M. Poglinco, and Jason C. Snipes, Career Academies: Building Career Awareness and Work-Based Learning Activities Through Employer Partnerships (New York: MDRC, 1999).

⁶The definition of these subgroups involved analyses using background characteristics to predict dropping out among students in the non-Academy group. This generated an index of average characteristics of likely dropouts who did not have access to an Academy. The index was then calculated for the Academy group using the same characteristics. Because the predicted relationship between background characteristics and dropout rates was based on the non-Academy group, however, it is likely to yield somewhat more accurate predictions of likely dropouts for that group than for the Academy group. This means that the dropout rate for the students in the high-risk non-Academy group may be artificially high. Extensive analyses were conducted to identify the potential magnitude of this distortion. These analyses indicate that whatever distortion exists is minimal and could not have changed the pattern of impacts. This issue is discussed in greater detail in Appendix B of the report.

⁷James J. Kemple, "Selected Dimensions," cited above

The Wallace Foundation, Ford Foundation, U.S. Department of Education, U.S. Department of Labor, The Commonwealth Fund, Charles Stewart Mott Foundation, William T. Grant Foundation, The Pew Charitable Trusts, The Rockefeller Foundation, The George Gund Foundation, The Grable Foundation, Richard King Mellon Foundation, American Express Foundation, Alcoa Foundation, Russell Sage Foundation, Center for Research on the Education of Students Placed At Risk (CRESPAR), Westinghouse Foundation, The Citigroup Foundation, Bristol-Myers Squibb Foundation, Inc.