Succeeding in the labor market depends now more than ever on having the right education and training. This reality poses a particular challenge for out-of-school youth, who are no longer connected to institutions designed to provide them with training and link them to good jobs. In addition, it is still not clear what is the most effective way to help these youth: Few of the programs that have been evaluated have produced impressive results. The Center for Employment Training, or CET, was one exception. CET in San Jose, California, was included in two large, multisite random assignment studies in the 1980s — the JOBSTART Demonstration for young high school dropouts and the Minority Female Single Parent (MFSP) Demonstration — and it was the only site in both studies to produce large, positive effects on employment and earnings.

The Evaluation of the Center for Employment Training Replication Sites, funded by the Department of Labor (DOL), is an outgrowth of this earlier success. Between 1995 and 1999, over 1,400 youth across twelve sites were assigned at random either to a program group that was eligible to receive CET services or to a control group that was not eligible for CET but could seek out and enroll in other education and training activities in the area. The replication evaluation was designed to test first whether the CET model could be implemented successfully in different settings. CET is noted for enrolling trainees with little prescreening, for providing training in a worklike setting, for requiring a full-time commitment from trainees, for involving employers in the design and delivery of training, for integrating instruction in basic skills into the training, and for allowing trainees to progress as they master competencies, without any fixed schedule. The second question was whether, once implemented, the program would have similarly positive effects for a broader sample of youth — all out-of-school youth, rather that just high school dropouts, as in the JOBSTART Demonstration — and in the economic environment of the late 1990s.

MDRC and Berkeley Policy Associates (BPA) are collaborating on the evaluation of the replication effort, and this is the third and final report in the evaluation. The first report documents program implementation in the twelve sites and finds that only four of the sites can be considered to have achieved high fidelity to the CET model.[1] The second report presents effects after 30 months and finds that the program increased training and certificate receipt and that it had much larger effects in the four “high-fidelity sites,” which represent the fairest test of the CET approach. In the high-fidelity sites, the program did not increase employment and earnings for the full sample but did have positive effects for young women.[2] The present report looks at the program’s effects after four and a half years (54 months). Did the effects that existed at the 30-month point persist longer term? And did the early training advantage eventually pay off for groups for whom there were no effects at 30 months?

Findings in Brief

• Implementing the CET approach is difficult, and fidelity to the original CET model varied greatly across the sites, affecting both implementation and program impacts. Only four sites were deemed to have replicated the model with high fidelity. Simply sustaining the model was a key challenge for several sites, and it also proved difficult to fully implement the job development component.



• Over the 54-month period, youth in the program group were more likely to have participated in training than their control group counterparts. The effect was largest in Year 1 and diminished thereafter, as the control group members continued to enroll in training on their own. Similarly, by Month 54, youth with access to the program were still more likely than control group youth to have a training certificate, although the impact was smaller than at the 30-month point.



• Effects on training and certificate receipt were much larger in the high-fidelity sites than in the other sites. For example, access to CET in the high-fidelity sites increased total time in training by 218 hours through Month 12 and by 145 hours through Month 54. The effects in the medium- and low-fidelity sites were 55 hours through Month 12 and no difference through Month 54.



• Across all sites, the program had no effect on youths’ employment and earnings. However, the fairest test of the CET approach is among the smaller sample of youth in the four high-fidelity sites.



• In the high-fidelity sites, the positive effects on women’s employment and earnings that were evident after 30 months did not persist beyond that point, while the negative effects on men’s employment also did not persist. Effects on employment and earnings did not emerge for most other groups for whom there were no effects at 30 months. Positive effects on earnings did emerge for younger youth in the fourth and fifth years, but these findings must be interpreted with caution due to small sample sizes.

Several factors most likely contributed to the pattern of results. For example, the replication sites operated in a very different environment than the CET program in JOBSTART: They served a broader and perhaps more employable group of youth, and they operated in a stronger labor market and in an environment with more training options, some of which may have been similar to the CET approach. In addition, employers in today’s labor market may view short-term training certificates differently than employers did in the past.

Implementing the CET Model

• The CET approach is difficult to implement; only four of the twelve replication sites put all the key aspects of the model in place.

Early implementation research determined that implementation of the model was strongest among four of the established sites in California that were part of the network of programs that CET developed and ran as it gradually expanded its operations. These high-fidelity sites were able to put in place all the key aspects of the program. Other sites that were newly established or that were operated by organizations other than CET — or that shared both characteristics — had much more difficulty implementing the full model. Six sites implemented it with medium fidelity, and two sites with low fidelity.

A key challenge for the sites was sustaining the CET model once it was implemented. While most sites implemented at least some program components, many of the sites experienced turnover in leadership and funding changes that led them to depart from the CET approach. As a result, four of the twelve sites shut their doors before the demonstration had ended — for example, in the second or third year of follow-up — and three other sites faced serious difficulties in maintaining program operations.

The program component that the sites were most likely to experience difficulty implementing was job development. Several sites did not have the close relationships with local employers that CET-San Jose has, and they were sometimes unable to provide participants with a suitable job opportunity on completion of training. Low intensity of participation was another frequent problem in medium- and low-fidelity sites: Many students did not attend regularly or dropped out before completing competencies and receiving job placement assistance.

Effects on Training and Education

• In the high-fidelity sites, access to CET significantly increased participation in skills training in the first 12 months of follow-up. By Month 54, the effect was still statistically significant but smaller in size.

In the first year of follow-up in the high-fidelity sites, survey respondents in the program group reported an average of 298 hours of skills training (which includes zero hours for those who did not participate), compared with 80 hours for control group members — for an impact of 218 hours. By Month 54, this difference had diminished to 145 hours. The effects in the medium- and low-fidelity sites were 55 hours through Month 12 and no difference through Month 54.

• Access to CET significantly increased receipt of training credentials, with the biggest increase occurring in the high-fidelity sites. The effects on credential receipt were largest at the end of Year 1.

• By the end of the follow-up period, total time spent in education and skills training activities was similar for the program and control groups.

Effects on Employment and Earnings

• The problems in implementing the program made the detection of impacts all the more difficult. The best test of the CET approach is within the smaller sample of high-fidelity sites.

Across all sites combined, access to CET had no positive effects on youths’ employment and earnings. However, the sample of all twelve sites does not represent the best test of the CET model, given that a majority of the sites did not implement it successfully. Therefore, this report focuses largely on effects in the high-fidelity sites. The cost of limiting the analysis to these sites is a substantial reduction in sample size, making the detection of impacts more difficult and the resulting estimates more uncertain.

• In the high-fidelity sites, access to CET did not increase youths’ employment or earnings during the 54-month follow-up period. Although there were some effects in the early years for different subgroups of the full sample, these effects did not persist. Positive effects on earnings did emerge for the younger of two age subgroups, although these findings are suspect because of small sample sizes.

Differences in effects when analyzed by education level also occurred in the early years of follow-up, including negative effects on earnings for youth who entered the study as high school graduates. These effects did not persist into Years 4 and 5. Finally, during the fourth and fifth years of follow-up, earnings impacts did become positive for the younger subgroup. However, because the sample size for this subgroup is only 115, these positive impacts must be interpreted with caution.

• In the medium- and low-fidelity sites, effects on employment and earnings were either negligible or negative.

Understanding the Results

Providing access to CET did not lead to better outcomes than these youth would have had on their own, either by enrolling in other training programs or by gaining experience in the labor market. Two possible reasons for the lack of effects may be the context in which the evaluation took place and the changing value to employers of short-term training.

The Context of the Replication Effort

The findings here differ from the large positive effects of CET that were found in the JOBSTART evaluation. But the replication effort took place in a very different context — so much so that these findings cannot be seen as a repudiation of the earlier results. The context can be regarded along three key dimensions: the population served, the labor market, and the training environment.

1. A broader and more employable group of youth. The application process for CET meant that only the most motivated applicants entered the evaluation. While this was true for the JOBSTART evaluation as well, that sample was restricted to youth who had low reading levels and had not completed high school. In contrast, the replication evaluation targeted all out-of-school youth, including high school graduates. (Efforts to identify a similarly disadvantaged subset of the larger replication sample were hindered by the small sample size within the high-fidelity sites.)



2. The strong economy. The CET replication effort began during a period of strong economic growth, with the result that employment rates for the control group were fairly high — considerably higher than the rates for a comparable JOBSTART sample. Although the economy did weaken later in the CET follow-up period, the effects on training received (which could lead to increased earnings) were substantially smaller by that point.



3. Increased access to employment and training services. Although CET was relatively unusual in the late 1980s, today’s youth have access to a variety of training options, including those offered by community colleges. In addition, partly because of the earlier CET findings, many of the education and training programs that do exist are similar in structure to the CET approach.

In addition, CET might be less successful even with less employable youth if those youth have a variety of other training options to choose from. The context for the replication sites is that CET and its approach are not as distinctive as they used to be. At a minimum, the existence of other options means that the evaluation is not measuring the effects of CET training compared with no training but, rather, is measuring the effects of access to CET training compared with access to the range of other education and training opportunities that are available in the local area.

The Changing Value of Short-Term Training

Youth who had access to CET received more training than their control group counterparts and yet still did not have higher employment rates or higher earnings. Although it could be argued that total hours in training is not a relevant measure unless that training is completed, access to CET also increased “completed training,” or the receipt of training certificates. Surprisingly, receipt of a training certificate had no effect on increasing either employment rates or earnings — suggesting that employers may not value such certificates any more than they value other types of training or even work experience.

In addition, the results here suggest that the training received may not have been high quality relative to other training options available or that participants were trained for jobs in low-demand industries. For example, many of the youth who participated in training under CET and received certificates did not subsequently find jobs in the industries for which they trained. Others did initially find jobs in relevant industries but were working in different jobs by the 54-month point. In addition — and perhaps even more telling — a significant proportion of youth who were surveyed at Month 54 did not remember participating in training or receiving certificates four years earlier.

Youth today are receiving training certificates from a variety of institutions, ranging from proprietary institutions to community colleges, and employers may value some of these credentials more than others. Although CET-San Jose is a respected and well-known community organization, employers in some of the newer replication sites may not know of this track record, and they may have had difficulty distinguishing the quality of CET certificates from certificates offered by other, more established institutions.

The Challenges for Program Design

Targeting the Less Employable

The one aspect of a program’s context that is changeable is the population it serves. The differences between the samples for the CET replication study and for JOBSTART suggest that these types of programs may be more effective for the more disadvantaged segment of out-of-school youth, particularly in a strong economy where job opportunities are more abundant. The negative effects reported here for high school graduates, although short-lived, also suggest a role for targeting; that is, the more educated youth may have been better off gaining work experience. Serving youth who have more barriers to employment would require additional efforts to keep them engaged in program services and, possibly, to help them retain the jobs they subsequently find. Helping them establish strong ties to the labor market at a young age could have important payoffs in the future.

Modifying the Program Components

In a rapidly changing labor market where other training options exist, perhaps there are some modifications to the CET approach that would make it distinctive again and more effective with the youth it serves. Among the suggestions — which are not based on hard evidence — is that the program could continually assess its employer focus, to ensure that it is training youth for high-growth industries. This would include staying up to date on the skills and aptitudes that employers are looking for in new employees. Given that the replication sites seem to have had the most difficulty implementing the job development component, the program could also consider adding an internship to the end of training, to strengthen the transition to work.

The Challenges for Replication

CET-San Jose is a unique institution, with its strong ties to local employers, its history of involvement in the broader community, and its strong leadership. Can such a program that has been homegrown over so many years be replicated? The answer seems to be yes, but the challenges in transplanting it to other settings are daunting, and a new site may have to struggle for many years before its survival is ensured. Even in a deliberate and well-planned demonstration project like this one, the obstacles that local program operators face — often with limited or insufficient resources — are difficult to overcome, especially during a program’s startup phase. The four programs that implemented the model with high fidelity in this study are all older, experienced, CET-operated programs in California. Future replication efforts should provide special outside technical assistance to facilitate the replication process and should also ensure that local programs have the resources and wherewithal to implement the intervention with high fidelity. Successful replication may also require extensive upfront marketing research to establish that there will be motivated customers (both trainees and employers) for the services that the local programs provide.