For the past 30 years, federal and state policymakers have been legislating various types of programs to increase employment among welfare recipients. How people can best move from welfare to work, however, has been the subject of long-standing debate. This report, summarizing the long-term effects of 11 mandatory welfare-to-work programs on welfare recipients and their children, represents a major advance in resolving this debate. The findings are the final ones from the National Evaluation of Welfare-to-Work Strategies (NEWWS), a multi-year study of alternative approaches to helping welfare recipients find jobs, advance in employment, and leave public assistance.

“What works best, and for whom?” is the central question animating this report and the NEWWS Evaluation as a whole. In particular, the evaluation compares the effects of two alternative pre-employment strategies, for different groups of welfare recipients: programs that emphasize short-term job search assistance and encourage people to find employment quickly (referred to as “Labor Force Attachment” [LFA] or, more broadly, “employment-focused” programs); and programs that emphasize longer-term skill-building activities, primarily basic education (referred to as “Human Capital Development” [HCD] or, more broadly, “education-focused” programs). The effects of each approach are estimated from a wealth of data pertaining to over 40,000 single parents (mostly mothers) and their children, and a five-year follow-up period (falling somewhere between 1991 and 1999, depending on the site), using an innovative and rigorous research design based on the random assignment of individuals to one or more program groups (with services) or to a control group (without services).

I. Findings in Brief

The research designs that were implemented in the NEWWS Evaluation permit many comparisons. The key ones examined the programs’ economic effects on adults and the “spillover” effects on noneconomic outcomes and child well-being, as summarized below.

Comparing All 11 Programs to What Would Have Happened in the Absence of the Programs

• In the absence of any welfare-to-work program over a five-year follow-up period, approximately three-quarters of single-parent welfare recipients found jobs, and more than half left the welfare rolls. Few of the 11 studied programs improved on this already-high rate of job-finding, but nearly all programs helped single parents work during more quarters of the follow-up and earn more than they would have in the absence of a program. Moreover, all programs decreased welfare receipt and expenditures over the five years.
  
  

• Measured combined income, however, was largely not affected: The programs led to individuals’ replacing welfare and Food Stamp dollars with dollars from earnings and Earned Income Tax Credits (EITCs), but the programs did not increase income above the low levels of the control group.
  
  
• The programs achieved their economic gains with few spillover effects on such family measures as marriage, fertility, and household composition. Notably, the adults’ gains in self-sufficiency (defined as increased employment and decreased welfare receipt) were achieved with few indications of harm or benefit to the well-being of their children. This was particularly true for mothers with young children, who in 1988 were newly mandated to participate in programs. Because the new mandate’s implications for children were of considerable concern at the time, these families were the subject of intense study in this evaluation.

Comparing Labor Force Attachment (LFA) and Human Capital Development (HCD) Programs

• By rigorously comparing LFA and HCD programs — versions of employment-focused and education-focused programs designed to magnify the differences between the two types of strategies and operated side by side in three evaluation sites — it was found that the HCD approach did not produce added economic benefits relative to the LFA approach.
  
  
• Moreover, the LFA approach moved welfare recipients into jobs more quickly than did the HCD approach ― a clear advantage when federally funded welfare months are time-limited.
  
  
• Finally, the LFA approach was much cheaper to operate than the HCD approach and, at the same time, did not affect sample members’ overall financial well-being or their children’s well-being any differently than the HCD approach.
  
  
• Surprisingly, these findings held true for program enrollees who lacked a high school diploma or a General Educational Development (GED) certificate as of study entry — the subgroup of welfare recipients who were expected to derive the greatest benefit from an initial investment in basic education — as well as for those who already possessed these education credentials.

Comparing Employment-Focused and Education-Focused Programs

• Dividing all 11 programs into two broad categories ― employment-focused programs and education-focused programs ― programs in the former category generally had larger effects on employment, earnings, and welfare receipt than those in the latter category.
  
  
• Given the large number of programs examined and their variety of served populations, implementation features, and labor markets, these results provide more support for the advantages of employment-focused programs than for education-focused ones.

These results should not be taken as an indictment of the benefits of education and training in general in welfare-to-work programs. Nonexperimental work done as part of the NEWWS Evaluation has suggested that obtaining a GED and, especially, obtaining a GED and then receiving some type of vocational training, can result in employment and earnings gains for those who achieve these milestones. [1] However, in the context of mandatory welfare-to-work programs, few people make it this far, for many reasons, including: people leave welfare and therefore do not stay in welfare-to-work programs, and thus education or training classes, for very long; adults supporting families cannot afford an up-front deferment of employment and earnings that may or may not have a longer-run payoff; and only a small minority of welfare recipients report that, if given a choice, they prefer to go to school to study basic reading and math over going to school to learn a job skill or going to a program to get help looking for a job. [2]   It should be noted as well that none of these programs made assignments to or emphasized college.

The Features of the Most Effective Program

• One program ― the Portland (Oregon) one ― by far outperformed the other 10 programs in terms of employment and earnings gains as well as providing a return on every dollar the government invested in the program.
  
  

• The Portland employment-focused program, unlike either the LFA or the HCD programs or the other education-focused programs, initially assigned some enrollees to very short-term education or training and others (the majority) to job search. Also, in another departure from the other programs, job search participants in Portland were counseled to wait for a good job, as opposed to taking the first job offered. While other aspects of the Portland program, such as its use of job developers and staff’s experience operating job search programs, were also noteworthy, these distinctive features, along with other past research, suggest that a “mixed” approach ― one that blends both employment search and education or training ― might be the most effective.

Findings for Children

• Considering the six programs (three sites) in which children who were preschool age at random assignment were studied in depth, impacts were found on a small number of measures of child well-being ― predominantly in the area of the young children’s social skills and behavior. Overall, the young-child impacts differed more often by site than by welfare-to-work approach.
  
  

• Program effects on child care ― one important way in which children might be affected by welfare-to-work programs ― diminished from the two-year follow-up point to the end of the five-year follow-up. As of this latter point, only the Portland program was still producing an increase in the use of child care.
  
  

• In the seven programs (four sites) in which a limited number of measures were examined for children of all ages, few effects were evident. Some impacts, however, were found relating to young adolescents’ academic functioning (but in only two of the four sites for which data are available), and these impacts on adolescents were predominantly unfavorable. As was the case for young children, impacts on children of all ages did not differ by welfare-to-work program approach.

Comparisons Shedding Light on Other Welfare-to-Work Program Design Issues

• Of the two programs with low enforcement of the participation mandate, one had no impact on employment and earnings, and the other had only small effects. It appears that a minimum level of enforcement by program staff is required to produce at least moderate employment impacts, likely because this extra “push” is needed in order to engage in program activities those who normally would not participate on their own initiative.
  
  

• Two of the three programs that used “integrated,” as opposed to “traditional,” case management worked well for those who entered the study without a high school diploma or GED. In integrated case management, one worker fulfills the responsibilities related to the payment of welfare and other benefits, normally performed by income maintenance staff, as well as the responsibilities related to the provision of employment-related services, usually assigned to welfare-to-work program staff. In traditional case management, each welfare recipient has two different case managers. Two programs that implemented different versions of well-funded and well-supported integrated case management produced relatively large impacts for nongraduates; the third program, which also used an integrated case management model but one that was hampered by tight funding, had limited impacts.

The Limits of Pre-Employment Strategies

Average income levels among control group members over the five-year follow-up period were low. Despite the successes of these programs, no program, not even Portland’s, met the long-range goal of making enrollees substantially better off financially. Most program group members continued to have low incomes from various combinations of earnings, the EITC, welfare, and Food Stamps. In fact, among individuals who lacked a high school diploma or GED as of study entry, some programs had the five-year result of making them financially worse off. These findings suggest that the challenge of the future is to identify other types of programs or initiatives that can provide welfare recipients with better and more stable jobs, increase their income, and improve the well-being of their children.

II. Background

In 1988, the federal Family Support Act (FSA) established a system of mutual obligation within the Aid to Families with Dependent Children (AFDC) benefit entitlement structure: Government was to provide education, employment, and support services to AFDC recipients, who in turn were required to participate in the Job Opportunities and Basic Skills Training (JOBS) program created under that act. Through its mandates and incentives, the FSA encouraged state and local program administrators to serve welfare populations with whom they had had little if any contact in the past and to experiment with new types of services, messages, and mandates ― often in advance of solid evidence of how well these innovations would work. For the first time, most single parents with children ages 3 to 5 (or ages 1 to 5 at the state’s option) were required to enroll in welfare-to-work programs. In addition, the FSA mandated that programs reserve at least 55 percent of funds for services to welfare recipients who were deemed to be at greatest risk of long-term welfare dependency. Furthermore, the FSA required enrollees to participate in employment-preparation activities for as long as they remained on welfare and eligible for services. Case managers were expected to monitor participation and to use a variety of informal and formal responses (including reductions of welfare grants) when enrollees did not attend. In addition, people were supposed to be assigned to additional activities if they completed participation in employment-preparation activities without finding a job.

The expansion of welfare-to-work programs and the requirement to work with more disadvantaged populations intensified a long-standing debate among program administrators and policymakers concerning how best to help welfare recipients ― especially those facing serious barriers to employment ― move from welfare to work. Rigorous research in the 1980s demonstrated that job search programs sped up the entry of welfare recipients into the labor market. Often, however, the jobs were neither long-lasting nor high-paying, and they did not increase family income. Furthermore, the programs generally did not benefit the most disadvantaged welfare recipients. During the years before passage of the FSA, administrators of welfare-to-work programs in several states and localities (most notably, California) began implementing programs that emphasized up-front investments in basic education and skill development as an alternative to job search. FSA regulations accelerated this trend by requiring states and localities to offer a variety of employment-preparation activities, including job search, basic education (classes in adult basic education, GED preparation, regular high school, and English as a Second Language), and vocational training and post-secondary education.

Proponents of education-focused programs argued that this approach offered the best chance of helping people ― especially those who lacked a high school diploma or faced other barriers to employment ― to get better and more stable jobs, increase their family’s income, and reduce returns to the welfare rolls. Some further hypothesized that education-focused programs would benefit children more than job search programs, because parents who attended education or training classes would become more involved in their children’s schoolwork and would serve as role models for succeeding in school.

There was little evidence at the time, however, that large-scale mandatory education programs for welfare recipients would achieve these goals. It was expected that programs that emphasized education and training would engage participants for months and perhaps years longer than programs that emphasized short-term job search assistance. As a result, education and training programs would be more costly to run than job search programs. To be considered cost-effective from a budgetary standpoint, these programs would have to produce savings in welfare and other benefits well in excess of what the less expensive job search programs would attain. Moreover, participants in education and training programs were expected to experience an initial “opportunity” cost (compared with participants in employment-focused programs) in the form of forgone earnings during the period when they were in the classroom rather than the workplace. It was hoped that participants in education-focused programs would make up for such forgone earnings as well as their later start in accumulating work experience and on-the-job skills by attaining better initial jobs and by advancing more quickly once employed than would otherwise be the case. However, it was not clear whether the people who were expected to attend education and training would actually remain in school long enough to attain credentials or enhance their job skills. This issue was especially crucial for people entering welfare programs with low levels of educational attainment or without the minimum literacy and math skills needed for employment.

The FSA also afforded program administrators an opportunity to address the shortcomings of low-cost job search services. Many job search programs before the FSA required participants to look for work but provided little instruction on how to find employment. In contrast, from the late 1980s onward, programs increasingly assigned enrollees to organized group job clubs, whereby participants received instruction on finding job leads, filling out résumés, and conducting job interviews. Many programs followed classroom instruction with one or more weeks of supervised job search, during which they provided job leads and the use of phone rooms to contact employers. Over time, program operators added new features to job club curricula, including career exploration, life skills and time management instruction, and self-esteem-building exercises. Some programs actively marketed their job placement services to area employers and engaged in job development activities to increase the pool of available jobs. Administrators also invested in new ways to communicate a pro-work message, although (as will be discussed below) the types of messages differed. Finally, the FSA’s ongoing participation requirement encouraged administrators to design follow-up activities (often short-term education and training) for job search participants who did not find employment.

In general, implementing these enhancements made job search activities more costly to operate, but not as costly as education-focused programs. It was hoped that these changes would be cost-effective by helping more disadvantaged welfare recipients find work and by helping people find employment sooner than they would have otherwise.

The most recent federal welfare reform effort, the 1996 Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA), replaced AFDC with a flexible, state-directed block grant program, Temporary Assistance for Needy Families (TANF); set lifetime limits on most families’ receipt of federally funded TANF assistance; and created financial incentives for states to run mandatory, work-focused, welfare-to-work programs. If anything, TANF’s time limit, its focus on work, and its requirement that agencies work with the entire welfare caseload brought a new urgency to the question of which welfare-to-work approach was most effective.

In addition, both the FSA and PRWORA strengthened the requirement that single parents prepare for employment in exchange for welfare benefits, and they extended this mandate to parents with young children. These developments increased the importance of learning how welfare-to-work programs affected families. To what extent would programs be able to involve mothers who had young children? Would parents with toddlers and preschool-age children be able to find stable, affordable, and high-quality child care while they participated in pre-employment activities and while they were working? Would child care prove a financial burden and a barrier to employment after parents left government assistance? More generally, how would children be affected if there were program-induced changes in their mothers’ educational attainment, hours of employment, or self-esteem; in their family income; or in the amount, type, or quality of child care they experienced?

It was within this context that the NEWWS Evaluation was conceived and funded, in 1989, by the U.S. Department of Health and Human Services (HHS), with support from the U.S. Department of Education. The Manpower Demonstration Research Corporation (MDRC) conducted the evaluation. Child Trends, as a subcontractor, conducted the Child Outcomes Study, the part of the evaluation that examined effects on young children.

III. Program Approaches and Implementation Features

The programs in the NEWWS Evaluation implemented many of the features described above. As shown in Table 1, the 11 programs in the NEWWS Evaluation were operated in seven sites across the country. Employment-focused programs were operated in Atlanta, Georgia; Grand Rapids, Michigan; Riverside, California; and Portland, Oregon. Atlanta, Grand Rapids, and Riverside also operated education-focused programs, as did Columbus, Ohio (two programs); Detroit, Michigan; and Oklahoma City, Oklahoma.  The studied programs were initially administered under the FSA, which created the national JOBS program for recipients of cash assistance under AFDC. The programs continued (with some modification) under the FSA’s successor, PRWORA, which replaced AFDC with the TANF block grant program. Under both welfare reform acts, the programs’ primary goal was to move welfare recipients off government assistance and into paid work.

The four employment-focused programs ― the LFA programs in Atlanta, Grand Rapids, and Riverside as well as Portland’s program ― assigned most enrollees to job club as their first activity, and they encouraged enrollees to find work as quickly as possible. Further, both Portland’s and Riverside’s program employed full-time job developers to help place program enrollees in unsubsidized jobs.

In contrast to the three LFA programs, however, Portland’s program offered GED preparation classes to people who case managers thought had a good chance of attaining a GED certificate relatively quickly. Furthermore, Portland case managers, more often than those in the LFA programs, encouraged enrollees to hold out for jobs that paid well above the minimum wage (about 25 percent higher) and that offered the best chance for long-lasting and stable employment. Case managers in the LFA programs, especially Riverside’s, stressed the value of starting off with any job, even a low-paying one, and then advancing toward more stable and better-paying jobs in the future.

The HCD programs in Atlanta, Grand Rapids, and Riverside; the Columbus Integrated and Traditional case management programs; [3] and the programs in Detroit and Oklahoma City can each be characterized as “education-focused.” A large percentage of program enrollees in these programs were initially assigned to some type of skill-building activity. The types of activities to which enrollees were first assigned depended, in part, on the level of educational attainment that individuals had achieved prior to entering the program. Those who had not completed high school or received a GED certificate but who were assessed by case managers as having high-school-level skills were assigned to GED preparation classes. Those with lower reading or math levels were assigned to adult basic skills classes. In addition, non-English speakers could be assigned to English as a Second Language (ESL) programs. Finally, those who had completed high school or held a GED certificate could be assigned to vocational training or employment-oriented skills courses at local community colleges. All in all, however, assignments to GED preparation or basic education courses predominated in these education-focused programs, and assignments to vocational training programs were less common, primarily as a result of welfare recipients’ low levels of educational achievement; enrollment in college played an even smaller role.

Other key program features varied across the 11 studied programs as well. All four employment-focused programs and five of the seven education-focused programs can be considered “high enforcement” programs: They worked with a cross-section of the welfare applicants and recipients who were required to participate; monitored participation closely; and, especially in the two programs in Columbus and the two in Grand Rapids, frequently invoked sanctions (reductions in welfare grants) for nonparticipation. The remaining two education-focused programs, in Detroit and Oklahoma City, did not have these characteristics (because of either lack of funds or program philosophy) and can be considered “low enforcement” programs.

The programs also differed in their child care policies and practices (within each site, however, child care assistance policies were identical for program and control group members). During the early to mid 1990s, the Atlanta, Oklahoma City, Portland, and Detroit programs provided the strongest staff support for arranging for child care, and the programs in Atlanta and Oklahoma City emphasized the use of licensed care; in contrast, case managers for both Riverside programs encouraged enrollees to find low- or zero-cost, informal child care.

The programs also differed in their case management strategies. Three programs ― Columbus Integrated, Portland, and Oklahoma City ― implemented an integrated case management staffing arrangement. The other programs used a traditional case management structure.

IV. Research Designs and Samples

The NEWWS Evaluation used a rigorous design ― called a social experiment — to estimate the effects of employment- and education-focused programs. Welfare recipients were randomly assigned to one of two or three research groups, depending on the site.

As part of a largely unprecedented effort to determine which welfare-to-work program approach works better, three sites ― Atlanta, Grand Rapids, and Riverside ― simultaneously operated, expressly for the evaluation, two different programs: an LFA program and an HCD program. These programs were multidimensional but varied in terms of the key features that program operators and researchers thought most clearly differentiated employment- and education-focused programs. Each type of program communicated a different message to welfare recipients about the best route to employment, and each type differed from the other in the way program services were sequenced and emphasized. The programs were, however, mandatory to the same degree: Nonparticipants risked a reduction in their monthly welfare grant. In these three sites, welfare recipients were randomly assigned to an LFA program group, an HCD program group, or a control group. (Control group members were not subject to the participation mandate and received no services through either type of program but, on their own, could seek out similar services within the community.) This random assignment research design produces the most reliable comparison between employment- and education-focused programs. It ensured that, within each site, there were no systematic differences between the background characteristics of people in the LFA, HCD, and control groups within each site when they entered the study. Thus, any subsequent differences in outcomes between groups ― comparing either the LFA or the HCD group to the control group, or the LFA and the HCD groups to each other ― can be attributed with confidence to the effects of a particular type of program. These differences, referred to as the programs’ impacts, are the primary focus of this report, and all differences reported are statistically significant unless otherwise noted.

In the Columbus site, a three-group random assignment design was used as well. Here, the two program groups represented two case management models: “integrated” and “traditional.” The remaining three sites in the evaluation ― Detroit, Oklahoma City, and Portland ― used random assignment to test the effectiveness of established programs, as opposed to programs designed to meet research protocols; individuals were randomly placed either in a group that entered the program or in a no-program control group. Note that control group members in all sites were eligible for child care assistance, similar to that offered to program group members, if they were participating in nonprogram activities in which they had enrolled on their own.

In each site, individuals were randomly assigned to research groups over approximately a two-year period. Random assignment for the evaluation began in June 1991, in Riverside, and ended in December 1994, in Portland. Thus, the five-year results presented in this report cover the calendar period of June 1991 (the first sample member’s entry into the study) through December 1999 (the last month of the five-year follow-up for the last sample member randomly assigned, in Portland).

The research designs that were set up in the seven NEWWS Evaluation sites permit many comparisons. The key ones examine the programs’ economic effects on adults and the spillover effects on noneconomic outcomes and children. The central comparisons may be expressed as follows:

• Compared with what would normally happen in the absence of any type of welfare-to-work program, how effective are employment- and education-focused programs and, more narrowly, LFA and HCD programs? That is, comparing outcomes for the program groups with outcomes for the control groups, what are these programs’ net impacts?
  
  

• Compared with one another, which approach is more effective? For example, comparing outcomes for the LFA and HCD groups directly (ignoring the control groups), what are the differential impacts of the two approaches?
  
  

• How effective are the programs for two key subgroups of welfare recipients for whom employment- and education-focused approaches might be expected to work differently ― namely, those who, as of study entry, had a high school diploma or GED (“graduates”) and those who did not (“nongraduates”)? [4]

It should be noted that while control group members were not exposed to the services and mandates of the sites’ programs for the first three years of follow-up, their status differed by site in the fourth and fifth years. For a few programs, net impacts for years 4 and 5 are understated somewhat because a small portion of the control group (a subset of those still on welfare) received program services toward the end of the five-year follow-up period. (In these sites, the “start of the clock” for welfare time limits necessitated allowing control group members access to welfare-to-work program services.) Importantly, however, this situation does not affect the results of direct comparisons of the LFA and HCD program approaches (that is, the differential impacts). The three-group random assignment designs in the three sites in which the LFA and HCD programs were operated side by side permit a direct comparison of these two approaches, that is, one that does not need to take into account the services received by, or the behavior of, control group members.

This report includes data from administrative records (unemployment insurance [UI], state and county welfare payments, and Food Stamp data) and from surveys administered to mothers and children over the five years after individuals entered the study.

V. Five-Year Effects on Use of Employment-Related Services and Costs

• All programs increased participation in employment-related activities relative to control group levels of self-initiated activity. Employment-focused programs produced large increases in participation in job search activities, and education-focused programs produced large increases in participation in basic education classes.

Over the five-year follow-up period, a majority of control group members in each site (up to 75 percent) participated in some type of employment-related activity: job search, basic education, vocational training, or post-secondary education. Almost all of this activity was the result of control group members’ own initiative; despite the potential in several sites for controls to be subject to mandatory welfare-to-work programs at the end of the follow-up period, there is little evidence that much control group participation in such programs did, in fact, occur. Some of this self-initiated activity took place while control group members were receiving welfare; much of it took place after they left the welfare rolls. In most sites, the most common activities in which control group members enrolled themselves over the five-year period were vocational training and post-secondary education programs.

All programs increased overall participation levels above those achieved by control group members. The employment-focused programs increased participation in job search by approximately 30 percentage points relative to control group members. The education-focused programs increased job search participation as well, but to a much lesser degree. Most education-focused programs produced large increases in education and training relative to control group members. Among those who entered the study without a high school diploma or GED (“nongraduates”), increases were particularly large in basic education. Among those who entered the study with these credentials (“graduates”), few programs increased participation in vocational training. (While education-focused programs most commonly assigned graduates to vocational training activities, rarely did program group levels of participation in such activities exceed those of the control groups.) Increases in education and training participation in the employment-focused programs were much less common and, where they did occur, were smaller than those in the education-focused programs.

The Portland program, with its employment focus but mix of initial program activity assignments, produced five-year increases in both job search and education participation. While the program produced large increases in job search participation for both nongraduates and graduates, it also resulted in a 10 percentage point increase (though not a statistically significant one) in basic education participation among nongraduates and a large, 21 percentage point increase in post-secondary education participation among graduates.

Overall, program group members’ length of stay was longer in education and training activities than in job search activities. For example, within the first two years of follow-up, the typical participant in an adult education program received the equivalent of about two-thirds of a year of instruction in a high school. [5] Length of stay in any type of program activity was often curtailed because program group members started working for pay and/or left welfare ― the goals, after all, of welfare-to-work programs.

Education-focused programs generally increased the proportion of nongraduates who obtained a GED or high school diploma over the five-year follow-up period, whereas only the Portland program among the employment-focused programs had such an effect (though this increase is not statistically significant). In addition, Portland had a notable increase in the proportion of nongraduates who obtained a high school diploma or GED as well as a second education or training credential. Overall levels of high school diploma or GED receipt, however, were low: By the end of the five-year follow-up period, less than one-quarter of initial nongraduates in the education-focused program groups had obtained a high school diploma or GED. Among graduates, only the two programs in Atlanta produced five-year increases in the receipt of some type of education or training credential ― generally, a trade license or certificate.

• As expected, education-focused programs cost more than employment-focused programs over five years. Regardless of program approach, costs were higher for individuals who entered the study already possessing a high school diploma or GED (graduates) than for those who entered the study as nongraduates.

The cost analysis considered all costs associated with providing employment services and associated support services to sample members (including case management costs). Costs paid by welfare departments and non-welfare agencies, and in-program as well as post-program or post-welfare costs, were included.

Five-year net per-person costs (the gross cost per program group member minus the gross cost per control group member) averaged $3,037 for the employment-focused programs and $3,972 for the education-focused programs. (These are 1999 dollars.) The most reliable comparison of the costs of these two types of program approaches, however, is one comparing LFA and HCD net program costs within each site. In Atlanta, Grand Rapids, and Riverside, HCD programs were 40 percent to 90 percent more expensive than their counterpart LFA programs.

NEWWS program costs were high compared with other programs that have been studied by MDRC. This is largely because of the greater use in all the NEWWS programs of high-cost education activities, such as post-secondary education and vocational training, relative to past welfare-to-work programs; as well as the enhancements made to job search activities by most NEWWS programs, relative to the above-described simple job search activities implemented in the 1980s. The average cost of the NEWWS programs was comparable to that of the two highest-cost California Greater Avenues for Independence (GAIN) programs, which operated in Alameda and Los Angeles Counties in the late 1980s and early 1990s.

VI. Five-Year Effects on Economic Outcomes for Adults

A.  Employment and Earnings

• Most control group members worked at some point in the five-year follow-up period, without the assistance of a welfare-to-work program.

At the high end, 88 percent of control group members (in Grand Rapids) were employed at some point during the five-year follow-up period; at the low end, 79 percent (in Oklahoma City) and 66 percent (in Riverside) worked during this same time frame. In addition to illustrating the strong interest that welfare recipients have in going to work ― regardless of any welfare-to-work program intervention — these figures suggest that there was little room left for programs to increase the proportion of program group members who “ever” worked. Employment levels for control group members grew steadily over the five-year follow-up period, although many control group members worked for less than one year and then experienced a spell of joblessness.

• Nearly all 11 programs increased how much people worked and how much they earned, relative to control group levels, but the four employment-focused programs generally produced larger five-year gains in employment and earnings than did most of the seven education-focused programs. Portland produced the largest, most consistent increases by far.

Not surprisingly, given the high levels of employment for control group members, programs generally had little effect on the percentage of sample members who “ever” worked. However, in 9 of 11 programs, the program group worked during more calendar quarters on average than the control group; and in 9 of 11 programs, the program group averaged higher total earnings than their control group counterparts.

Portland produced the largest, most consistent employment and earnings effects by far: Over five years, program group members worked 1.6 quarters more than control group members ― a 21 percent increase in employment duration ― and their average five-year earnings were about $5,000 greater than those of control group members. (See Figure 1, which depicts the impacts, or program-control differences, on earnings for all programs.) Portland’s program also produced the largest impacts on measures of stable employment and earnings growth among the 11 programs. Portland’s success may have resulted from its unique combination of a focus on employment, the use of both job search and education, and an emphasis on finding good jobs. In addition, the program made extensive use of job development, and staff were experienced in operating welfare-to-work programs. Portland’s relatively strong economy also may have contributed to the success of the program; however, other programs in localities where the demand for labor was similarly high did not do as well.

The employment-focused LFA programs in Atlanta, Grand Rapids, and Riverside also affected employment and earnings, but less so than the Portland program. Five-year earnings gains ranged from about $1,500 in Grand Rapids to about $2,500 in Atlanta and Riverside. The programs also increased employment duration by an amount ranging from 0.7 quarter in Grand Rapids to 1.1 quarters in Riverside.

The effects of the seven education-focused programs, as a group, were smaller than the effects of the employment-focused programs. Neither of the two programs with low enforcement of the participation mandate (Detroit and Oklahoma City) significantly affected employment. Among the other five education-focused programs, employment duration gains over five years ranged from 0.3 to 0.8 quarter, and earnings gains ranged from about $800 to about $2,000.

Employment-focused programs produced effects almost immediately, whereas education-focused programs generally did not have effects until more than a year after random assignment. In the middle of the follow-up period, most of the programs increased employment and earnings, but effects diminished during the final two years and were statistically insignificant for most programs by the end of year 5. (An example of this trend for the Grand Rapids LFA program can be seen by comparing the black bars and the white bars in Figure 2, and for the Grand Rapids HCD program by comparing the shaded bars and the white bars.) These results were especially disappointing for education-focused programs. As discussed above, education and training services were intended to help program group members eventually move into more stable and higher-paying jobs (compared with control group members and compared with those subject to employment-focused programs) in order to make up for forgone earnings early in the follow-up period. However, most programs ― education- or employment-focused — had little or no effect on measures of stable employment and earnings growth.

A comparison of earnings impacts for nongraduates demonstrates more clearly the disappointing results for education-focused programs. For this subgroup, only two of the seven education-focused programs significantly raised five-year average earnings above control group levels, whereas three of the four employment-focused programs did so. The employment-focused program with the largest earnings impacts for nongraduates was Portland, however, which used a mix of initial activities that resulted in substantial use of education by nongraduates. Furthermore, the education-focused Columbus Integrated program led to earnings impacts among nongraduates that, along with those of the Grand Rapids LFA program, were the next-largest among all programs. This suggests that education activities, in some instances, may contribute to earnings impacts.

Employment-focused programs, compared with education-focused ones, also more consistently increased the earnings of high school graduates above control group levels. Once again, however, the largest earnings impact for high school graduates was found for the Portland program, which substantially increased the use of post-secondary education among graduates.

• Directly comparing the LFA and HCD programs in the three sites in which these programs were run side by side (thus using the most rigorous method for assessing the relative effectiveness of employment- and education-focused programs), employment and earnings levels over five years were largely similar for the two types of programs. Where there were differences between the two types of programs ― for early follow-up years or for a particular subgroup or outcome measure ― they were in favor of the LFA approach.

Cumulatively, over the five-year follow-up period, few LFA-HCD differences in employment or earnings (that is, differential impacts) are found when both graduates and nongraduates are included in the calculations (see the top panel of Figure 1). [6] Year by year, however, there were some differences. The first set of bars in Figure 2, showing only the Grand Rapids site, illustrates these. LFA average per-person earnings were higher than those of the HCD sample members in at least year 1 in Grand Rapids as well as in Atlanta and Riverside. The gap between the two types of programs narrowed, however, in year 2 of follow-up in Grand Rapids and in Atlanta and in year 3 of follow-up in Riverside. In addition, there was one LFA-HCD difference for the full sample on the measure of average quarters employed: In Grand Rapids, the LFA group worked more quarters than did the HCD group. Notably, the HCD programs, relative to the LFA ones, did not produce more earnings growth over the follow-up period or increase the likelihood of employment in “good” jobs. Finally, the yearly trends suggest that the story would not change if longer follow-up were available.

• Again directly comparing the LFA and HCD programs in the three sites in which these programs were run side by side, employment and earnings impacts were greater in the LFA programs than in the HCD programs among nongraduates. Among graduates, the two approaches produced similar impacts.

As shown in the lower panel of Figure 1, two LFA programs and one HCD program produced five-year earnings increases, relative to control groups, for nongraduates; as shown in the middle panel of this same figure, a different set of two LFA programs and one HCD program produced five-year net earnings impacts for graduates.

Contrary to expectations, earnings impacts generally were larger for nongraduates in LFA programs than in HCD programs. (Proponents of the HCD approach anticipated that education-focused programs might be particularly effective for those without high school diplomas or GEDs, inasmuch as their lack of skills or credentials might inhibit employers from offering them jobs. But this did not turn out to be the case.) Among this subgroup, LFA-HCD differences in five-year earnings were $920 in Riverside, $1,095 in Atlanta, and $1,945 in Grand Rapids. While only the Grand Rapids difference is by itself statistically significant, the average difference across the three sites is statistically significant at the 5 percent significance level. Furthermore, in no year of the follow-up were nongraduates’ average earnings higher in HCD programs than LFA programs; rather, earnings were generally higher in the LFA programs, with statistically significant differences found in the early years of follow-up in every site. (See the last set of bars in Figure 2 for an illustration of this pattern in Grand Rapids.) Among graduates, earnings impacts were very similar in the two types of programs.

B.  Welfare Receipt and Payments

• The majority of control group members in all sites were off the welfare rolls as of the end of the five-year follow-up period, without the assistance of a welfare-to-work program.

The average control group member remained on assistance for about two to three years during the five-year follow-up period. Levels of welfare receipt fell steadily over time for control group members, reflecting “normal” welfare exits. Welfare receipt reached particularly low levels in Columbus and Portland, where less than 20 percent of control group members were receiving a welfare payment at the end of year 5.

• All programs reduced months on welfare and Food Stamps as well as welfare expenditures over five years, relative to control levels, with most programs leading to relatively large welfare savings. Welfare reductions were not consistently larger in the employment-focused programs than in the education-focused ones.

All programs had an effect on the number of months that people received welfare. On average, employment- and education-focused program group members received AFDC or TANF assistance for two to six fewer months than their control group counterparts.

All programs also reduced total welfare payments below control group levels, and most produced savings of 10 percent or more (a historically large effect). (See Figure 3.) For many programs, welfare savings were larger and more persistent than earnings gains: Few programs continued to affect employment and earnings in year 5, but most programs continued to generate welfare savings at the end of year 5. This finding implies that some program group members who exited welfare for employment early in the follow-up did not return to assistance after leaving employment, even though they may have been eligible to do so. It is possible that the national welfare climate in the aftermath of the federal welfare reform legislation of 1996 contributed to this pattern, and since more program than control group members left welfare in the early years of follow-up (before enactment of the legislation), the climate may have had more of an effect on program than control group members.

Welfare savings were generally larger for programs that had greater effects on earnings, but they varied for other reasons as well. Total payments were reduced more in higher-grant sites such as Riverside, Portland, and Grand Rapids and were reduced less in lower-grant sites such as Atlanta. In addition, welfare benefits were reduced more in sites that strictly enforced program participation mandates, such as Columbus and Grand Rapids, but benefits were reduced relatively little in sites that did not enforce mandates, such as Detroit.

The programs had similar welfare impacts for high school graduates and nongraduates. Most programs produced welfare savings for both groups, and there is little evidence that the effects were larger for one group than the other: In five programs, welfare savings were larger for graduates, but in five other programs, welfare savings were larger for nongraduates.

Over five years, program group members in all programs spent less time on Food Stamps and on average received smaller Food Stamp payments than control group members. Food Stamp impacts were generally smaller than welfare payment impacts, however, because some program group members appropriately continued to receive Food Stamps after they left welfare.

• In the three LFA-HCD sites, LFA sample members left welfare at a slightly faster pace than HCD sample members in the first year of follow-up, but the gap narrowed in subsequent years. Only in one site did the LFA and HCD programs differ with respect to the number of months on welfare or welfare expenditures over five years.  In this site, welfare months and expenditures were lower in the LFA program than the HCD program.

Cumulatively, over the five-year follow-up period, a statistically significant LFA-HCD difference (differential impact) in welfare expenditures was found in only one site (Grand Rapids), where the LFA program produced savings of $785 more than the HCD program (see Figure 3) and where the average number of months in which welfare or Food Stamps were received also was lower in the LFA program. In Grand Rapids, this pattern held for graduates as well as nongraduates. In Atlanta, welfare expenditures and months on welfare were lower for the LFA programs than for the HCD programs, but these differences are statistically significant only among the Atlanta nongraduates. In Riverside, the LFA and HCD programs led to similar reductions in welfare receipt and expenditures for nongraduates. The fact that in all three sites LFA sample members left welfare more quickly than HCD sample members represents a clear advantage in an environment where federally funded welfare months are time-limited.

C.  Combined Income

• The combined income from earnings, welfare and Food Stamp payments, and Earned Income Tax Credits for control group members was low. On the positive side, over the five years, program group members received a larger portion of such combined income from earnings, compared with the control group. The programs, however, were largely unable to increase total combined income. Income impacts varied more by site than by program approach, but, among nongraduates in the three LFA-HCD sites, those in the LFA program groups had higher combined income than those in the HCD program groups.

Both employment- and education-focused programs helped sample members become more self-sufficient relative to the control group by increasing employment and earnings and reducing public assistance. As a result of these changes, program group members received a higher percentage of their income from earnings, compared with the control group. This impact averaged about 4 percentage points across all programs but was somewhat larger for employment-focused programs, especially the Portland one, than for education-focused programs.

Program group members, however, received about the same amount of income as their counterparts in the control group. Two programs ― one employment-focused and the other education-focused ― led to especially large decreases in welfare and Food Stamp payments and to decreases in combined income. In general, program effects on combined income were less positive (larger decreases or smaller increases) for nongraduates than for graduates. In addition, among nongraduates, LFA programs did better than HCD programs. Among nongraduates in the LFA-HCD sites, a simple average of the impacts across the three sites indicates that the three LFA programs as a group resulted in almost $1,000 more in combined income over five years than the HCD programs, a statistically significant difference.

Including estimates of sample members’ Earned Income Tax Credits and payroll taxes did not change the above results. (These findings do not account for program effects on other possible sources of income, such as child support payments or unemployment insurance benefits or income from spouses, partners, or other household members. The available data, however, suggest that the inclusion of these other income sources would have changed the impact estimates very little, if at all.)

D.  The “Most Disadvantaged” Subgroup

• Neither employment-focused nor education-focused programs consistently had the largest earnings impacts for sample members who could be considered the “most disadvantaged.”

As discussed above, FSA programs were required to target welfare-to-work resources on individuals at greatest risk of long-term welfare dependency. The NEWWS Evaluation was designed to determine which types of employment-preparation services provide the greatest benefit to at-risk populations.

In several respects, both employment- and education-focused programs were successful. Most programs raised earnings above control group levels for sample members with serious barriers to employment, such as no recent work history and a lengthy history of prior welfare receipt. Similarly, in most programs, the most disadvantaged group members ― welfare recipients who did not have a high school diploma or GED, who had a history of welfare receipt, and who had not worked recently ― earned more on average than their counterparts in the control group. Differences for several of these programs are small, however, and are not statistically significant.

In Atlanta and Grand Rapids, the LFA programs, compared with the HCD ones, led to considerably higher earnings impacts for the most disadvantaged sample members ― especially in Atlanta, where the HCD program had no effect on this subgroup. However, earnings increases for this subgroup (relative to the control group) were similar for the two programs in Riverside.

Although the more disadvantaged groups had higher earnings as a result of most NEWWS programs, they still earned very little. In addition, most programs reduced welfare and Food Stamps by a larger margin than they increased earnings. As a result, programs did not raise the combined income of the most disadvantaged recipients above control group levels.

VII.   Benefit-Cost Analysis

The benefit-cost analysis extends the findings on program impacts and costs presented above. It considers program effects on additional outcomes, such as fringe benefits from employment, income and sales taxes, Medicaid expenditures, and the administrative costs of transfer programs. These additional outcomes were estimated or imputed from administrative records and published data. The analysis also considers program effects from the standpoint of sample members (referred to as the welfare sample perspective) and of government (referred to as the government budget perspective).

The benefit-cost analysis from the welfare sample perspective considers (in a more comprehensive way than presented above) whether programs increased program group members’ income ― from any source — relative to the control group. (While society generally favors earnings over welfare payments, the benefit-cost analysis from the welfare sample perspective does not favor either income source.) The welfare sample derives a net gain from a welfare-to-work program if the program increases earnings (plus fringe benefits) by an amount that exceeds the sum of the welfare, Food Stamp, and Medicaid benefits lost and the increase in taxes paid  (net of Earned Income Tax Credits).

From the government budget perspective, in contrast, increases in tax revenues and reductions in benefits are a net gain. The government budget perspective also counts as benefits any savings in administrative costs from reductions in receipt of transfer payments. These gains are compared with the program-control group difference in cost, that is, with the net cost of providing employment-related services.

Programs may lead to net gains from both the welfare sample and the government budget perspectives; they may also lead to net losses from both perspectives. Other times, programs may benefit either the welfare sample or the government budget.

• From the benefit-cost perspective of the welfare sample, most of the 11 programs resulted in financial losses. From the perspective of government bud­gets, the majority of the programs saved the government about as much as they cost.

From the benefit-cost perspective of the welfare sample, most of the 11 programs produced net financial losses. Moreover, gains were close to zero in the few programs that resulted in them. (See Table 2.) In contrast, from the benefit-cost perspective of government budgets, the majority of programs broke even (that is, saved the government only slightly more or slightly less than they cost), a few produced clear savings, and one produced a clear cost. Government budget savings in several programs were larger for nongraduates than for graduates.

• Directly comparing the benefit-cost results for LFA and HCD programs shows that full-sample results were similar ― from the perspective of the welfare sample ― for both programs within each of the three sites. Nongraduates, however, uniformly experienced losses, which in each site were greater in the HCD programs than in the LFA programs. From the government budget perspective, returns to investments in each site were greater in LFA than HCD programs.

From the perspective of welfare sample members, neither LFA nor HCD programs consistently yielded gains or losses. (See Table 2.) Rather, in each site, both programs produced either a net loss for all enrollees (in Grand Rapids and Riverside) or a gain close to zero (in Atlanta). Nongraduates, however, experienced losses over the five years in both types of programs, but the losses were consistently greater in the HCD than the LFA programs. For graduates, results were mixed.

From the standpoint of government budgets, neither LFA nor HCD programs consistently produced budget savings or losses. In every site, however, for all sample members as well as for nongraduates and graduates, five-year government budget savings were greater, or losses were smaller, for the LFA programs. (See Table 2.) The differences between the gains or losses for the LFA and HCD programs within each site were quite large, particularly in Atlanta and Grand Rapids.

• Compared with the returns to government budgets per net dollar invested in previously studied welfare-to-work programs, the NEWWS programs’ government budget returns are similar, if not higher.

To facilitate comparisons with the benefit-cost results of previously studied welfare-to-work programs, an additional measure of the cost-effectiveness of the NEWWS programs from the government budget perspective was used. This measure, called the return to budget per net dollar invested, is calculated by dividing the gains (taxes and savings in transfer payments and associated administrative costs) by the total net costs of services. Using this metric, government budgets come out ahead if programs produce more than a dollar’s worth of additional revenues and savings for each dollar spent on employment-related services for program group members (compared with control group members).

Using this measure, the Portland program and the Grand Rapids LFA program both produced over $2.00 in increased revenue and savings for every additional dollar spent on program group members. The Riverside LFA program also produced a considerable return, $1.47 per dollar invested. The Grand Rapids HCD and the Columbus Integrated programs essentially caused the government to break even ($0.92 to $1.06). The Atlanta, Detroit, and Columbus Traditional programs were not as successful, returning considerably less than one dollar for each dollar invested, ranging from $0.41 in the Atlanta HCD program to just over $0.80 in the other programs. The average return across all the programs was $1.29.

On average, these results are more positive than those found in benefit-cost analyses of prior, recent programs. The California GAIN programs, for example, had returns to government budgets that ranged from a low of $0.17 per dollar invested (Tulare County) to a high of $2.84 (Riverside County). The average across all counties in the GAIN evaluation was $0.76. The return on investments in the two NEWWS programs that were most successful from a government budget perspective ― Portland ($2.83) and Grand Rapids LFA ($2.46) ― compare favorably with previously studied programs that had high government budget returns: the Riverside GAIN program and the mid-1980s San Diego Saturation Work Initiative Model (SWIM) program, which returned $2.84 and $2.34, respectively.

VIII. Effects on Family Circumstances and
Children’s Well-Being

No aspects of the welfare-to-work programs studied as part of the NEWWS Evaluation were designed to directly change family circumstances ― for example, to specifically affect marriage or fertility rates or to improve child well-being in specific ways. Theoretically, however, the programs could indirectly affect family circumstances or children through their impacts on such adult outcomes as educational attainment, employment, earnings, welfare status, and income. Data on five-year family and child outcomes are available for seven programs in four sites (Atlanta, Grand Rapids, Riverside, and Portland). The evaluation examined the net impacts on family circumstances and child well-being, that is, increases or decreases relative to the situations among control group members.

• As of the end of the five-year follow-up period, no programs increased or decreased adults’ or dependent children’s health care coverage. Only two programs ― the ones in Riverside ― increased the use of Transitional Medicaid at any point over five years.

From 70 percent to 80 percent of sample members had health care coverage at the end of follow-up year 5, and most of them had coverage through public sources, such as Medicaid or other public programs, rather than through private sources, such as employers’ health plans. All of those who did not have coverage had left welfare. (Many received Transitional Medicaid but were not able to find alternative coverage when their transitional benefits expired.) None of the programs had an impact on health care coverage for adults or children, although most programs, because they increased employment, led to a shift from public to private coverage. The two programs in Riverside increased the use of Transitional Medicaid during the foll