|
This report presents interim results for the Salem, Oregon, site in the national Employment Retention and Advancement (ERA) project. The ERA project was conceived and funded by the Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services and is also supported by the U.S. Department of Labor (DOL). The project is testing 16 innovative models across the country that aim to promote steady work and career advancement for current and former welfare recipients and other low-wage workers. A great deal is known about how to help these groups find jobs, but there are few proven strategies for promoting retention and advancement. MDRC, a nonprofit, nonpartisan research organization, is conducting the ERA evaluation under contract to ACF and is producing a similar interim report for each site in the project.
The Salem ERA program, which operated beginning in May 2002, targeted applicants of Temporary Assistance for Needy Families (TANF) cash assistance benefits who were unemployed. The program was designed to increase job placement, job quality, and advancement in the labor market for participants, who are mostly single mothers applying for welfare.
Origins and Context of the Salem ERA Program
Of the four ERA programs located in Oregon, the Salem ERA program, VISION (Valuing Individual Success and Increasing Opportunities Now), was the only one designed to begin working with clients prior to their employment. The program began with job search and placement services, and, once clients were employed, VISION continued to provide services intended to promote job retention and advancement.
As mandated by the State of Oregon, all TANF applicants, unless exempt, were required to participate in job search activities; participation in job search activities occurred concurrently with applicants’ eligibility determination. During a period that could last up to 45 days (called the “assessment period”), the regular welfare-to-work program, Job Opportunities and Basic Skills Training (JOBS), provided applicants with a series of basic job search and preparation workshops. Services were provided by staff from the local community college, Chemeketa, and were delivered at branch offices of the Salem Department of Human Services (DHS). The JOBS program embodied a “work-first” philosophy, and services ended once clients found jobs and DHS determined them to be “over income” for TANF assistance.
In Salem, the welfare, workforce, and community college systems had a long history of working together to deliver employment-related services. VISION built on these longstanding relationships and was created to provide applicants with more than the basic preemployment services provided by JOBS. Unlike that program, VISION was designed to help clients identify and work toward a job of interest, and services took place at the local One-Stop Career Center, rather than at the welfare office.[1]
Retention and advancement were emphasized from the onset of the VISION program, and the same staff worked with clients both before and after clients’ employment. The reasons behind this design were threefold: (1) to introduce the idea of job retention to clients prior to job placement, (2) to encourage clients to think about career goals and advancement early on, and (3) to establish a rapport that would ensure clients’ long-term engagement and, therefore, participation in VISION’s postemployment services. The goal for the postemployment phase was to help clients follow through on the goals that they had identified prior to their employment. The ERA Salem test determines whether such a model helped participants fare better in the labor market than those who went through the regular welfare-to-work program, JOBS.
VISION operated in a difficult environment. Oregon has consistently had a higher unemployment rate than the national average. During the time that VISION operated, Oregon’s economy was often named the “worst” in the country. Its economic environment impacted not only the type of job opportunities available to VISION’s clients but also the staffing capacity and cohesiveness of the program itself. The program had hoped to place many participants in jobs with the State of Oregon, Salem’s largest employer, but, due to budget shortfalls and a statewide hiring freeze, this goal never materialized. On the staffing side, VISION experienced a high rate of turnover and was constantly trying to integrate new staff and acquire their buy-in. As a result of the statewide hiring freeze, the program was unable to fill these vacancies in a timely manner.
The ERA Evaluation
As in the other ERA sites, MDRC is using a random assignment research design to assess the effectiveness of the VISION program. Starting in May 2002 and ending in May 2004, TANF applicants entering the two participating Salem DHS welfare offices who met the criteria for Oregon’s welfare-to-work services were assigned, at random, to either the VISION group or the JOBS group, with 50 percent being assigned to each group. Sample members were notified of their research status by the random assignment clerk, who explained the ERA study and gave clients written material describing the project.
MDRC is tracking both groups using data provided by the State of Oregon that show each individual’s monthly welfare and food stamp benefits and quarterly earnings in jobs covered by the state’s unemployment insurance (UI) program. One year of UI follow-up data is available for each person in the report’s analysis. One year of administrative records covering welfare and food stamp benefits is available for sample members who were randomly assigned through September 2003 (roughly 54 percent of the total sample). In addition, the ERA 12-Month Survey was administered to a subset of VISION and JOBS group members about one year after they entered the study.
Because individuals were assigned to the VISION and JOBS groups through a random process, the two groups were comparable at the start of the study. Thus, any differences that emerged between the groups during the study’s follow-up period can be attributed to the VISION program; such differences are known as the “impacts” of the program. A total of 1,504 single parents in the VISION and JOBS groups are included in this report’s analysis.
The ERA Target Population
The VISION program targeted low-income families with children who were TANF applicants and who were potentially eligible for TANF. People became eligible for the ERA program when they first applied for aid. Once randomly assigned, unless it was determined that they were exempt from participating in work-related activities, TANF applicants entered into a job search period (called an “assessment period” in Oregon), lasting up to 45 days. Approximately 83 percent of the TANF applicants in the research sample for this study were single-parent families at the time of random assignment, and the analysis in this report focuses on the single parents in this group. As is typical of TANF applicants, the sample members seemed fairly “job-ready”; nearly 50 percent of them had been employed for more than two of the prior three years. Most had received TANF assistance for less than two years, and 41 percent of the sample had never received it. Nearly 70 percent had a General Educational Development (GED) certificate or a higher degree of education. Most sample members are white and English-speaking. Not surprising for a sample of TANF applicants, most had young children when they entered the study.
Key Findings on Program Implementation
- During the preemployment period, VISION was successful in engaging the large majority of program-eligible clients. VISION’s additional preemployment workshops, its location at the One-Stop Career Center, and its ability to extend job search were the primary differences between it and JOBS.
VISION was successful in engaging the large majority of program-eligible clients during the preemployment period. However, since participants in both VISION and JOBS were held to the same job search participation requirements, the two programs had similarly high levels of engagement during this time. VISION created a set of preemployment workshops that were different from those in JOBS, inasmuch as they focused on career planning and job retention. Participation in the range of workshops was uneven, with a larger percentage of participants attending a few workshops but with few attending the full range of workshops available to them.
While the services at the One-Stop Career Center were also available to individuals in JOBS, the VISION participants had a stronger and more immediate connection to the center. VISION was physically located there, and VISION staff provided participants with direct assistance in using its resources, such as the computer room, the Oregon Labor Marketing Information System, résumé writing, mock interviews, and employer presentations. VISION staff could also access vocational training dollars by becoming co-enrolled in WIA, and VISION staff helped their customers through the enrollment process.
If there were promising prospects for a client, VISION staff had the option of extending job search beyond the 45-day period. In the end, VISION still operated within the parameters of welfare-to-work regulations, and since job opportunities were limited, clients often had to take “survival” jobs rather than jobs that had advancement opportunities or were in their field of interest.
- VISION staff struggled to develop and implement the postemployment retention and advancement component that made the program distinctive from JOBS. Even so, participants clearly appreciated staff’s support and their long-term message about advancement.
Each VISION client worked with a case manager, who was employed by DHS, and a training and employment specialist, employed by Chemeketa Community College. The VISION case manager was primarily responsible for eligibility determination, and the training and employment specialist was in charge of the client’s job search, job retention, and career advancement. VISION staff had a difficult time managing both employed and unemployed clients, and the bulk of their time was spent on preemployment responsibilities, particularly eligibility issues and individual crises. Staff often expressed frustrations about being crisis managers and, thus, about being unable to focus on the larger advancement goals of VISION. Turnover also contributed to the staff’s inability to move beyond preemployment concerns, because when there was a vacancy in the case manager positions, the training and employment specialists were forced to take on some of the administrative duties.
In addition, staff had a difficult time articulating what advancement should “look” like, and despite the various training that staff received on how to deliver retention and advancement services, staff reported that they did not have the skills or experience to develop and deliver advancement services. Consequently, VISION had few services to offer their working clients, and the program struggled to engage them.
Only when VISION was nearing its end and random assignment had stopped did staff have the time and resources to turn attention to working clients and the program’s postemployment objectives. At that point — essentially, when random assignment had ended and caseloads had stopped growing — staff began dividing into specialized positions and focused more intently on providing advancement services. As a result of the late focus on advancement, the services that were provided — such as individual meetings with the advancement training and employment specialist and “Career Night” — suffered from low engagement in general. In hindsight, most of the staff felt that VISION would have operated more efficiently had roles been specialized earlier.
Despite these challenges, participants repeatedly named VISION’s message about advancement and the staff’s customer service as the program’s greatest strengths. Participants stated that VISION staff recognized that getting a job was not an end in itself and encouraged them to pursue a desired career path, or a “field of fascination.” They clearly appreciated staff’s support and guidance, and once advancement meetings began, the strong staff-client rapport enabled the advancement training and employment specialist to better address clients’ potential barriers and on-the-job issues. Less clear was how VISION staff helped clients achieve their long-term career objectives.
In contrast, like most traditional welfare-to-work programs, JOBS conveyed a clear work-first message throughout its contacts with clients. Once clients were employed, the postemployment contacts — if there were any — dealt mainly with eligibility or supportive service issues.
- Welfare and community college staff entered VISION with different philosophies about the program’s policies and goals, which affected the delivery of services.
Delivering retention and advancement services within the parameters of the welfare-to-work system was a new and difficult pursuit. The two types of staff (which included five welfare case managers and four community college employment specialists) struggled to reconcile their different work cultures and philosophies, and welfare staff, in particular, had to make a large transition. They had to adjust to a customer-service, team environment, as opposed to one of process and regulations, and had to use “advancement” as the ultimate benchmark, rather than the work-first philosophy of the welfare system. These challenges affected team cohesiveness and service delivery, and they contributed to, and were exacerbated by, the high turnover of VISION case managers.
- The design of the VISION program helped link clients to public assistance.
While VISION operated under the same eligibility requirements for public assistance as JOBS did, VISION’s program design helped clients navigate the application processes and, therefore, provided better access to public assistance. Due to the program’s emphasis on customer service, a VISION case manager or training and employment specialist was readily available to deal with eligibility issues and to help clients complete applications. Such aid was particularly helpful in ensuring continued receipt of the Oregon Health Plan.
Since VISION staff had more long-term and frequent contact with participants than JOBS staff did, they were able to ensure that clients were at all times accessing the supportive services for which they were eligible. Finally, since VISION continued after clients were employed and served those who were no longer eligible for TANF, staff were able to connect clients to government benefits immediately, if at any point they became eligible (for example, due to a job loss).
Key Findings on Participation Impacts
- Both VISION and JOBS clients reported relatively high levels of contact with staff. However, those in VISION had more staff contacts, including a slightly greater number of in-person meetings.
While both research groups had extensive contact with program staff in the year following random assignment, the VISION program still produced an increase in the average number of staff-client contacts during this period. According to survey data, an individual in VISION had 22 contacts with program staff during the year, compared with 16 contacts for a member of the JOBS group. VISION group members were also slightly more likely to have in-person contacts. These impacts provide further evidence that VISION staff worked hard to reach out and be available to their clients.
- A higher proportion of individuals in VISION than in JOBS participated in job search activities.
VISION increased the proportion of clients who participated in group job search or job club during the year following random assignment: 64 percent of the VISION group did so, compared with 49 percent of the JOBS group. This difference is most likely attributed to the fact that VISION participants had access to the program’s preemployment workshops and had a stronger connection to the Career Center’s job search services. The difference might also reflect VISION’s focus on client engagement. The program’s additional workshops focused on job retention and career paths and were unique to VISION; the workshops were not available to JOBS participants.
In addition, VISION group members were engaged in job search for an average of nine weeks — three weeks longer than JOBS participants. This could be a result of VISION’s ability to extend the job search period if clients had promising job prospects.
- Individuals in VISION were more likely than those in JOBS to receive help with retention and advancement — particularly retention.
The VISION program increased the proportion of clients who received some kind of help with job retention or advancement: 34 percent of the VISION group reported receiving help in this area, compared with 21 percent of the JOBS group. This nearly 14 percentage point increase appears to be driven by retention-related assistance — assistance in dealing with on-the-job and personal problems that could affect job retention.
There was a small increase in the proportion of clients who received help with finding a better job while working: 9 percent of the VISION group reported this, compared with 4 percent of the JOBS group. Although this is encouraging, the difference is small.
Key Findings on Economic Impacts
- VISION did not generate statistically significant impacts on employment or earnings during Year 1.
The VISION program in Salem, compared with JOBS, did not generate any statistically significant impacts on employment or earnings outcomes during the first year after clients entered the program (Table ES.1). In a typical quarter, about 40 percent of VISION group members were employed, compared with 42 percent of JOBS group members. Year 1 earnings for VISION group members are estimated to be about $270 below the JOBS group average of $4,100, but this difference is not statistically significant. By the last quarter of the follow-up period, the program was still not generating increases in earnings or employment. VISION, compared with JOBS, had no effects on employment and earnings across a variety of subgroups and cohorts. The program also had no effect on various measures of overall employment, as reported by respondents to the ERA 12-Month Survey.
- VISION increased public assistance receipt but had no statistically significant effect on total income.
Table ES.2 shows the impacts of VISION on public assistance receipt and income. The ERA program in Salem increased the percentage of clients receiving public assistance. The implementation research found that VISION increased the likelihood that TANF applications would be accepted, because staff maintained contact with clients longer and actively helped clients navigate the application requirements. Year 1 TANF payments were $365 (or 19 percent) higher among VISION group members, compared with the JOBS group average of $1,954. VISION group members were almost 6 percentage points more likely to have ever received TANF than JOBS group members. VISION group members also received $160 more in food stamp payments than the JOBS group average of $2,579 in Year 1. The increase in food stamp receipt is a positive impact — particularly in Oregon, which was considered one of the “hungriest” states in the nation when VISION started. VISION also increased public (and overall) health insurance coverage, which could be a byproduct of increasing TANF receipt and overall efforts to help clients with eligibility issues.
Conclusions and Policy Implications
Though a one-year follow-up period is far from definitive, so far the employment, retention, and advancement impacts of the VISION program are not encouraging. Overall, it seems that a combination of factors made it difficult for VISION to achieve its goals.
- Program operators indicated that a pre- to postemployment model of retention and advancement was difficult to implement. Furthermore, staff had a difficult time understanding how to operationalize advancement in the context of everyday services.
The program model appears to require a great deal of specialization and staff. Making the same staff responsible for both pre- and postemployment issues may have been possible only with very small caseloads. With large caseloads, job placement and reemployment naturally take priority over job retention and career advancement.
VISION’s program design would likely have been more successful had staff been equally comfortable and skilled in providing both the retention and the advancement service components. The postemployment component was most fully implemented late in the evaluation period, and there were some increases in participation in retention and advancement services. Thus, VISION may have benefited if staff had first had a chance to understand and gain experience in the less familiar advancement component. This might have encouraged VISION case managers to buy into the concept of advancement in general and might have allowed the training and employment specialists to build confidence in their career-coaching abilities.
- A challenging external environment can prevent a program model from being implemented as intended.
A weak economy, state staffing cutbacks, staff turnover, a hiring freeze, and management staff whose time was divided among a number of competing duties impacted the VISION program throughout the evaluation. The hiring freeze affected both clients and caseworkers. Clients were unable to find jobs in the public sector — a major employer of former welfare recipients in Salem — and staff became overextended and burned-out as program positions went unfilled. In addition, VISION was evaluated against a competing program, JOBS, that had a strong work-first focus. While it may be that, in difficult external circumstances, a program like VISION is unlikely to generate impacts, other ERA sites that have attempted to work with clients both before and after employment have also had serious operational difficulties, even under better circumstances.
- It is important to account for institutional differences when merging line staff of partner organizations.
Conflicting work cultures and philosophies came into play when merging community college and welfare staff in the Salem ERA program. VISION managers not only had to align partner organizations but also had to account for the different environments and operating cultures to which staff were accustomed. Facilitating communication and establishing clearer program boundaries up-front may have helped.
* * *
It is likely that some combination of the factors discussed above contributed to the weak impacts of the VISION program, compared with the JOBS program, to date. However, these results are not the final word on the ERA program in Salem. MDRC will continue to track the impacts of VISION using unemployment insurance records.
|
