| I.
Introduction: The Goals and Rationale of Jobs-Plus
Jobs-Plus is a new
way for large numbers of people who live in public housing to get and
keep jobs, making their communities better places to live. It is being
launched as a research demonstration project in seven cities, where its
success will be carefully studied along a variety of dimensions. The cities
are: Baltimore, Chattanooga, Cleveland, Dayton, Los Angeles (two developments),
St. Paul, and Seattle.
The public housing
developments selected for the project are places where work is rare and
welfare receipt widespread. Jobs Plus will attempt to increase work dramatically
by providing all working-age residents with state-of-the art employment
and training services, financial incentives, and social supports for work.
It will strive to help those not working enter work, those working inconsistently
work more consistently, and those with low-paying jobs move on to better-paying
jobs and climb a career ladder. Through this focus on work, Jobs-Plus
will address important issues concerning community revitalization, the
future of public housing, and welfare reform, thus promising to make the
lessons from the demonstration broadly relevant.
The main purpose of
this paper is to provide an overview of the demonstration’s comprehensive
research agenda. It turns to that task after taking a closer look at the
main objectives and rationale of Jobs-Plus and how it will attempt to
achieve its ambitious goals.
Community revitalization.
In its broadest formulation, Jobs-Plus represents a new attempt at
achieving a long-standing though elusive objective of urban public policy:
creating mixed-income urban communities in which most working-age residents
are employed, and doing so without displacing poor people. It is widely
accepted that such communities offer poor families a better quality of
life and better future prospects than do communities with very high concentrations
of non-working poor people. Consequently, many believe that increasing
work among residents is key to revitalizing inner-city communities, an
idea advanced most prominently in recent years by William Julius Wilson
(for example, in his 1996 book, When Work Disappears). Jobs-Plus
is premised on this theory and, as such, it offers a rare opportunity
to test whether whole neighborhoods can indeed be transformed through
a work-focused initiative.
The most immediate
program objective is to transform the lives of residents of public housing.
However, because public housing complexes occupy a central and often defining
role in many urban neighborhoods — all too often serving as a social
and economic drag on those communities — it is hoped that positive
changes within the developments will eventually have spillover effects,
leading to improvements in the surrounding neighborhoods as well.
Public housing
reform. Increasing and sustaining employment among public housing
residents may be critically important not only for making public housing
communities better places in which to live, but also for ensuring the
future viability of public housing as a source of decent and affordable
housing for low-income families.
Nationally, almost
half of public housing recipients rely on some form of cash welfare benefits
as their primary source of income. New restrictions on access to these
benefits, owing to recent federal and state welfare reform legislation,
may lower many residents’ ability to pay rent. At the same time,
shrinking federal operating subsidies to local public housing authorities
(PHAs) will make it harder for them to fill that budget gap. The resulting
financial strain could threaten PHAs’ very solvency, making it imperative
for residents to increase their incomes through work.
Increasing residents’
employment and earnings may also help more of them "climb the housing
ladder" toward better, unsubsidized rental housing, or even home
ownership. (For most people, steady employment and regular earnings are
prerequisites for home ownership.) A rise in this "positive"
turnover would then allow the government, over time, to extend public
housing assistance to a greater share of the eligible low-income population,
even if the supply of public housing remained constant.
It is also possible
that the public’s willingness to continue supporting public housing
may increasingly depend on boosting residents’ efforts to work. That
would mirror a factor shaping important welfare and Food Stamp reforms,
which in recent years have toughened the obligation of recipients to seek
or prepare for work as a condition of receiving full benefits.
Welfare reform.
The Jobs-Plus housing developments are likely to include a disproportionate
share of longer-term welfare recipients who presumably are among the more
difficult-to-serve portion of the welfare caseload. Consequently, the
program’s success in moving them into work should be as much of a
concern to welfare administrators as it is to public housing administrators.
Jobs-Plus offers an opportunity to test a new set of strategies for responding
to many of the challenges posed by the dramatic welfare changes now underway
across the nation.
The program design
for Jobs-Plus will build on the lessons learned from the most successful
welfare-to-work programs of the past, but also address many of their shortcomings
in the hope of achieving much greater success. Many of the same strategies
it uses may also be suitable for welfare-to-work initiatives outside of
public housing, making it important to learn as much as possible about
their feasibility and effectiveness in Jobs-Plus.
II.
The Jobs-Plus Approach
Jobs-Plus is distinctive
in the way that it builds on but also moves beyond past self-sufficiency
initiatives. (See the box below for a summary of the program’s key
features.) First, the program adopts "community-building" principles
common to a growing number of comprehensive neighborhood improvement initiatives.
These include the principles of local control, collaborative decision-making,
resident empowerment, and building on residents’ pre-existing social
assets. The demonstration thus places the burden of detailed program design
and implementation squarely and collectively on the shoulders of those
who should have the biggest stake in the success of the program —
the public agencies that serve public housing residents (especially the
PHA, the welfare department, and employment and training agencies) and
the residents themselves. They are to build local partnerships —
or "collaboratives" — for Jobs-Plus.
One important rationale
for local collaboration is the assumption that no single entity can accomplish
by itself the profound changes that Jobs-Plus aspires to achieve. The
problems have no single cause and no single solution, so agencies and
residents must pool their knowledge, resources, and energies in order
to be most effective. A high premium is placed on resident involvement
in these collaboratives. The logic here is that in order to build a program
that enjoys widespread resident support and "buy-in," residents’
own perspectives on what will work in their community must be understood
and given great weight in deliberations over program design. It is also
assumed that, if the program works, it has a much greater chance of being
sustained and institutionalized as part of ongoing policy if a broad set
of local institutions themselves come to feel a sense of mutual ownership
over and stake in the program’s long-term viability.
Of course, partnerships
like this, which require agencies that do not normally work together to
begin doing so in intimate ways and share decision-making power with residents,
do not just happen — and do not happen quickly. Recognizing this,
MDRC and other outside experts are providing intensive technical assistance
to the local collaboratives to help them build strong partnerships. In
part, this assistance aims to help residents strengthen their capacity
to contribute to program design and implementation decisions and to exercise
influence within the collaboratives. It also aims to help the agency partners
become supportive of and receptive to that empowerment of residents, guide
the agencies in instituting changes in their own practices on behalf of
Jobs-Plus, and help the full group of partners design an intervention
that builds productively on social assets that already exist among residents.
A second distinctive
aspect of Jobs-Plus is its plan to integrate state-of-the-art employment-related
services with financial work incentives that "make work pay."
It will add to this mix a "community support for work" component
that aims to strengthen and expand the social capital on which residents
can draw to support their efforts to prepare for, find, and keep jobs.
Previous research from welfare-to-work programs suggests that neither
services nor incentives alone could reasonably be expected to enable
sites to increase employment and earnings dramatically. It is possible,
however, that successful simultaneous implementation of services
and incentives, in combination with new forms of social support, might
create a synergy among the three strategies, with effects that far exceed
the impacts of even the best-performing programs of the past. That, at
least, is a central theory of Jobs-Plus.
A third distinctive
aspect of Jobs-Plus is its neighborhood focus and scale. Unlike past welfare-to-work
and job training programs, Jobs-Plus is a "place-based, saturation-level"
initiative. It will attempt to provide innovative services, incentives,
and supports to all working-age residents living in the targeted
housing development, with the explicit goal of transforming those locations
from "low work/high welfare" communities into "high work/low
welfare" communities.
III.
Evaluation Strategies
The distinctive features
of Jobs-Plus offer an unusual opportunity to learn about the challenges
and payoff of concentrating an innovative, multi-component, saturation
employment initiative in delimited geographic areas with high concentration
of welfare recipients. The demonstration thus includes a comprehensive
evaluation of the program.
The evaluation will
assess the program’s feasibility, effectiveness, economic costs and
benefits, and implications for best practices. In contrast to other evaluations
of comprehensive community initiatives — which are usually single-site
studies without the benefit of comparison groups — this study will
have multiple sites with randomly selected comparison groups in each of
them. Within each of the seven demonstration cities, several housing developments
that were reasonably well-matched on demographic and other criteria were
identified as being potential candidates for hosting Jobs-Plus. One of
these developments was then randomly selected to be the Jobs-Plus development,
and one or two were randomly chosen to be comparison sites. Across the
demonstration cities, then, the evaluation takes the form of a social
experiment with relatively comparable treatment and control groups.
This design was adopted
as an alternative to a traditional social experiment in which individuals
are randomly assigned. Such a strategy was precluded by the place-based
and "saturation" aspects of the Jobs-Plus treatment. The fact
that the program will attempt to assist all working-age residents
in each city’s Jobs-Plus housing development rules out the option
of randomly assigning some residents in the same development to a control
group that would remain "untouched" by Jobs-Plus. The same kind
of constraint has been a major impediment to determining reliably the
effectiveness of other comprehensive community initiatives.
Table
1 shows the Jobs-Plus developments, their size, and some characteristics
of their residents. Particularly noteworthy is the substantial degree
of ethnic and racial diversity across the demonstration sites. This is
important because many high-poverty housing developments and neighborhoods
throughout the country are themselves racial and ethnic enclaves. The
opportunity to test Jobs-Plus in such a variety of settings thus enhances
the policy relevance of demonstration.
Appendix
Table 1 compares the Jobs-Plus and comparison developments. It shows
that within each city the background characteristics of the residents
living in the Jobs-Plus development are similar to those of residents
living in the comparison developments (although the match is not perfect).
This will strengthen those parts of the evaluation in which interim and
longer-term outcomes will be compared across these two sets of developments.
The evaluation will
include implementation, impact, and benefit-cost studies. The implementation
study will examine what it takes to build and operate effective collaborations
and the kind of complex program treatment envisioned for the demonstration.
In addition, it will attempt to understand the underlying theories or
hypotheses guiding the choices each site makes in specifying its Jobs-Plus
intervention. It will also examine the sites’ overall success in
operationalizing the central elements of the basic Jobs-Plus model. If
that implementation success varies across the sites, the study will try
to explain why it varies. The implementation study will use data from
several sources. Most important will be information collected through
intensive field research at each of the treatment and comparison developments,
surveys of residents in those developments (which will also be used for
the impact analysis), and various city, housing authority, and other agency
records and documents.
The impact study
will determine the effectiveness of Jobs-Plus in increasing residents’
employment and earnings, reducing their use of welfare, and improving
their personal and family well-being and quality of life. At the core
of this study will be a comparative interrupted time-series analysis involving
working-age residents in roughly 8,000 households (in the treatment and
comparison developments combined). Applying this technique, trends in
employment, earnings, welfare receipt and Food Stamp receipt will be measured
for the Jobs-Plus and comparison developments using administrative records.
These trends will be examined for a period beginning five years before
the start of Jobs-Plus and continuing for five years afterward. If, for
example, residents’ employment after Jobs-Plus deviates sharply from
their historical trend in a positive direction, and do so to a markedly
greater extent than residents in the comparison developments deviate from
their own historical trend, this will lend credibility to a conclusion
that Jobs-Plus — and not other factors — caused the increase
in employment. The same logic would apply in analyzing trends in other
outcomes. (For a detailed discussion of the impact research design, see
Bloom, 1996.)
The impact study (like
the implementation analysis) will also use data collected from "before-after"
surveys of a subsample of residents in the treatment and comparison developments.
The surveys will collect information on a wide variety of topics not only
to help gauge and explain the sites’ success (or failure) in implementing
Jobs-Plus, but also to help assess the program’s effects on various
employment, welfare, and quality-of-life outcomes. Other
data, such as neighborhood indicators (e.g., local crime statistics) may
also be used in the impact (and implementation) study.
The benefit-cost
study will estimate the cost of operating Jobs-Plus and analyze the
net economic gain or loss created by the program from the several different
perspectives. For example, it will examine the net gain or loss from the
vantage point of public housing residents, government budgets and taxpayers,
specific agency budgets, and society at large. The benefit-cost study
will make use of data from a variety of sources, including the information
collected for the implementation and impact studies and government expenditure
data.
Overall, the Jobs-Plus
demonstration’s combination of randomly selected treatment and comparison
sites, long-term trend data, before-after survey data, and comprehensive
qualitative information will provide an unusually powerful foundation
for addressing important theoretical and policy questions that are relevant
even beyond the world of public housing. The next few sections discuss
these questions and hypotheses in more detail.
IV.
Evaluating the Implementation Success of Jobs-Plus
Jobs-Plus presents
a number of extraordinary implementation challenges, especially in creating
collaborations, delivering a bold new program treatment that differs profoundly
in scale and content from the services, incentives, and supports normally
available to public housing residents, and in fostering enduring institutional
change within the demonstration cities. The implementation study will
therefore examine how well the sites meet these challenges. Diagram
1 illustrates the categories of variables and the anticipated relationships
between them that this part of the evaluation will study. Attention here
will focus largely on tests of hypotheses in three main areas, as described
below.
A. Testing the feasibility and
robustness of inter-agency and resident partnerships
Hypothesis: True and
effective inter-agency and resident partnerships can be established
across very diverse settings and resident populations
Many other "comprehensive
community initiatives" involve inter-agency collaboration at the
local level. However, Jobs-Plus is unusual in bringing together agency
representatives and residents (a rare combination in itself) for the explicit
purpose of achieving employment goals. The agencies involved already
serve public housing residents in some capacity, but separately, and for
different purposes. Never before in these communities have they all been
included as joint stakeholders in a place-based, employment-focused initiative.
Together they are entering uncharted waters.
The expectation
that a collaborative approach, though difficult, is broadly feasible is
based on several assumptions. One is a belief that although different
agencies serve public housing residents for different purposes, welfare
reform and budget constraints push them in a common direction toward the
goal of increasing self-sufficiency among the people they serve. In other
words, each agency’s success in achieving its own institutional mission
may be enhanced by the extent to which residents’ employment and
earnings are raised. Thus, the agencies should have a mutual stake in
success of Jobs-Plus as a potential vehicle for achieving critical employment
objectives they are coming to share for their overlapping caseloads. Working
together on this project may allow them to pool their strengths and avoid
individual agency actions that work at cross-purposes.
Another assumption
behind the expected feasibility of the collaborative model is that all
agency partners will welcome the special contributions that residents
themselves can make as partners in designing and managing an intervention
that seeks to transform their lives and communities. Finally, the
demonstration’s provisions for technical assistance to help the sites
forge strong partnerships may also contribute to the collaboratives’
success.
Still, the challenge
facing the local collaboratives is profound, and their ability to evolve
into true and effective partnerships is by no means assured. Furthermore,
the magnitude of the challenge may vary tremendously across demonstration
sites, which not only serve very different types of public housing populations,
as noted above, but which also have different institutional and political
histories on which to build. In some cities, for example, the partners
have been part of earlier joint enterprises for other purposes, while
in other cities the partners have little shared experience on which to
draw. Thus, the most important question about the collaborative approach
is not whether it can be fully achieved in any of the demonstration
cities, but whether it can be achieved in most of them. In other
words, is it broadly feasible, or "robust"?
In addressing this
question, the evaluation will need to define and measure the concept of
"true and effective collaboration" and the pre-existing conditions
that, theoretically, could affect the sites’ success in achieving
that goal. Many of these conditions would fall within the areas listed
on the far left column of Diagram 1. If the evaluation
finds that true partnerships emerge across the sites despite the vast
differences in local circumstances and populations, that will be taken
as partial support for the hypothesis that the collaborative model is
broadly feasible. If success varies, the analysis will examine what conditions
appear to facilitate or impede success. In doing so, it will assess, through
cross-site comparisons and in-depth analysis, whether differences in some
of the specific conditions listed in Diagram 1
(e.g., pre-existing institutional relationships, local context, role of
the external change agents — MDRC and other national and local technical
assistance providers) help explain the variation in successful collaboration.
In addition, it will attempt to understand the processes or practices
of collaborating adopted at each site (e.g., governance structures created
for Jobs-Plus, the frequency of meetings and how they are conducted, the
roles assigned to various members of the collaborative and how they execute
those responsibilities, and the emergence of strong internal leadership
and change agents within the collaboratives) influence the success of
building strong and effective partnerships.
Other major work-focused
neighborhood revitalization efforts are likely to require similar kinds
of partnerships. Consequently, determining whether or not the collaborative
model can work across very diverse settings and populations, and learning
what it takes to achieve real collaboration, may shed light on how to
promote and foster a similar approach to program design and management
in other welfare reform and inner-city revitalization initiatives.
B. Testing the feasibility and
operational robustness of the Jobs-Plus program design
Hypothesis: The key
programmatic features of Jobs-Plus can be successfully implemented across
diverse settings and for different types of people.
At the programmatic
level, sites can be judged as having succeeded in implementing Jobs-Plus
if they are able to: (1) ensure that all working age residents have access
to the program’s services, incentives, and community support for
work (in keeping with the goal of "saturation"), (2) ensure
that those three components are implemented in ways that are mutually
reinforcing and well-coordinated, (3) ensure that, overall, the services,
incentives, and supports, and the levels at which they are made available,
amount to a "treatment" that differs very substantially from
what is normally available (as indicated by what is available to residents
in the comparison developments), and (4) actually engage residents in
work-promoting activities and work experience opportunities at substantially
higher levels than normally occur (again, as indicated by the experiences
of residents in the comparison developments). The evaluation will therefore
assess how well the sites’ programs meet these criteria.
Achieving all of
these implementation goals will be a tremendous challenge. Although the
size of the program in each site will be small relative to city- or county-wide
welfare-to-work initiatives, it will be vastly more complex because of
the multi-pronged nature of the intervention, its emphasis on saturation,
and its targeting of a diverse and reputedly difficult-to-serve population.
Some sites, of course, may perform much better than others. For example,
in some sites, pre-existing employment and training services or social
support networks or other forms of social capital (e.g., a tradition of
volunteerism in the developments or strong resident organizations) may
offer a ready foundation on which to construct new activities and expanded
community support for work in Jobs-Plus; in other sites, these conditions
may exist in much more rudimentary forms. (See Diagram
1.) Also, differences in residents’ pre-existing employment barriers
and assets (e.g. , their existing education and skill levels and attitudes
toward work and welfare) and their past experiences with training programs
may affect how they initially react to Jobs-Plus — for example, whether
they view the program’s services and financial incentives cynically
or enthusiastically.
The substantial
racial and ethnic diversity across the sites may be associated with the
variation in some pre-existing conditions that may affect the implementation
of the program. For example, in some sites, pre-existing social networks
among residents may be strong within but not across different racial and
ethnic groups. In general, it will be important to examine the differences
in social networks, cultural patterns, vulnerability to discrimination,
and other conditions associated with racial and ethnic identity and to
assess whether and how these "matter" for engaging different
groups in the program. The findings might also suggest how program strategies
might need to be adapted in order to accommodate important group differences.
At the same time,
it is hoped that successful implementation can be achieved in all sites
despite the variation in advantages and disadvantages with which they
start. An important reason for optimism can found in the underlying rationale
for adopting a collaborative approach to program design and implementation.
By investing responsibility for designing the specific ways in which the
program’s services, incentives, and social supports are operationalized
in the local partners, and by making sure that residents’ own perspectives
and insights get heard and have influence, chances are greater that the
program that emerges will be well-suited to the special circumstances
and types of people in each site.
In addition, it
is hoped that these local partners will benefit from external technical
assistance in designing and implementing their programs. Here again, the
role of external change agents (MDRC and other national or local technical
assistance experts) may be key in guiding the local collaboratives in
making sound choices, informed by lessons — positive and negative
— from many past welfare-to-work and job training programs. The implementation
analysis will therefore examine the role and contributions of these groups
in the process of building local Jobs-Plus programs (and in strengthening
their local capacity more generically, as discussed in the next section).
Still, success
here is by no means assured, and assessing the broad feasibility of the
overall Jobs-Plus program model is an important research goal. If it happens
that implementation success varies greatly across the sites, the evaluation
will then investigate whether particular pre-existing conditions at the
sites and other factors help explain that variation.
C. Testing institutional
change
Hypothesis: Successful
implementation of Jobs-Plus can produce broad and enduring institutional
change, in terms of inter-agency relationships, resident empowerment,
and local capacity.
The Jobs-Plus demonstration
is striving to produce institutional change in local communities that
encompasses but also transcends the Jobs-Plus program itself. (This is
illustrated in moving from the middle box to the final box on the right
side of Diagram 1.) Indeed, even if the Jobs-Plus
program fails to achieve its proximate goals of improving residents’
employment and quality of life, it might succeed in improving "how
policy business gets done" at the local level, which is an important
broader objective of the demonstration’s community-building agenda.
As one example,
some agencies, especially the welfare department and PHA, may take steps
to ensure that their broader policies or administrative procedures are
better coordinated in order to minimize the chances of working at cross-purposes,
increase agency efficiency, and improve service delivery for their common
caseloads. Moreover, both agencies may include residents who are welfare
recipients more directly and routinely in these deliberations. As another
example, they and other Jobs-Plus partners may together undertake new
initiatives aimed at community economic development, local workforce development,
crime reductions, urban infrastructure planning, and other quality-of-life
improvements.
The Jobs-Plus field
research will thus study agency and resident relationships to determine
whether the local stakeholders are on a path toward institutionalizing
the process of collaborative decisionmaking they engaged in for Jobs-Plus.
In addition, it will assess whether the local stakeholders are likely
to emerge from the demonstration with a greater generic capacity
to work together on important future initiatives to build stronger communities.
Other important
evidence of institutional change would be a finding that the program (if
shown to be effective) evolves toward a more permanent status. The evaluation
will determine whether Jobs-Plus is itself institutionalized, under the
continuing guidance of the inter-agency and resident collaboratives, after
MDRC’s and others’ technical assistance concludes and as the
demonstration period itself ends.
In addition to
understanding whether the sites endure these enduring institutional
changes, it is important to learn how such change is achieved or
why it is not achieved. This will involve studying the roles and
strategies of members of the collaboratives and their local and national
technical assistance partners and the institutional and political environment
in which they work. For example, do "visionary" leaders or change
agents emerge within the collaboratives and effectively advance the goals
of Jobs-Plus? Are alliances built with powerful political allies to help
foster institutional change? Do reform impulses and goals across different
government agencies coincide or clash? Do bureaucratic structures and
processes impede or facilitate change? Do external change agents play
a significant role in advancing the process of change?
V.
Evaluating the Effects of Jobs-Plus on Work Outcomes
A fundamental question
for the Jobs-Plus demonstration is: Does the program dramatically increase
employment and earnings among public housing residents? Through a number
of innovative analyses, the evaluation hopes to provide reliable evidence
for judging the effectiveness of the program in achieving this central
goal.
A. Testing the power and robustness
of Jobs-Plus to increase employment and earnings at the individual level
Hypothesis: Taken as
a whole, and when all elements of the model are successfully implemented,
Jobs-Plus can dramatically increase employment and earnings across diverse
settings and for different types of people.
As previously discussed,
it is hoped that the distinctive features of Jobs-Plus will allow it to
produce effects on employment and earnings that surpass the more modest
achievements of even the most successful past welfare-to-work and job
training programs. The first-order objective of the impact evaluation
is, then, to determine whether this goal is achieved. Using the comparative
interrupted time series analysis mentioned above, part of the impact study
will measure the program’s employment effects on the individuals
and households residing in the Jobs-Plus developments at the start of
the program, regardless of whether they continue living there over the
course of the evaluation follow-up period. This analysis will determine
whether Jobs-Plus represents a viable means of transforming public housing
developments into places that launch residents into work, help them remain
employed, and increase their earnings.
Part of the impact
analysis will measure program effects for all seven demonstration cities
combined (i.e., a "pooled" analysis). This will show whether,
on average, living in a Jobs-Plus housing development increased employment
and earnings for residents by more than would have occurred had those
residents not lived in a Jobs-Plus development. However, average effects
can mask important variations in results. Equally important for the evaluation
is to determine whether Jobs-Plus is broadly effective or robust. Can
big employment effects be achieved across a number of demonstration cities,
despite the vast differences in the types of people living in public housing
and the types of labor markets and other local conditions found across
those cities and neighborhoods? Of particular interest from the perspective
of welfare reform is whether Jobs-Plus is effective for the subgroup of
residents who are welfare recipients.
It is expected
that for Jobs-Plus to have large, positive effects on employment and earnings,
it must be successfully implemented in terms of the criteria described
in the foregoing section. As illustrated in Diagram
2, successful implementation of Jobs-Plus should lead to an increase
in the work preparedness of residents in the Jobs-Plus development (relative
to their counterparts in the control developments). The program’s
outreach to employers is also expected to increase employer demand for
Jobs-Plus residents and, hence, residents’ access to jobs. Together
these effects should yield positive impacts on a variety of work-related
outcomes, as shown in the box on the right side of Diagram
2 (e.g., employment rates, employment duration, earnings, and job
quality).
If the implementation
of Jobs-Plus or its employment impacts are extremely uneven across the
sites, the evaluation will examine whether the sites that were most successful
in implementing the program also produced the biggest impacts. Such a
correspondence would support the hypothesis that large employment effects
may indeed depend on successful implementation of Jobs-Plus in the ways
that have been defined for this evaluation.
The evaluation
may, of course, find that even with successful implementation Jobs-Plus
does not succeed in increasing employment outcomes. If so, this would
be important evidence against the theory underlying Jobs-Plus, which posits
that because of its distinctive features the program will be unusually
effective.
B. Testing the
"synergy" hypothesis
Hypothesis: The successful
implementation of all three Jobs-Plus program elements — saturation-level
services, incentives, and supports — is a pre-requisite
for producing big effects on employment outcomes.
Fundamental to
the Jobs-Plus approach is its emphasis on combining and coordinating three
different treatment components, each of which addresses a different category
of work impediments: inadequate preparation for work or work-search skills;
inadequate financial rewards for work; and inadequate social support for
work. The theory behind this approach is that all of these work impediments
must be addressed, and that when all three treatment components are implemented
together, each will strengthen the effectiveness of the others. It is
thus predicted that through a form of synergy between the program’s
services, incentives, and supports, Jobs-Plus will produce much more powerful
effects than would occur if one or more of these treatments were omitted.
In short, all three of these features are assumed to be vital.
If the demonstration
sites uniformly succeed in implementing all three components, and if the
program also produces uniformly large employment impacts, it would leave
open the possibility that all three components were indeed making an important
contribution. Unfortunately, though, it would not rule out the alternative
possibility that only one or two of the components, or perhaps other features
of Jobs-Plus, were entirely responsible for the results.
The evaluation
research design will not permit a rigorous test of the relative effects
of different program components. However, depending on the pattern of
actual implementation and impact findings across the sites, it may still
be possible to address the synergy hypothesis in an informative way. If,
for example, some sites do not fully implement one or two of the three
components but, nonetheless, do succeed in producing large impacts that
rival those where all three features are fully implemented, that would
be evidence against the hypothesis that all three components are vital.
Alternatively, if the only sites that produce large impacts are those
that fully implement the program’s services, incentives, and supports,
that would lend weight in favor of the hypothesis that combining the three
components is indeed essential to the effectiveness of Jobs-Plus.
C. Testing a
hypothesized causal chain
Hypothesis:
Jobs-Plus can produce big effects on work if it substantially increases
residents’ "preparedness for work" and efforts to look
for work.
It is expected
that if Jobs-Plus achieves large employment impacts, it will do so by
changing certain work-related attributes, attitudes, or behaviors among
residents. For example, it may increase their diligence in searching for
employment, their knowledge about how best to go about this task, their
occupational or basic skills or work habits, their education or training
credentials, or their commitment to working. Moreover, it is predicted
that these types of changes will follow directly from a fully successful
implementation of Jobs-Plus. (See Diagram 2.)
The impact study,
relying largely on survey data, will attempt to measure whether such changes
are indeed occurring, whether they are larger in those sites where employment
impacts are larger (assuming variation in impacts occurs), and whether
they occur prior to employment outcomes. If evidence can be generated
to support this hypothesized causal chain leading from the implementation
of Jobs-Plus to work-preparedness impacts and then to work impacts, it
will lend credibility to the overall conclusion linking Jobs-Plus to any
big changes observed in residents’ employment outcomes.
An alternative
— or complementary — causal chain involving "employer demand"
may also be relevant. The demonstration sites are likely to undertake
special efforts to build linkages to private sector employers, such as
through systematic job development (e.g., where program staff try to locate
employers with open positions that might be suitable for Jobs-Plus participants),
or where new job opportunities are created explicitly for Jobs-Plus residents
by a network of prominent employers in the community (such as those involved
in some form of business with the housing authority or other partners
in the collaborative) with whom the program establishes special relationships.
Jobs-Plus sites may also establish "community service jobs,"
paid for with public and private funds, to create employment opportunities
for residents who have difficulty qualifying for unsubsidized jobs. As
previously mentioned, these and other efforts may directly increase "employer
demand" for workers from the Jobs-Plus developments. In turn, this
increased demand might by itself cause substantial employment and earnings
impacts. Or (and perhaps more realistically) it may be that a program-induced
increase in employer demand may only contribute to employment impacts
if the program also increases residents’ preparedness for work, so
that residents are better able to take advantage of those new opportunities.
Given the importance
that sites are likely to place on employer outreach in the Jobs-Plus demonstration,
the evaluation will describe the intensity and productivity of their efforts
to locate and stimulate job opportunities for residents.
D. Testing whether Jobs-Plus
can transform public housing developments into communities of workers
Hypothesis: Jobs-Plus
can help ensure that a dramatically larger share of the population living
in public housing at any given time will be working at that time.
As previously mentioned,
it is hoped that Jobs-Plus increases the long-term employment and earnings
of residents living in Jobs-Plus housing developments at the start of
the demonstration, regardless of whether they continue living in those
developments. It is also hoped that the program increases the proportion
of residents who at any given time are employed while living in the Jobs-Plus
developments. Achieving the first objective — big employment and
earnings effects on individuals (or households) — does not necessarily
ensure achieving the second — big impacts on developments. This is
because many of the residents who benefit from Jobs-Plus may choose to
move out of their public housing development after becoming employed.
The impact analysis
will therefore include a second level of inquiry that focuses on the effects
of Jobs-Plus in which the development is the unit of analysis. For example,
it will measure the proportion of residents employed at specific points
in time (e.g., annually) among all residents living in the Jobs-Plus development
at that same time. It will examine how much this proportion changes over
time, starting five years before the implementation of Jobs-Plus and continuing
for five years afterward. It will determine whether the proportion employed
rises sharply and substantially during the five-year follow-up period,
and, if so, whether that departure from the past trend is much sharper
and much more substantial than any departure from trends observed in the
comparison developments. Such a pattern would support a conclusion that
Jobs-Plus caused that large employment effect for the development itself.
(See Bloom, 1996. A similar development-level analysis will be applied
to welfare and other outcome data.)
Of course, an influx
over time of new residents who were already working before moving into
the Jobs-Plus developments could cause development-level employment rates
to grow even in the absence of any effect of Jobs-Plus. Fortunately, the
individual-level analysis along with data on resident turnover and the
employment status of new residents will make it possible to disentangle
the contribution of Jobs-Plus from the influence of a change in the resident
population in accounting for increases in employment and earnings at the
development level.
VI.
Evaluating the Effects of Jobs-Plus on Residents’ Reliance on Welfare
and Food Stamps
A. Testing the impact of Jobs-Plus
on residents’ use of welfare and Food Stamps at the individual and
development levels of analysis
Hypothesis: Dramatically
increasing the employment and earnings of residents will substantially
reduce residents’ reliance on welfare and Food Stamps.
If Jobs-Plus succeeds
in increasing residents’ employment and earnings, these effects should,
in turn, reduce residents’ use of welfare and Food Stamps, as illustrated
in Diagram 3. The evaluation will study the effects
of the program on these outcomes at both the individual (or household)
level and the development level, following the strategies referred to
previously for the analysis of the program’s impacts on work outcomes.
The welfare and Food Stamps outcomes will be measured primarily with administrative
records data. Outcomes such as the percentage of residents receiving TANF
payments, the average amount of TANF payments received, the average number
of months receiving TANF benefits, and similar outcomes for Food Stamps
will be measured. Survey data will allow some assessment of the program’s
effects on the receipt of other transfer benefits, such as state general
assistance payments and Supplemental Security Income (SSI).
VII.
Evaluating the Effects of Work Impacts on Quality-of-Life Impacts
A. Testing the
link between work and quality-of-life impacts at the individual level
Hypothesis: Dramatically
increasing the employment and earnings of residents will substantially
improve their quality of life (whether or not they remain in public
housing.)
Although increasing
sustained employment is the fundamental goal of Jobs-Plus, part of the
rationale for this goal is an assumption that increased work is a route
to a better quality of life for poor people living in public housing.
This assumption cannot be taken for granted, however, particularly if
the work is largely in the low-wage sector of the labor market and does
not lead to substantial gains in overall family income, even with the
more advantageous financial incentives offered by Jobs-Plus. The Jobs-Plus
impact analysis will therefore test whether substantially increasing work
— if this is achieved by the program— does in fact substantially
improve residents’ well-being, as illustrated by the path from the
work impact box to the family well-being and quality-of-life box in Diagram
3.
A variety of indicators
of individual and family well-being will be analyzed. These will include,
for example, residents’ poverty and material hardships, their freedom
from drug or alcohol dependency, various school, behavioral, and health
outcomes for their children, and a variety of other quality-of-life outcomes.
If Jobs-Plus achieves
its intended effects on individuals’ employment and earnings, it
will be important to understand whether they move out of their developments
in response. Some newly employed residents may choose to stay despite
having their rent increased (as a result of higher earnings) because of
fear that they might lose their job or that the job might end and that,
if that happened, they might face a long wait for housing assistance if
they needed it again. Many residents may also enjoy living in their development
and have strong social ties with other residents there. Their desire to
stay may grow even stronger if they experience and want to maintain strong
supports for work not available to them elsewhere, if the quality of life
in the development improves, and also if more desirable and affordable
housing outside of the development is hard to find. Other residents, however,
may take advantage of any opportunities they have to move out of public
housing.
As part of this
investigation, the study will examine how residents’ housing choices
and options change as their earnings grow. For example, do they "climb
the housing ladder," from public housing to equal or better-quality
subsidized or unsubsidized rental housing or to home ownership? What circumstances
and considerations influence their choices?
B. Testing the link between
work and quality-of-life impacts at the development level
Hypothesis: Dramatically
increasing the proportion of residents who at any given time are working
will dramatically increase the quality of life within public housing
developments.
Jobs-Plus is not only
expected to improve the well-being of individual residents — who
may or may not continue living in their public housing development, it
is also expected to improve the quality of life within public housing
for those who remain there.
The hypothesis that
quality-of-life improvements within public housing can be achieved by
increasing residents’ employment is consistent with the broader and
widely shared view that boosting residents’ employment is fundamental
to inner-city revitalization. For example, Wilson (1996) postulates that
high rates of joblessness in inner cities produce severe social problems
that make those areas unpleasant and dangerous places in which to live.
He argues, further, that those problems cannot be solved without substantially
raising the proportion of residents in those communities who work.
This is a compelling
theory on its face. At the same time, there is no clear empirical evidence
that substantially increasing employment rates in high-poverty communities
(without simply "changing" the people who live there, as in
a gentrification process) would actually produce major quality-of-life
improvements, especially if much of the new employment is in poor-quality
jobs. Furthermore, if newly employed residents choose to leave a community,
community-wide improvement in quality of life may be impeded, or conditions
might even worsen. It may also be that quality-of-life improvements do
not come about until a "tipping point" is reached when a certain
critical mass of residents are working steadily at any point in time.
To determine the effects
of Jobs-Plus on the quality of life within public housing, and whether
positive impacts here (if achieved) result from big impacts on employment,
the impact study will again adopt a "development-level" perspective.
In that analysis the unit of analysis will be the population of residents
living in a development at a given point in time (which, over time, may
include a growing share of newcomers). A variety of outcomes will be measured,
such as whether residents view their development as a good place to live,
and whether they feel safe living there, whether they have recently been
victims of crimes, and whether they have social support networks. Much
of this information will come from survey data, but some will also come
from field research and possibly from housing authority and other agency
(e.g., police department) data. The analysis will determine whether, since
the start of Jobs-Plus, these aspects of life in public housing improve
more dramatically in the treatment developments than in the comparison
developments.
It is possible that
Jobs-Plus will produce big quality-of-life improvements within public
housing, independent of any effects the program has on work — and
may come about even if Jobs-Plus fails to generate employment impacts.
For example, these improvements might occur if Jobs-Plus, through its
efforts to mobilize residents and make them collaborative partners, leads
them to become more effective advocates for their communities, with greater
ability to get official agencies to respond to problems within their developments.
(See Diagram 3.) Moreover, the efforts to mobilize
residents around the goals of Jobs-Plus may lead to more of them becoming
directly active in neighborhood improvement projects sponsored by residents
themselves. Of course, it may also be that Jobs-Plus affects quality of
life within the developments both through its impacts on work and by increasing
resident activism.
To assess the effects
of the work impacts of Jobs-Plus on quality-of-life impacts at the development
level, the evaluation will determine whether improvements in both of those
domains tend to be found together across the sites, or whether they vary
independently across the sites. If the latter pattern prevails, or if
quality-of-life improvements fail to emerge despite big employment effects,
it would seriously challenge the hypothesis that achieving big employment
effects is a way to transform communities into better places in which
to live. The evaluation will also examine whether Jobs-Plus appears to
influence quality-of-life outcomes more directly, such as by affecting
resident activism within and on behalf of their developments.
In a related though
much less rigorous analysis, the evaluation will make some effort to gauge
whether Jobs-Plus has positive "spillover" effects in the surrounding
neighborhood. For example, it may try to assess whether residents of the
development are viewed more positively by outside neighbors as a result
of the program, and whether the implementation of Jobs-Plus seems to coincide
with positive trends in various quality-of-life indicators for the surrounding
neighborhood.
VIII.
Establishing Causality Between Jobs-Plus and Implementation and Impact
Outcomes
As this paper has
shown, the Jobs-Plus evaluation will include a number of causal analyses
as part of its implementation and impact studies, and its research design
is unusually well-positioned to address many of them. But, despite its
strengths, the evaluation still faces significant challenges. For example,
the random assignment design, while potent, is much more advantageous
for estimating the effects of Jobs-Plus on work and other outcomes when
all program sites are combined into a pooled sample. Site-specific impact
estimates will have less certainty, because only two or three developments
were randomly assigned within each site. Thus, by chance, a "better"
development (if there is one) may have been chosen within any given site
to be the Jobs-Plus development, or may have ended up as a comparison
site. Even for the pooled analysis, however, the small number of sites
makes this a less rigorous design than is typical in social experiments
in which large numbers of individuals are randomly assigned.
If Jobs-Plus produces
very large impacts, that result may compensate for some of these design
weaknesses. In other words, the bigger the estimated impacts, the more
plausible it becomes to attribute causality to the program rather than
to other possible factors. Big impacts will be even more important in
attributing causality to Jobs-Plus within any given demonstration site.
The fact that the policy objective in launching the Jobs-Plus demonstration
was to create an intervention that could produce much larger impacts than
have been detected in past welfare-to-work and job training programs makes
it easier to set a high impact threshold for inferring causality.
If impacts differ
across the Jobs-Plus sites, it will be particularly difficult to determine
whether that variation flows from differences in program strategies or
differences in other factors, such as the types of people served, local
economic conditions, or other aspects of the local context. This is a
common problem in multi-site evaluations. Here, again, the larger the
cross-site differences in impacts — and the more pronounced the differences
in site practices or treatments — the more plausible it may be to
infer that the distinctive practices of particular sites caused their
superior performance. (Also, statistical adjustments to control for measured
differences in resident characteristics across the sites can strengthen
this analysis.)
The evaluation will
also need to consider other ways in which it can support causal inferences.
One way is to surface and explicitly test compelling alternative hypotheses.
For example, welfare reform may be one of the most important factors contributing
to changes in employment outcomes over time for the Jobs-Plus sample.
To the extent that the evaluation can muster evidence against that factor
as a likely causal agent, the stronger the case in favor of Jobs-Plus
becomes.
Drawing causal inferences
from the survey of residents, which will be administered in both the treatment
and comparison developments prior to the start of Jobs-Plus and then again
at one or two points during the subsequent five years, will be more difficult
than drawing them from the administrative records data. This is because
the survey does not include information on long-term trends prior to the
start of Jobs-Plus, as is required by an interrupted time-series analysis.
However, when the sample of sites is pooled, the survey analysis will,
in essence, become an analysis of a social experiment, drawing power from
the evaluation’s random assignment of housing developments.
In addition, the survey
analysis can be "anchored" in the administrative records analysis.
For example, the survey will include measures of before-after changes
in employment. If those data show bigger changes for the Jobs-Plus developments
than the comparison developments, and if this pattern is consistent with
the findings from the longer-term administrative records data, it would
be reasonable to conclude that the survey trends are being influenced
by the same underlying factor — e.g. , the Jobs-Plus intervention.
It may then become plausible to attribute big changes in other survey
outcome measures to Jobs-Plus as well.
Any evidence that
the evaluation can provide in support of anticipated causal chains between
Jobs-Plus and the ultimate employment and quality-of-life impacts it hopes
to produce would also help in attributing causality to Jobs-Plus. For
example, if the analysis finds not only that the treatment sites reveal
a much sharper deviation in employment trends than the comparison developments
after Jobs-Plus begins, but also that residents in the program group experienced
or used services, incentives, and supports to a much greater degree than
those in the comparison group, and that their preparedness for work and
job-seeking efforts also improved more substantially, it stands to reason
(on theoretical grounds) that their employment outcomes should differ
as well. In other words, if Jobs-Plus appears to work in the ways that
prior theory says it should, that would be important corroborating evidence
of the program’s causal influence.
Field research may
also have a useful supporting role to play, especially for the development-level
analysis, and particularly in estimating the effects of Jobs-Plus on quality-of-life
improvements within public housing. For example, the field researchers
and their key informants who live or work at the sites may be able to
point to major changes within the Jobs-Plus developments that have occurred
only after Jobs-Plus began and may be able to provide a plausible explanation
linking those changes to explicit actions undertaken as part of Jobs-Plus.
They may be able to establish that similarly profound changes have not
occurred in the comparison developments.
Through all of these
methods together, the Jobs-Plus study may be able to address causal hypotheses
with an unusual degree of rigor for a community initiative evaluation.
IX.
Conclusion
As noted in the beginning
of this paper, the research findings from Jobs-Plus are expected to have
relevance beyond the world of public housing. Many of the hypotheses discussed
in the foregoing pages are directly pertinent to welfare reform and community
revitalization initiatives. Consequently, the demonstration’s findings
may yield important lessons and insights on which policymakers and administrators
can draw in attempting to build a work-based welfare system and improve
distressed inner-city neighborhoods.
References
Bloom, Howard. 1996. "Building
a Convincing Test of a Public Housing Employment Program for the Jobs-Plus
Demonstration." Paper prepared for the Manpower Demonstration Research
Corporation and presented at the Annual Research Conference of the American
Association for Public Policy and Management (APPAM), Pittsburgh, PA,
October 31–November 2, 1996.
Hollister, Robinson G., and Jennifer
Hill. 1995. "Problems in the Evaluation of Community-Wide Initiatives."
In James P. Connell et al. (eds.), New Approaches to Evaluating Community
Initiatives: Concepts, Methods, and Contexts. Washington, D.C., The
Aspen Institute.
Kingsely, G. Thomas, Joseph B.
McNeely, and James O. Gibson. 1997. Community Building Coming of Age.
Washington, D.C.: The Development Training Institute, Inc., and The Urban
Institute.
Wilson, William Julius. 1996. When
Work Disappears: The World of the New Urban Poor. New York: Alfred
A. Knopf, Inc.
|
