|
Abstract
This paper examines issues and options for the design
of a major non-experimental study to measure the impacts of a large-scale,
saturation-level demonstration program to promote employment among residents
of selected public housing developments. The program, Jobs-Plus,
is being launched by the Manpower Demonstration Research Corporation in
cooperation with the U.S. Department of Housing and Urban Development
and The Rockefeller Foundation. [An updated, full list of the Jobs-Plus
funding partners is provided at the front of this paper.] Because
Jobs-Plus will be a comprehensive community initiative, available
to all residents of the several public housing developments where it is
implemented, the program cannot be evaluated using a randomized experiment,
the now-standard method for measuring the impacts of employment and training
programs. However, because community-wide initiatives are becoming an
increasingly important component of social policy, it is essential to
develop methods for determining their success. It is the purpose of this
paper, therefore, to explore possibilities for doing so.
I.
Introduction
Jobs-Plus
Anticipated sea-changes in welfare policy and housing
policy are likely to place an enormous financial burden on U.S. public
housing. In particular, welfare reform initiatives that limit the duration
of benefits received by families will reduce the ability of public housing
residents, almost half of whom are welfare recipients, to pay their rents.1
In addition, proposed measures to consolidate federal housing subsidies
probably will reduce the overall level of such funding. In response to
these financial threats to the survival of public housing, it is imperative
that local Public Housing Authorities (PHAs) find effective ways to increase
the share of public housing residents who are employed.2
The benefits of resident employment go beyond financial
and economic gains. Holding a job provides social opportunities, linkages
to a larger community beyond public housing, positive role models for
children in the development, and personal satisfaction. Creating a mixed-income
community may reinvigorate the development, change public perceptions
about public housing, and revitalize the surrounding neighborhood.
The Jobs-Plus Initiative being launched by the
Manpower Demonstration Research Corporation (MDRC), in partnership with
the U.S. Department of Housing and Urban Development (HUD) and The Rockefeller
Foundation, is intended to help policy-makers and program managers learn
some important lessons about what is required to increase the employment
of public housing residents. To do so, MDRC will work closely with six
PHAs that will develop locally-based approaches to providing saturation-level
employment opportunities for working-age residents from at least one family
housing development in each PHA.
Although the design of Jobs-Plus will vary across sites,
reflecting their local resources, preferences and constraints, each site
will rely on a combination of three strategies:
- implementing state-of-the-art employment and training
services that have proved effective for welfare recipients and public
housing residents;
- developing financial incentives that promote work
through state welfare reform efforts and PHA modifications to rent rules
and eligibility requirements;
- building a public housing community that actively
supports work through resident groups and other local organizations.
Because of the enormous challenges facing public housing
at his time, Jobs-Plus cannot afford merely to modify "business as
usual." Instead it must provide a bold departure from previous attempts
to increase the economic self-sufficiency of public housing residents.
The "theory of change"3
which underlies Jobs-Plus, and which gives rise to the expectation
that it can, indeed, produce the major changes needed, derives
from the premise that the synergy created by combining the initiatives
three main strategies will be substantial.
Previous studies have shown that state-of-the-art employment
and training services can markedly increase the earnings of welfare recipients,
in general, and welfare recipients who are public housing residents, in
particular.4
Especially promising are services with direct and immediate links to jobs.
Previous research also has shown that public assistance
programs and housing subsidy programs contain powerful financial incentives
that inhibit work effort.5
If a poor family earns "too much" it can lose its Medicaid health
benefits. In addition, its housing subsidy might be reduced, its AFDC
benefits might be reduced, and its food stamp benefits might be reduced.
If local welfare agencies and local PHAs can combine their efforts and
find new ways to mitigate these disincentives, at least temporally, this
could help motivate families on welfare and in public housing to work
harder toward economic self-sufficiency.6
Previous research also has shown that in some cases, public
housing residents can build effective organizations and take more control
over their lives.7
Although such organizations are difficult to create, and require substantial
time and resources to sustain, they are a potentially important vehicle
for building the "social capital" necessary to foster a work-oriented
environment in public housing.8
There is no evidence, however, about the impacts of
combining these strategies in a concentrated way in a specific location.
Nevertheless, given the considerable demonstrated potential of each separate
strategy, and the likelihood that each would reinforce the others if they
were implemented together, the impacts of a joint strategy could be far
greater than the sum of the impacts of the separate strategies.
Research Questions
Research on Jobs-Plus should address the following questions:
- To what extent was it possible for the study sites
to implement the three Jobs-Plus strategies: high-quality employment
and training services, financial incentives that reward work, and a
community culture that promotes a positive work ethic? To what extent
were these strategies implemented together, and to what extent were
they mutually reinforcing?
- What institutions, resident groups, local civic groups,
and individuals played key roles in the development of Jobs-Plus
at each site? In what ways and how well did these groups and individuals
work together? How did they influence the design and the evolution of
Jobs-Plus? In particular, what role was played by public housing residents?
- How long did it take for each study site to develop
a Jobs-Plus initiative? What resources were necessary to make this to
happen? How much did these resources cost? How, and from where, were
these resources obtained?
- To what extent did public housing residents participate
in the Jobs-Plus initiative? What leadership roles did they play
in designing, implementing, and sustaining the initiative? What services
did they receive, and in what activities did they participate? What
were the main barriers to their participation? How, and to what extent,
were these barriers overcome? What were the main forces that helped
stimulate participation in the initiative?
- In what ways, and to what extent, did Jobs-Plus affect:
(1) public housing residents in the study sites; (2) public housing
developments where the initiative was implemented; and (3) communities
within which these developments were located? How long did it take to
produce these impacts, and to what extent were they sustained over time?
- Was Jobs-Plus cost-effective: (1) from the perspective
of public housing residents who were involved; (2) from the perspective
of local PHAs that were involved; (3) from the perspective of taxpayers;
and (4) from the perspective of society overall?
- How did the evolution of Jobs-Plus differ across sites?
How did its structure differ across sites? How did its results differ
across sites? To what can we attribute this variation? What can be learned
from this variation about the conditions necessary for an initiative
like Jobs-Plus to be successful elsewhere?
Measuring Jobs-Plus
Impacts9
To assess Jobs-Plus properly requires measuring its impacts.
By definition, a Jobs-Plus impact is:
the difference between what happened when Jobs-Plus
was implemented (its outcome) and what would have happened without
Jobs-Plus (its counterfactual).
Consider the following situation. What if one year after
Jobs-Plus was implemented in a public housing development, 25 percent
of its households had an adult who was employed? Some of these adults
probably would have been employed without Jobs-Plus. Thus, to estimate
what Jobs-Plus actually caused to happen its impact
one must estimate what the employment rate would have been without the
initiative. If, for example, this rate would have been 10 percent without
Jobs-Plus, then the impact of the initiative would be 15 percentage
points. If, on the other hand, the employment rate would have been 25
percent without Jobs-Plus, then its impact would be zero.
In the field of employment and training research, randomized
experiments are now widely regarded as the best way to estimate program
impacts.10
In the words of one prominent researcher, randomized experiments are "a
bit like the nectar of the gods: once youve had a taste of the pure
stuff it is hard to settle for the flawed alternatives" (Hollister
and Hill, 1995, p. 134).
Randomized experiments are like a lottery. They randomly
assign eligible program applicants to either a program group, which is
allowed to participate, or a control group which is excluded from the
program, at least temporarily.11
This process ensures that the two groups are similar in all ways, except
for their access to the program.12
Hence, the subsequent experience of the control group provides a valid
estimate of the counterfactual for the program group. Differences between
the experiences of the two groups therefore provide valid program impact
estimates.
During the past two decades, this approach has been used
widely to estimate the impacts of programs for economically disadvantaged
adults (for example, see Bloom et al., 1997; and Manpower Demonstration
Research Corporation, Board of Directors, 1980), programs for displaced
workers (for example, see Bloom, 1990, Corson et al., 1991, and
Speigelman et al., 1992), programs for welfare recipients (for
an extensive review of these studies, see Gueron and Pauly, 1991), programs
for "at risk" youths in school (for example, see Walker and
Vilella-Velez, 1992), and programs for young school dropouts (for example,
see Cave, Bos, Doolittle, and Toussaint, 1993).
Likewise, housing policy debates have relied extensively
on findings from the two major randomized housing experiments conducted
to date, the Experimental Housing Allowance Program (EHAP) and the Freestanding
Housing Voucher Demonstration Program.13
In addition, HUD recently commissioned a large-scale randomized experiment,
the Moving to Opportunity Demonstration, to study the effect of helping
poor families move from inner-city ghettoes to moderate-income neighborhoods.14
Unfortunately (from a methodological perspective), Jobs-Plus
is not a program to which individuals or households can be assigned randomly.
Instead, it is a "comprehensive community initiative" to which
all residents of the public housing developments involved will be "exposed,"
and in which all residents who desire can participate. Hence, it will
not be possible to measure Jobs-Plus impacts by randomly assigning
individuals.
It might be possible, however, to randomly select a program
group and a control group of public housing developments at each
site from among those willing and able to participate. This approach cannot
provide the methodological protection available from standard large-scale
randomized experiments, however, because of the small number of housing
developments involved.15
Nevertheless, because Jobs-Plus impacts are expected to be large, the
approach might provide enough statistical power to identify them. Furthermore,
because these impacts are expected to be large, quasi-experimental methods
might be able to provide plausible causal inferences about them (discussed
later).16
Jobs-Plus is intended for a broad range of stakeholders
who have very different priorities for the types of impacts that might
be produced. To provide a coherent structure for reporting these impacts,
we recommend examining them from three perspectives:
- impacts on public housing residents,
- impacts on public housing developments, and
- impacts on the surrounding community.
The first perspective represents the experience of persons
who live in a public housing development where Jobs-Plus is implemented,
regardless of whether or not they stay there. From this perspective, researchers
will study the same people over time. The second perspective represents
the experience of public housing developments where Jobs-Plus is implemented,
recognizing that residents will move into and out of the developments.
From this perspective, researchers will study the same developments
over time. The third perspective represents the experience of a larger
residential and business community, the composition of which will change
over time. From this perspective, researchers will study the same communities
over time.
Replicability
of the Research Findings
As mentioned earlier, because Jobs-Plus will represent
a major departure from previous attempts to increase the economic self-sufficiency
of public housing residents, very little is known about this approach.
Hence, to give Jobs-Plus a fair test at this early stage in its
development will require selecting sites that are committed to making
the concept work and judged likely to be able to do so. Thus, the goal
of the initiative probably will be to measure what Jobs-Plus can achieve
under conditions that are more favorable than average. Nevertheless, it
would be unwise to choose sites for Jobs-Plus that are so unusual that
their findings are not replicable. Therefore, a careful balance will be
maintained between the expected commitment and capability of the sites
selected for the demonstration and the likelihood that other sites, but
not necessarily all sites, could replicate their experience in the future.
Developing
New Methods to Assess Comprehensive Community Initiatives
As mentioned earlier, Jobs-Plus is envisioned to be a comprehensive community
initiative, not a single program operating in isolation. Comprehensive community
initiatives have several features which make them especially difficult to
evaluate.17
- A horizontally complex intervention. Jobs-Plus
will integrate the efforts of numerous social service systems (especially
welfare and housing agencies) plus the activities of many civic, business,
and community groups. Therefore, it will have many facets, whose separate
impacts will be difficult to distinguish.
- A flexible and evolving intervention. Jobs-Plus
sites will have wide latitude to mold the initiative. Furthermore, each
sites vision for the project will change over time. Hence,
Jobs-Plus projects probably will differ from their initial plans,
and probably will take on different forms at different times. This makes
it especially important, and difficult, to document precisely what the
intervention was at each site, and how it changed over time.
- A broad range of outcomes to examine. Jobs-Plus
is intended to produce changes in many different dimensions (economic,
psychological, social, political, and physical) and changes that will
occur over widely varying time-frames (immediate, intermediate, and
long-run). Hence, the number of outcome measures to consider is substantial,
and the task of summarizing them will be difficult.
- Vertically complex impacts. Jobs-Plus is intended
to produce impacts that are experienced at different levels and that
therefore should be examined from different perspectives. As indicated
above, Jobs-Plus is expected to produce impacts on public housing
residents, on public housing developments, and on the communities surrounding
these developments.
- Important contextual issues. Although Jobs-Plus
will be implemented in local communities, its evolution, its final
form, and its results will be influenced by many forces which are outside
of its control. For example, federal regulations (and possible waivers)
which govern welfare and housing programs will determine the range of
available programmatic options. Economic conditions in the community,
in the region, and nationally will influence the job opportunities through
which economic self-sufficiency can be achieved. Barriers due to racial
and ethnic discrimination will limit the success and influence the direction
of the initiative. Therefore, to understand why certain impacts did
or did not occur requires a detailed knowledge of the environment within
which Jobs-Plus operated.
- Absence of an adequate experimental control group.
For reasons outlined above, it will not be possible to randomly assign
individuals to Jobs-Plus or a control group. Thus, it will not be possible
to estimate Jobs-Plus impacts using the approach now relied on by evaluations
of most employment and training programs. Nevertheless, it will be crucial
to determine what, in fact, Jobs-Plus caused to happen.
The preceding problems, inherent in evaluations of comprehensive
community initiatives, have been the focus of current work by the Roundtable
on Comprehensive Community Initiatives for Children and Families, sponsored
by the Aspen Institute. This group has: (1) identified and clarified the
many serious problems experienced by such evaluations (Hollister and Hill,
1995); (2) made a forceful case for the need to find ways to overcome
these problems (Kubisch, Weiss, Schorr, and Connell, 1995, and Brown,
1995), and (3) begun to sketch an alternative evaluation framework, based
on "theories of change" (Weiss, 1995, and Connell and Kubisch,
1995).
This work is in its very early stages, however, and no
new evaluation methods or models have been developed or tested. Therefore,
research designs for Jobs-Plus will have to rely on combining existing
methodologies in creative ways and developing new approaches.
This Paper
This present paper focuses on estimating the impacts
of Jobs-Plus from the three perspectives described above. Section 2 discusses
impacts from the perspective of the residents of Jobs-Plus developments.
Section 3 discusses impacts from the perspective of the developments themselves.
Lastly, Section 4 discusses impacts from the perspective of the community
surrounding a Jobs-Plus development. All plans for the Jobs-Plus evaluation
are tentative at this time. The final design will evolve during the pilot
phase of the project.
II.
Jobs-Plus Impacts on Public Housing Residents
Introduction
This section explores ways to measure the impacts of
Jobs-Plus on residents of public housing developments. To measure these
impacts requires observing the same people over time, regardless
of whether they stay in the development where Jobs-Plus was implemented
or move away.
The section begins with a simple model of the process
by which Jobs-Plus is expected to produce its intended effects.
This model provides a conceptual framework for defining program outcome
measures. Three research designs for measuring Jobs-Plus impacts are then
considered, followed by a discussion of sample size and minimum detectable
effects. The section concludes with a discussion of data issues.
A Model of Jobs-Plus
Impacts and Measures of Jobs-Plus Outcomes
Exhibit 2.1 illustrates a four-stage
process which describes how Jobs-Plus is expected to improve the lives
of public housing residents. The process starts with the Jobs-Plus program
and baseline characteristics of the residents, their public housing development,
and its surrounding community. The three main components of Jobs-Plus
program services, financial incentives, and community support
will provide the impetus for improvements in the lives of public housing
residents. The baseline characteristics of these residents, their development,
and its surrounding community will influence their ability to benefit
from Jobs-Plus.
For example, personal characteristics (such as residents
education levels, labor market experiences, dependence on public assistance,
and attitudes toward life, work, and welfare) will play a large role in
determining their value to potential employers. Likewise, characteristics
of public housing developments (such as their perceived safety, the presence
of working adults, the availability of child-care services, the existence
of supportive organizations, and their proximity to job opportunities)
can affect residents willingness and ability to seek employment.
Similarly, characteristics of the community surrounding a public housing
development (such as its perceived safety, the existence of nearby jobs,
access to public transportation, and the presence of retail facilities)
can facilitate or impede the labor market success of public housing residents.
This constellation of baseline characteristics defines the setting in
which Jobs-Plus will take place and will be documented carefully.
The second stage in the process involves some initial
Jobs-Plus outcomes that must occur in order for the initiative to influence
the decisions and behavior of public housing residents. First, residents
must understand how Jobs-Plus changes their incentives to work, through
modifications of welfare eligibility requirements and benefit determination,
and through modifications of public housing eligibility requirements and
rent calculations. If residents understand these provisions, they can
respond to them accordingly; if they dont understand them, they
cannot respond systematically. In addition, public housing residents will
need a wide variety of services to help them overcome their many serious
barriers to employment. If they receive these services, they should be
better able to compete effectively for jobs. Without these services, they
will not be as competitive. Finally, Jobs-Plus will help tenant organizations
and other neighborhood groups to create a personally-supportive, work-focused
environment. If such a climate is created, and if public housing residents
perceive it to exist, then it could provide an important boost to their
self-improvement efforts.
These initial outcomes are anticipated prerequisites for
the success of Jobs-Plus and will be examined as part of the implementation
research for the project. Hence, they will provide an early indication
of whether or not Jobs-Plus is likely to be successful.
The third stage in the process involves changes produced
by Jobs-Plus in the attitudes, the human capital (knowledge, skills, and
credentials), and the behavior of public housing residents, plus changes
in the extent and nature of their interactions and their level of organization.
It is directly through these changes that Jobs-Plus is expected to produce
its intended effects on economic self-sufficiency. For example, to the
extent that exposure to Jobs-Plus increases the self-confidence of public
housing residents about finding and holding a job, they are likely to
try harder to do so. To the extent that Jobs-Plus increases the knowledge
and skills of public housing residents, and helps them obtain educational
and vocational credentials, it will increase their potential productivity,
and thereby increase their value to employers. To the extent that Jobs-Plus
increases the amount of time that public housing residents spend in productive
activities (such as study, job-search, volunteer work, or helping children)
and to the extent that Jobs-Plus decreases the incidence of negative behaviors
(such as alcohol abuse, drug abuse, and criminal activities), it will
help to foster life skills and behavior patterns that support, instead
of impede, gainful employment. Likewise, to the extent that public housing
residents interact more frequently with each other, and develop more cohesive
organizations, they are more likely to reinforce each others attempts
to become and remain employed.
Only if these changes occur, can Jobs-Plus improve the
lives of public housing residents. Hence, measuring these intermediate
impacts provides an interim test of whether Jobs-Plus is working as it
was intended.
The fourth and final stage in the process involves actual
measures of the progress made by public housing residents toward economic
self-sufficiency. There is disagreement, however, about what constitutes
economic self-sufficiency.18
At a minimum, it requires gainful employment. In addition, most people
would agree that it involves a lack of dependence on AFDC and food stamps.
Agreement is less uniform, however, about whether or not persons who receive
housing subsidies (which have higher income eligibility levels than AFDC)
should be considered self-sufficient. It is even less clear whether or
not persons who receive subsidized health care, through Medicaid, should
be considered self-sufficient. Lastly, there are many families that have
a full-time worker and receive no form of public assistance, but nevertheless
remain in poverty. Should these families be considered self-sufficient?
Given the complexity of the concept of self-sufficiency, and the widely
varying views of what the concept means, the impact of Jobs-Plus should
be measured in ways that reflect these different views.
Exhibit 2.2 lists outcome measures
that reflect the preceding model, and thereby could serve as the basis
for assessing the initial, intermediate, and long-term impacts of Jobs-Plus.
Impact Study Design #1:
A Two-Group, Before-After Analysis
The Approach
Previous research on the impacts of self-sufficiency programs for public
housing residents (Shlay and Holupka, 1992, and Rohe, 1993) have measured
program impacts by comparing changes in outcomes for a program group that
was exposed to the program and a comparison group that was not exposed.19
The basic logic of this approach is as follows:
- the program outcome is represented by the change
in an outcome measure from before the program was implemented (usually
at baseline) to after it was implemented,
- the counterfactual is represented by the corresponding
change for the comparison group,
- the impact estimate is the difference between
the observed change for the program group and the observed change for
the comparison group.
In theory, this approach could be used to estimate impacts
with regard to any of the outcome measures listed in Exhibit 2.2. To the
extent that the observed change in the measure for the comparison group
accurately reflects what the change for the program group would have been
without the program, the approach provides a valid estimate of the impact
for the sample.
The key to making this approach work is finding the right
comparison group. Shlay and Holupka (1992), in their study of the Family
Development Center at the Lafayette Courts public housing development
in Baltimore, Maryland, chose their principle comparison group from families
at a nearby public housing development, Murphy Homes.20
Rohe (1993), in his study of the Gateway Transitional Families Program,
in Charlotte, North Carolina, chose his comparison group from persons
who applied to the program but did not participate.
To increase the validity of findings from a two-group,
before-after design, it is sometimes possible to match comparison group
members to program group members in terms of measured baseline characteristics.
There are two basic versions of this approach:
- cell matching, whereby: (1) subgroups (cells) are defined
according to combinations of baseline characteristics (e.g.,
age category, gender, race, and welfare status), (2) each program group
member is placed in the cell to which he or she belongs, and (3) comparison
group members are chosen to match the number of program group members
in each cell;21
- distance function matching, whereby: (1) a weighted
function of baseline characteristics is computed for each program group
member and for each potential member of the comparison group; and (2)
a comparison group member is chosen to match each program group member
in a way that minimizes the difference (or "distance") between
their values of the weighted function.22
In addition to statistical matching procedures, researchers
usually estimate program impacts from a multiple regression equation,
of the following form, which attempts to control for differences between
program group and comparison group baseline characteristics which are
included in the model
where:
Y2 = the follow-up value of the outcome measure,
Y1 = the baseline value of the outcome measure,
T = one for program group members, and zero for comparison
group members,
Xj = a baseline characteristic,
a = an intercept
term,
B 0 =
the program impact,
B 1 and
B j = regression
coefficients, and
e = a random error
term.
Strengths and Weaknesses
The principal strengths of the approach are its relative
simplicity, and the fact that it could be implemented readily for Jobs-Plus.
The main weaknesses of the approach are several important "threats"
to the internal validity of its program impact estimates. These threats
represent specific reasons why the change in the comparison group outcome
might not accurately reflect what the change would have been for the program
group without Jobs-Plus.
First, is the possibility that the comparison group and
the program group will experience different events that affect
their willingness and ability to work.23
Hence, it is probably advisable to choose comparison group members who
are from the same local labor market and are served by the same welfare
agency and public housing authority.
Several versions of this problem are likely to occur.
For example, while the study is underway, other government programs probably
will change in ways that affect the Jobs-Plus target population. In particular,
welfare reform measures are likely to be taken by some, if not all, of
the jurisdictions where Jobs-Plus is implemented. If the comparison group
experiences local welfare reform and the program group experiences Jobs-Plus,
then impact estimates will represent the differential effect of Jobs-Plus
versus local welfare reform. If welfare reform in the Jobs-Plus PHAs is
sufficiently dramatic, its effects could over-shadow those of Jobs-Plus,
and thereby make it very difficult to measure Jobs-Plus impacts.
A second potential problem of this type is represented
by the fact that HUD will be conducting another major demonstration project
in many of the PHAs where Jobs-Plus might take place. This new project,
the Moving to Work Demonstration, will test increases in the flexibility
of federal regulations that control PHA operations. The increased flexibility
might enable PHAs to modify their eligibility standards for public housing
and their rent determination procedures in ways that also will stimulate
work effort. Furthermore, changes in federal regulations might enable
PHAs to better coordinate services and activities with other organizations.
In short, the Moving to Work Demonstration could provide a competing "treatment"
to Jobs-Plus. If it is not possible to withhold this treatment from enough
public housing developments to maintain an untreated comparison group,
then impact estimates obtained by comparing Jobs-Plus sites and comparison
sites will represent the "differential" impacts of the two programs.
A second common problem with two-group, before/after designs
is the fact that the program group and the comparison group are often
on two different long-term trends before they were chosen for the
study.24
Hence, even if the study had not been conducted, subsequent changes observed
for the two groups would be different. If, for example, tenants at one
public housing development were more likely to find jobs than were the
tenants at another development, the subsequent changes in their employment
rates would be different in the absence of Jobs-Plus. With only one baseline
outcome measure for each group, there is no way to identify their underlying
trends. Hence, there is no effective way to compensate for them.25
A third common problem with two-group, before/after designs
also arises from the fact that only one baseline outcome measure exists.
This problem results not from the existence of underlying trends, but
from a temporary random departure from this trend that occurs at baseline.
For example, previous studies of employment and training
programs have identified a "pre-program dip" whereby average
earnings in the year before participants enter a program are well below
their preceding trend.26
This phenomenon has been the subject of extensive debate, and there is
still no agreement about the extent to which it represents a temporary
aberration (due to illness, a business failure, or other "bad luck"),
or the extent to which it represents the onset of a permanent decline
in earnings (which happens to economically displaced workers).
Individuals are probably more likely to enroll in a job-training
program when they have just experienced an unusually bad period than when
they have just experienced an unusually good period. In other words, their
motivation to enroll is probably negatively correlated with random fluctuations
in their earnings prospects. If so, then their average earnings in the
next period are likely to increase regardless of whether or not they enroll
in a program. This phenomenon is a statistical artifact called regression
to the mean.27
With two groups, the problem becomes one of different
changes in earnings due to different regression artifacts. Hence, the
observed change for the comparison group will not accurately represent
what this change would have been for the treatment group, without the
treatment.
The second and third problems mentioned above represent
two different reasons why the program group and the comparison group differ
initially in ways that would cause them to experience different outcomes,
even in the absence of Jobs-Plus. Hence, they represent two different
forms of "selection bias."
Impact Study Design #2: Longitudinal Data
The Approach
The second approach to estimating the impacts of Jobs-Plus
is a logical extension of the first, which is intended to deal with: (1)
program groups and comparison groups that have different underlying outcome
trends, and (2) program groups and comparison groups which have different
regression artifacts. To address these issues requires data for both groups
on outcome measures for a number of years (four to five) before Jobs-Plus
begins, and as long as needed to identify impacts which materialize thereafter.
The longer it takes for impacts to materialize, the longer the followup
period that is required. This type of information is generally referred
to as longitudinal data or panel data.
From these data, one can compute a "pre-program"
trend for each program group member and each comparison group member.
It is then possible to compute a separate post-program deviation from
each persons pre-program trend.28
The difference between the mean deviation from trend for the Jobs-Plus
group and the mean deviation from trend for the comparison group provides
an estimate of the impact of Jobs-Plus. This estimate represents: the
extent to which Jobs-Plus caused participants to change their long-term
labor market behavior beyond the change that would have occurred without
Jobs-Plus.
Because past behavior, especially past behavior for a
number of years, is usually the best predictor of future behavior, comparing
sample members post-program outcomes to their pre-program trend
(thereby allowing each sample member to serve as his or her comparison
group) is probably the best way, absent a randomized experiment, to control
for underlying differences in their likely future behavior.29
Giving individuals their own trend implies estimating
a separate intercept and a separate slope for each. This is a straightforward
extension of standard "fixed-effect" models for longitudinal
data, which specify a separate intercept for each sample member.30
It is possible to include additional explanatory variables in such models,
but this generally does not increase their explanatory power appreciably
because the additional variables are usually reflected in the past outcome
trends for each sample member.31
Strengths and Weaknesses
The principal strength of this approach is its ability
to control for differences in the underlying trends of the program group
and the comparison group. In addition, because the approach provides an
explicit description of the behavior of sample members over time, it can
identify aberrations that might have occurred before an intervention took
place. If no such aberration exists, then regression artifacts should
not be a problem. If, on the other hand, an aberration appears in the
data for either the program group or the comparison group, or both, then
the potential for differential regression artifacts exists. Nevertheless,
impact estimates based on longitudinal data are less susceptible to the
influence of regression artifacts than are simple before/after comparisons.
One potential weakness of longitudinal impact estimators
is the possibility that the program group and the comparison group are
subject to different major local events (history). If this occurs, then
differences in the changes observed for the two groups might reflect differences
in the events to which they were exposed, not the impact of Jobs-Plus.
Only by choosing program and comparison groups that are likely to be exposed
to the same events, and carefully documenting the events to which they
were exposed, can one properly interpret the impact estimates obtained.
A second potential weakness of longitudinal impact estimators
is the possibility that past behavior will not adequately reflect potential
future for program groups members because they have reached a point in
their lives where they will make a major change.32
To see this point, consider the following situation.
Enrolling in a job-training program might signal a desire
to change ones situation in life. Even if someone has been out of
the labor force for a long time, or has been in and out of low-pay jobs
for many years, he or she might be ready, able, and willing to make a
change with or without participating in a program. If so, then data over
time on prior employment, earnings, welfare receipt, or housing subsidies
will not provide an adequate statistical control for likely future changes
in these outcomes. Indeed, nothing short of a randomized experiment will
do so.33
Fortunately, because of a fundamental difference between
the participant selection process for employment and training programs,
and that for Jobs-Plus, longitudinal impact methods (even two-group, before-after
methods) might be more suitable for evaluating Jobs-Plus than for evaluating
employment and training programs.
Individuals will become members of the Jobs-Plus program
group if they happen to live in a public housing development where Jobs-Plus
is implemented. They will not become members of the program group if they
do not live in a Jobs-Plus development. Hence, at least when the initiative
begins, Jobs-Plus "comes to the individuals," the individuals
do not "come to Jobs-Plus."34
Therefore, the program group should not exhibit a pre-program dip. Consequently,
it should not exhibit a regression artifact.
Furthermore, there is no reason to expect that the proportion
of program group members who are planning to change their lives differs
appreciably from the proportion of residents at other similar developments
who plan to do so.
Impact Study Design #3: Random Assignment of
Public Housing Developments35
A third approach for estimating the impacts of Jobs-Plus,
which builds on the preceding two approaches, is a randomized experiment,
in which: (1) several public housing developments are chosen as candidate
Jobs-Plus sites from each PHA that participates, (2) these developments
are matched as closely as possible, and (3) a lottery is used to determine
which development implements Jobs-Plus and which become members of a control
group.
If this approach is feasible, it could be used in conjunction
with either or both of the preceding approaches. By randomly deciding
which candidate developments become Jobs-Plus sites and which become
control sites, one can eliminate systematic forces that might produce
initial differences between the two groups. Only by chance would such
differences occur. Hence, this procedure would eliminates bias in program
impact estimates. Nevertheless, a substantial margin for random
sampling error would remain, because residents would be randomly assigned
by development, not independently as individuals.36
Note that this random sampling error will be present whether
public housing developments are chosen randomly for the program and control
groups or whether they are chosen in some other way (which would not eliminate
bias). The problem with respect to random sampling error arises because
Jobs-Plus applies to whole public housing developments, not to specific
residents.
Sample Size and Minimum Detectable Effects37
Sample size is key to the Jobs-Plus impact analysis.
To place this issue in perspective, first consider the magnitude of impacts
produced by successful welfare-to-work programs that have some of the
planned features of Jobs-Plus (Exhibit 2.3).
Column one in the exhibit reports some of the first available
findings from demonstration programs being conducted in Atlanta, Georgia;
Grand Rapids, Michigan; and Riverside, California, under the Job Opportunities
and Basic Skills (JOBS) Program. These findings reflect the impacts on
employment rates of a "Labor Force Attachment" approach to stimulating
employment, through job-search assistance, followed by work experience
or short-term education and training activities, if needed. The second
column presents unpublished findings from the same study for a subsample
of public housing residents in Atlanta. The third column reports recently
published findings from a major Canadian study of an earnings supplement
designed to "make work pay" by guaranteeing (for up to three
years) a substantial minimum earnings to welfare recipients who find a
full-time job within a year and remain employed full-time thereafter.
In each case, the employment rate for program group members
observed eighteen months to two years after enrollment was 8 to 9 percentage
points higher than it would have been without the program. These findings
were statistically significant.
Furthermore, these findings represent substantial impacts
on employment. To see this, it is useful to express each impact as a percentage
of the corresponding employment rate for the control group. Hence, the
8.1 percentage point impact in the first column represents a 23.5 percent
increase from the 34.4 percentage point employment rate for control group
members. In other words, 23.5 percent more of the program group members
were employed than would have occurred without the program. Corresponding
impacts in these terms are a 28.0 percent increase in employment for column
two and a 28.4 percent increase in employment for column three.
The impacts of Jobs-Plus should exceed those in Exhibit
2.3, because it will combine the separate approaches used by the programs
represented in the exhibit, and add a third important component to the
mix. As discussed earlier, Jobs-Plus will include state-of-the-art
employment and training services; financial and other incentives that
promote work; and vigorous efforts to build a community that supports
work. Hence, Jobs-Plus should increase employment rates by at least
10 percentage points. Therefore, the impact study design for Jobs-Plus
must have enough statistical power to detect impacts of roughly this size.
Findings in the exhibit were based on total samples that
ranged from just under 800 persons to just over 1,900 persons, which is
typical of many recent experiments, but is considerably smaller than the
largest ones.38
These samples could detect impacts on employment rates of 8 to 9 percentage
points. Thus, if the Jobs-Plus sample were selected in a similar way (by
randomly assigning individuals to program or control status), 1,000 to
2,000 public housing residents would be adequate to detect the types of
impacts anticipated if Jobs-Plus is successful.
However, Jobs-Plus sample members will enter the study
in groups, or "clusters," either as residents in public housing
developments selected for the Jobs-Plus program, or as residents in public
housing developments selected for the comparison group. This "cluster
assignment" will increase the standard errors of Jobs-Plus impact
estimates, thereby reducing their statistical power. This loss of power
could be substantial, because the clusters (developments) will be large
(containing 200 to 400 families each).39
Indeed, the standard errors for impact estimates from a Jobs-Plus sample
might be several times as large as those from a sample of the same size
produced by random assignment of individual public housing residents.40
Without information about the "intra-class correlation"
for the outcome measures to be used, it is not possible to determine whether
any given number of public housing residents will be adequate for the
study. Nevertheless, to provide some insight about the likely statistical
power of Jobs-Plus impact estimates, Exhibit 2.4
presents a hypothetical example.
The exhibit lists employment rates for five hypothetical
housing developments in two consecutive years, without Jobs-Plus. The
size of these developments is not specified, but I assume that they are
relatively large, and focus on aggregate employment rates.41
The magnitudes of these employment rates are similar to those observed
by the studies represented in Exhibit 2.3.
What is more important, however, is the variation in
the changes in employment rates. These changes range from an increase
of 10 percentage points to a decrease of 10 percentage points (which is
substantial). Their standard deviation is 8 percentage points.
If this situation approximates the actual variability
of changes in employment rates for public housing developments like those
in the Jobs-Plus sample, then one can approximate the minimum detectable
effects of Jobs-Plus using the public housing development as the unit
of analysis.
For example, what if one Jobs-Plus development and one
comparison development are chosen from each of five PHAs? Estimates of
impacts on changes in employment rates obtained from this sample of 10
developments would have a standard error of 5 percentage points.42
This means that an impact estimate greater than roughly 10 percentage
points would be statistically significant at the .05 level.43
If instead, we chose one Jobs-Plus development for each
PHA and two developments for the comparison group, to increase statistical
power, the standard error of the impact estimate would be 4.4 percentage
points. Thus, an impact estimate greater than about 9 percentage points
would be statistically significant at the .05 level.
Given the preceding assumptions, it therefore should
be possible to detect impacts on employment rates that are comparable
to those produced by state-of-the-art employment and training programs
and targeted financial incentives.
I next briefly discuss some important data issues and
options.
- Whom should we observe?
- What types of outcomes should we measure?
- For what period should we measure these outcomes?
- Where can we get information about these outcomes?
To measure the impacts of Jobs-Plus on public housing
residents will require defining the target population as all persons living
in a Jobs-Plus development when the initiative begins. It also probably
will require drawing a sample at the same time from a comparison development
for each site. During the followup period, it will be necessary to track
these individuals wherever they go.
Earlier I presented a model of Jobs-Plus impacts which
implies a series of initial, intermediate, and long-term outcomes.
Exhibit 2.2 lists these measures, which are not repeated here. Instead,
I consider aspects of these measures which determine the types of data
collection methods they will require. In addition, I briefly indicate
how different outcome measures will provide different types of information
about changes over time and, consequently, will require different statistical
analysis methods.
First consider measures of differences in levels or
differences in states at two points in time, say, for example, at
baseline and at the end of followup. This focus applies to outcomes such
as residents attitudes about themselves, about welfare, and about
work, their understanding of the incentives of Jobs-Plus, and whether
or not they are employed, on welfare, or still living in public housing.
The emphasis here is on describing how the outcome differed
at two points in time, not on identifying when it might have changed,
or on describing fully its pattern of change over time (if it changed
more than once). Hence, one can measure such outcomes at two specific
times and summarize them as mean changes a simple before-after
analysis.
True longitudinal data are different. They provide continuous
information over time about a level, a flow, or a state, and thereby make
it possible to summarize a pattern over time, which describes both
how an outcome changed and when it changed.
To analyze longitudinal data for continuous outcomes,
such as earnings or welfare benefits, one can use conventional fixed-effect
models for panel data, or extensions of these models, such as time-varying,
fixed-effect models (Bloom, 1984).44
To analyze such data for discrete outcomes, such as employed
or not, on welfare or not, or living in public housing or not, requires
statistical methods for "event history" analysis, such as "hazard
rate" models (Allison, 1988).
Two questions arise here: (1) how long should the followup
period be? and (2) how long should the baseline period be? Different answers
to these questions will be necessary for different outcome measures obtained
from different data sources with different time limitations.
The overall followup period for the project probably should
be at least five years, because of the time that it will take to implement
Jobs-Plus fully, and because of the likely long time required for
the initiative to increase the economic self-sufficiency of public housing
residents. For example, Rohe (1993) describes a seven-year process for
participants in the Gateway Transitional Families Program sponsored by
the Charlotte, North Carolina Housing Authority.45
We also recommend that the baseline period for some of
the studys key outcome measures include four to five years before
sample members are exposed to Jobs-Plus. Shorter baseline periods
would enable researchers to control for differences in the initial conditions
of sample members, but longer periods are required to control for differences
in their underlying trends.
Where to Get the Data?
There are four main potential sources of outcome data for public housing
residents.
- sample surveys,
- PHA administrative records,
- UI wage records, and
- AFDC and food stamps records.
Sample surveys could be conducted at baseline and at one
or more times thereafter. The first survey could measure the baseline
characteristics listed earlier. It also could obtain retrospective longitudinal
information about employment, earnings, welfare receipt, subsidized housing,
and other key outcomes during the previous two-to-five years. The time
period for this information would be limited by the ability of survey
respondents to recall past events.46
PHA administrative records, especially those based on
the HUD Form 50058, which is used to certify a familys eligibility
for public housing when it applies, and is used each year thereafter to
recertify eligibility and set family rents, provide information about
selected socioeconomic characteristics. If these forms are completed properly,
and if they are accessible, they could provide an annual "snapshot"
of a households composition, its income by source, and its net income
(minus allowances and expenses). Examination of the quality and availability
of these data for sites that are selected to participate in Jobs-Plus
will be necessary to determine whether or not they should be used.
As a source of longitudinal baseline data, the 50058 Form
is limited by the period for which it is available, and the fact that
it covers different periods for different residents.47
Nevertheless, it might provide a good description of each household in
the study at the time Jobs-Plus begins. As a source of longitudinal followup
data, the 50058 Form is limited by the fact that it is not collected after
a family leaves a PHAs housing programs.48
UI wage records contain data on earnings that are reported
quarterly to each state by employers, as required by law, for all workers
covered by Unemployment Insurance.49
This information is reported for well over 90 percent of all employees
in most states. It is used to determine eligibility for Unemployment Insurance
and weekly benefit rates for persons who file a UI claim.
By matching sample members Social Security numbers
to their UI wage records, one can measure their total quarterly earnings
and their quarterly employment rates. This approach has been used by many
studies of employment and training programs. As a source of longitudinal
followup data, UI wage records are excellent, because they can be obtained
for an indefinite period. Furthermore, they have been found to produce
program impact estimates that are similar to those based on more costly
followup survey data (Bloom et al., 1993).50
As a source of pre-program longitudinal data, this information might be
limited to four to six quarters, which is all that is usually kept in
active records by state UI agencies because they only maintain active
data for the period used to establish UI eligibility and weekly benefit
rates. It might be possible, however, to increase the coverage of this
information substantially, if state agencies agree to provide information
from their archives.
AFDC and food stamps records can be obtained by matching
the Social Security numbers of sample members to the administrative records
of local and state welfare agencies. This information also has been used
by past evaluations of employment and training programs. When available,
it represents an effective way to produce an accurate long-term, monthly
history.51
III.
Jobs-Plus Impacts on Public Housing Developments
Introduction
This section explores ways to measure the impacts of
Jobs-Plus on public housing developments. The goal here is to study
what happens to a particular residential environment because of the initiative.
I first present a model of how Jobs-Plus might affect a public housing
development and corresponding measures of its success. I then consider
research designs for measuring the impacts of Jobs-Plus.
Exhibit 3.1 presents a model
of how Jobs-Plus might affect a public housing development. Just as the
model for individual residents was an oversimplification, so is the model
for public housing developments. Nevertheless, it provides a conceptual
basis for specifying program outcome measures and a logical structure
for organizing research findings.
The structure of the model is as follows. Jobs-Plus is
designed to directly influence the characteristics of public housing residents
and the prevailing culture of their development. Changes in resident characteristics,
in turn, produce further changes in local culture, and vice versa.
These changes increase the success of the development as a place to live
and as a financial enterprise.
Changes in the resident characteristics of a public housing
development can occur both because individuals change over time and because
different households move out of and into the development. These changes
are related to residents attitudes, human capital, labor market
success, welfare dependence, mobility, and level of organization and interaction.
As residents begin to change, improvements in the local
culture become possible. If Jobs-Plus is successful, we will begin to
see the evolution of a safe, personally-supportive and work-oriented environment.
Factors to consider in this regard, include the absence of crime, drug,
and gang activities, and the presence of organizations that promote social
capital, such as resident groups and civic organizations.52
The final stage of the model represents the overall success
of the public housing development, as a place to live, and as a financial
enterprise. To the extent that the lives of individual residents are improved,
and the prevailing culture becomes more positive, one should expect to
see improved physical conditions, more favorable perceptions of the safety
and desirability of the development, and improved financial conditions.
Exhibit 3.2 lists some measures
of a public housing developments overall success. Measures of changes
in resident characteristics and changes in local culture were presented
in Exhibit 2.2.
To estimate impacts from this design requires comparing
the mean change in an outcome variable for Jobs-Plus developments
to the mean change for comparison or control developments.53
For example, to estimate impacts on vacancy rates, one would compute the
mean change in vacancy rates for a Jobs-Plus sample, repeat this procedure
for a comparison sample, and take the difference. A standard t test can
assess the statistical significance of this difference.
The primary strengths of this approach are its simplicity
and feasibility. Its main weaknesses are:
- the Jobs-Plus developments and the comparison developments
might be exposed to different events which affect their outcomes differently
(they might experience different histories),
- the margin for random sampling error will be high (the
statistical power of program impact estimates will be low) because of
the small number of developments in the sample,
- Jobs-Plus developments and comparison developments
might have a different average underlying trend (they might be maturing
differently),
- random factors might affect the baseline conditions
of the two groups differently (they might exhibit different regression
artifacts).
To address the first problem requires selecting Jobs-Plus
developments and comparison developments that are as likely as possible
to experience similar events in the near future. As discussed in Section
2, they probably should be selected from the same PHAs, and one should
make an attempt to match their past histories, resident characteristics,
and political situations. Furthermore, it will be crucial to document
events which transpire in each development to determine whether different
events affected their future outcomes in important ways.
Aside from choosing more project sites or choosing more
developments from each participating site, there is little that can be
done to increase the statistical power of impact estimates for Jobs-Plus.
Nevertheless, if these impacts are as large as those documented by past
rigorous studies of key Jobs-Plus components, impact estimates for Jobs-Plus
could be statistically significant.54
The best way to address the last two problems listed above
is to use longitudinal data. Designs of this type, when applied to aggregate
units, such as public housing developments, are referred to in the evaluation
research literature as "interrupted time-series" analyses.55
The Approach
The simplest interrupted time-series analysis involves
a single site with multiple years of data before an intervention and multiple
years of data during its followup period. This analysis can be applied
to outcome measures for public housing, such as vacancy rates, the percentage
of households on welfare, total rents paid, the percentage of households
with a working adult, and so on. Exhibit 3.3
illustrates how to use the analysis to estimate the impact of Jobs-Plus
on the percentage of households in a development who are on welfare.
If four or five years of baseline data on this measure
can be obtained, then one can fit a pre-Jobs-Plus trend line.56
A linear trend probably will be adequate for most outcomes, but a curvilinear
trend (discussed later) can be estimated if the curvature of the baseline
trend is pronounced. Extrapolation (extension) of the baseline trend provides
the best available estimate of what the outcome would have been without
Jobs-Plus (the counterfactual).57
The deviation from the baseline trend in the first year
after Jobs-Plus begins (line D1 in the exhibit) provides an
estimate of the impact of Jobs-Plus for that year. Deviations from
trend in subsequent years (lines D2 and D3 ) provide
impact estimates for these years. Estimating the impact for each followup
year provides an easy way to describe the pattern of the impact over time
(whether it is constant, it decays, or it grows). As a way to summarize
this information, one can then average these estimates to determine the
mean annual impact during the followup period.58
For example, the hypothetical results in the bottom panel
of Exhibit 3.3 indicate that there was a 3 point
reduction in the percentage of residents who were on welfare in the first
year after Jobs-Plus began, an 11 point reduction in the second year,
and a 10 point reduction in the third year. The outcome was about 9 percentage
points lower than predicted, on average, during the followup period.
The following regression model can be used to estimate
the impacts in the exhibit:
where:
Yt = the value of the outcome variable
in year t,
Tt = one if year t is after Jobs-Plus began
and zero otherwise,
tt = the value for year t of the annual
counter,
B 0
= the impact of Jobs-Plus (the deviation from trend),
B 1
= the slope of the baseline trend,
a = the intercept
of the baseline trend.
If only years zero through six are included in the analysis,
the coefficient, B
0, equals the deviation from trend in year six (line D1
in the exhibit) and the t statistic for this coefficient provides a test
of its statistical significance. To include all three followup years in
the analysis, and allow each to have a separate deviation from trend,
one can replace T with a separate dummy variable for each year. The coefficient
for each dummy variable equals the deviation from trend for the year that
it represents (lines D1, D2, and D3),
and the t statistic for each coefficient provides a test of its statistical
significance. If years zero through eight are included in the analysis,
and one dummy variable is used to represent all followup years, then its
coefficient is the mean deviation from trend for years six through 8 (9
percentage points in the example above).
For an interrupted time-series analysis to be effective
there must be a stable baseline trend and a pronounced deviation
from this trend during the followup period. The more stable the baseline
trend is (the less the points vary around the trend-line), the more confidence
one can place in the forecast of the counterfactual for the followup period.
The larger and more abrupt the post-Jobs-Plus deviation from trend
is, the easier it will be to identify this deviation statistically.
Thus, for public housing characteristics that do not change
appreciably before Jobs-Plus, or that change gradually over time, there
might be enough statistical power to produce meaningful estimates of Jobs-Plus
impacts. For outcomes that change erratically before Jobs-Plus, the statistical
power of impact estimates probably will be limited.
Because we anticipate the impacts of Jobs-Plus to be large,
we are hopeful that an interrupted time-series analysis can identify them.
However, because these impacts might appear gradually over several years,
we are somewhat concerned that it will be difficult to identify them with
confidence. On balance, then, we are cautiously optimistic.
A separate interrupted time-series analysis could be
conducted for each outcome measure for each Jobs-Plus development.
One then could pool findings for a specific outcome measure across developments
by taking the mean of their impact estimates and computing the standard
error of this mean.59
Pooling these findings across sites will increase their statistical power.
One could account for non-linear trends, if the need arises,
by replacing the year counter (t) with its logarithm, by replacing the
outcome measure (Y) with its logarithm, or both. This will allow for curvature
without reducing the number of degrees of freedom.
Impact Study Design
#3: Interrupted Time-Series Analysis
for Jobs-Plus Developments and Comparison Developments
A logical extension of the preceding analysis is to construct
a separate interrupted time-series for a comparison development where
Jobs-Plus is not implemented. Exhibit 3.4 illustrates
how this time-series can be used to improve estimates of the impacts of
Jobs-Plus. The approach is applicable regardless of how the comparison
development was chosen (whether random assignment was used or not, and
whether matching was used or not).
The top panel of Exhibit 3.4
repeats the time-series for the hypothetical Jobs-Plus development
in Exhibit 3.3. The bottom panel presents findings
during the same period for a comparison development. The interrupted time-series
analysis for the comparison development yields deviations from trend in
years 6, 7 and 8 equal to E1, E2, and E3,
respectively. If the comparison development and the Jobs-Plus development
were chosen from the same local environment, then the comparison development
deviation from its trend provides an estimate of what the deviation would
have been for the Jobs-Plus development without Jobs-Plus (the counterfactual).
Hence, Dt - Et provides an estimate
of the impact of Jobs-Plus in year t. The variance of this difference
equals the sum of the variances of Dt and Et, so
one can readily test the statistical significance of the difference.60
It is useful to proceed with this analysis in two steps.
First, compute and test the deviation from trend, Dt, for the
Jobs-Plus development. This step asks the question: "Was
there a statistically significant improvement in the outcome for the Jobs-Plus
development?"61
Next compute and test the difference between the two deviations from trend,
Dt - Et,. This step asks the question: "Did
the outcome improve by an amount that was statistically significantly
greater than its likely change without Jobs-Plus?"62
There are many ways to pool these findings for a sample
of Jobs-Plus developments and comparison developments. The simplest way
to do so is to subtract the mean deviation for the comparison developments
from the mean deviation for the Jobs-Plus developments and compute the
standard error for this difference of means.63
One important variation on the interrupted time-series
theme occurs when only two or three baseline observations exist. This
will apply to Jobs-Plus outcomes that have limited consistent past data
because: (1) their collection began only recently, (2) the way they are
collected changed recently, or (3) they are only kept for short periods.
Two or three baseline observations are better than only one (for before-after
analyses), and are less effective than more baseline observations (for
interrupted time-series analyses). But how should this information be
used?
A conservative approach is to assume that the underlying
system is not changing rapidly over time. Under this assumption, one would
use the mean value of the baseline outcome to forecast the future outcome
without Jobs-Plus (the counterfactual). The observed difference between
the actual future outcome and this forecast provides an estimate of the
impact of Jobs-Plus. Comparing this estimated difference with the corresponding
difference for a comparison development could further strengthen the finding.
Pooling these findings across developments could strengthen the findings
even further. In all cases, simple tests for statistical significance
are possible.
A less conservative approach is to assume that the observed
annual change in the outcome for the baseline period will continue into
the future.64
Hence, this annual change rate becomes the counterfactual. One then could
compare the annualized change rate for the baseline period with the annualized
change rate for the followup period.
Instead of annual data for an outcome, it might be possible
to obtain information for shorter units of time for example, monthly
crime rates. Data for shorter units of time provide more observations,
which in turn, make it possible to use more sophisticated time-series
models to estimate program impacts (see McCain and McCleary, 1979, and
McDowall, McCleary, Meidinger, and Hay, 1980).65
Such models can enable researchers to accommodate the
serial correlation of observations over time and account explicitly for
the effects of seasonality. There are probably very few Jobs-Plus
outcomes for which such data are available, however. In addition, the
ability of these data to represent long-term trends is not necessarily
greater than the ability of annual data to do so. Furthermore, data for
shorter time periods have a larger relative variance and greater serial
correlation.66
Hence, procedures based on such data probably will have limited application
to the Jobs-Plus impact study. Nevertheless, they will be considered.
IV.
Jobs-Plus Impacts on the Community
Introduction
This section explores ways to measure the impacts of
Jobs-Plus on the surrounding community of a public housing development.
The focus here is on changes in an area over time caused by changes
in people and firms who remain in the area plus differences
in the people and firms who move in and out of the area.
I first present a simple model of how Jobs-Plus might
affect a local community and identify some outcome measures that could
be used to examine these effects. I then briefly discuss issues that must
be addressed when trying to identify the relevant community for a public
housing development. I next examine approaches for estimating Jobs-Plus
impacts. Because of the elusive and indirect nature of these impacts,
and because of the limited data available for estimating them, the approaches
available for this purpose are less well-structured than those discussed
in earlier sections of this paper.
A Model of Jobs-Plus
Impacts and Measures of Community Outcomes
Exhibit 4.1 presents a simple
model of how Jobs-Plus might affect a community that is located
near a public housing development where the initiative is implemented.
It begins with the three basic elements of the Jobs-Plus program, which
are expected to produce positive changes in the lives of public housing
residents and improvements in the living conditions at their developments.
As the environment in the development improves, it can start to become
a "positive anchor tenant" for the surrounding community. Much
like a major department store draws other establishments to a shopping
center, a newly-transformed public housing development could attract residential
and commercial development to its surrounding community. Similarly, it
might stimulate physical improvements, increased public services, healthier
and more stable social conditions, increased civic engagement, and increased
community empowerment. Exhibit 4.2 lists a number
of possible outcome measures that could be used to test for community
improvements.
It should be noted, however, that while it is possible
for improved conditions at a public housing development to spill over
into its surrounding community, and thereby leverage improvements to a
wider area, these spillover effects are indirect and secondary, relative
to the direct effects of Jobs-Plus on public housing residents and the
developments in which they live. In addition, these effects might take
a long time to materialize, given the many forces that must converge in
order for them to occur. Hence, of all the impacts hypothesized for Jobs-Plus,
community effects are the most speculative. This is especially true
given the many outside forces such as macroeconomic conditions,
racial conflict, government funding decisions, and other public policies
that play a large role in determining the condition of an urban
community.67
Defining the Appropriate Community
A community usually refers to a social, rather than a
geographic, unit.68
However, to assess the impacts of Jobs-Plus will require defining local
communities in spatially distinct terms. To do so, will present a major
challenge, because there is little agreement about what constitutes a
community.69
Nevertheless, four factors should be considered when trying to delineate
community boundaries:
- physical barriers, such as rivers, highways, parks,
railroad tracks, and large-scale industrial, commercial or residential
development;
- social interactions, such as shopping patterns, visiting
patterns, church attendance, and recreational patterns;
- political units, such as voting wards or districts,
and school districts;
- statistical units for which data are reported, such
as census blocks or tracts, police precincts, neighborhood planning
districts, and zip codes.
Local residents should be involved in defining community
boundaries. Their input will help to ensure that the relationship between
the public housing development and the residential, commercial, and industrial
development adjacent to it is considered when the analytic boundaries
of a developments community are established. The key here is to
designate an area to which the public housing development is, or could
be, genuinely related when the barriers which tend to isolate public housing
are removed. If the designated area is too small, however, estimates of
the impacts of Jobs-Plus on it might omit some important spillover effects.
If the area is too large, estimates also will understate the true impacts
of Jobs-Plus, because they will be diffused too widely to be identifiable.
In the end, pragmatic considerations of data availability will play a
major role in determining community boundaries for analysis purposes.
One important issue in this regard is the fact that boundaries for reporting
areas shift over time; hence, the comparability of such data should be
examined very carefully.
Issues to Consider
When Measuring Community Impacts
Before discussing specific approaches for estimating
the community impacts of Jobs-Plus, it is important to acknowledge some
key issues that will arise when attempting to measure these impacts.
First, it probably will be necessary to use different
methods to measure impacts on different outcomes. Some outcomes will be
measurable through quantitative methods, and some will be measurable through
qualitative methods.
Second, due to limited data and resources, it might be
necessary to restrict certain impact analyses to a few Jobs-Plus sites,
and limit their generalizability accordingly.
Third, because of the indirect, diffuse, and somewhat
speculative nature of the potential community impacts of Jobs-Plus,
resources probably should be limited to measuring only those impacts judged
most likely to occur, and most likely to be observable. In particular,
the primary emphasis of the impact analysis for Jobs-Plus should
be on its direct effects on public housing residents and public housing
development residents, discussed earlier.
Fourth, an explicit emphasis should be placed on "triangulating"
impact estimates.70
This means trying to gauge the effects of Jobs-Plus in as many different
ways as possible. Each method will be subject to different limitations,
but if the findings produced by different methods tell the same basic
story, a plausible empirical case can be made.
Fifth, one should expect the findings from any analysis
of the community impacts of Jobs-Plus to be "messy."
Hence, the story represented by these findings will be difficult to tell,
and will require a high degree of craftsmanship from its authors.
Sixth and last, one should not proceed with a detailed
analysis of the community impacts of Jobs-Plus, unless the initiative
has been successful in terms of its implementation, and its impacts on
public housing residents and developments.
Having acknowledged these conditions, the following are
some potential approaches that might be used to assess the community impacts
of Jobs-Plus.71
This approach is mainly subjective and, hence, relies
heavily on qualitative methods. The basic idea is quite simple. In communities
where Jobs-Plus is implemented, one could ask public housing residents,
other community residents, community merchants, and community leaders
how, if at all, Jobs-Plus improved their community. Through a combination
of pre-coded and open-ended questions, one could elicit responses that
provide specific examples of community improvements that were believed
to be caused by Jobs-Plus and detailed explanations for why these outcomes
should be attributed to Jobs-Plus.
These questions could be asked through sample surveys,
in-depth personal interviews, focus groups,72
or some combination thereof. They could be asked at one point in time
after Jobs-Plus had begun (presumably long enough after it had started
to allow for community effects to occur), or at several points in time.
The basic approach would be to look backwards in time, try to identify
community improvements that had occurred during the period of interest,
and try to articulate how Jobs-Plus caused these improvements.
Two strengths of this approach are its feasibility and
its relative simplicity. More important, perhaps, is the fact that it
would enable participants in the Jobs-Plus process to express in their
own words and from their own perspectives how the initiative improved
their community. This feature satisfies a key requirement of qualitative
research methods and can, in some cases, produce highly convincing findings.73
A potential weakness of the approach is the possibility
that enthusiastic Jobs-Plus stakeholders will overstate its community
impacts. Requiring specific examples of impacts and detailed explanations
for how these impacts were created by Jobs-Plus, might restrain this tendency
somewhat, and might provide a factual basis for gauging the validity of
the approach. Nevertheless, there always will be a margin for overstating
the initiatives accomplishments. Hence, the approach should be used
only in conjunction with other, more objective, although perhaps less
rich, methods.
A detailed case study, conducted at one or more of the
Jobs-Plus projects, could provide important insights into how and why
the initiative did or did not produce community impacts. For this approach
to be most effective, it should focus on "telling the Jobs-Plus story"
in a structured way, that helps to develop and test a theory of change
for the initiative (discussed earlier). This theory would trace the key
intervening steps through which the initiative changes attitudes and behavior,
and thereby produces community improvements.
A detailed case study would require extensive and continual
presence by research staff at a Jobs-Plus site. It could include, among
other activities: (1) extensive repeated interviews with key participants
in Jobs-Plus; (2) attendance at Jobs-Plus meetings, planning sessions,
and events; (3) focus groups with community residents, merchants, and
leaders; (4) physical inspections of community conditions through windshield
surveys and other "on-the-ground" approaches; and (5) access
to written records and newspaper accounts that document key Jobs-Plus
events, decisions, and actions. These activities could form part of the
basis for the research on Jobs-Plus planning and implementation discussed
earlier, as well as analyses of program impacts. Such extensive on-site
presence might require a local researcher, much like that used to study
the HOPE VI Program at the Hillside Terrace housing development in Milwaukee,
Wisconsin.75
Case studies can utilize both qualitative methods and
quantitative methods. In addition, they can combine in-depth analyses
of how Jobs-Plus evolved over time and the key forces which shaped its
development, with descriptive analyses of existing conditions at one or
more points in time (for example, the results of a sample survey of existing
attitudes or the results of a windshield survey of existing physical conditions).
As indicated above, the "glue" that will be needed to hold these
disparate analyses together is a well-specified theory of how Jobs-Plus
produced community impacts. Unfortunately, it is simple to state this
requirement, but difficult to meet it. Nevertheless, without doing so,
one cannot properly interpret the findings from a case study.
The primary strength of a case study is its richness of
detail because of its depth of analysis. In particular, case studies that
incorporate ethnographic methods76
might be able to capture the cultural context of Jobs-Plus in a way that
will help explain community residents responses to it. Such observations
can provide a framework and perspective within which to understand some
of the more quantitative measures that are obtained.
The primary weakness of a case study is its limited generalizability
because of the limited number of cases that can be studied in this way.
Hence, the more Jobs-Plus sites to which a case study approach is applied,
the greater the generalizability of its findings will be. In addition,
the more sites that are studied in this way, the greater the opportunities
will be for obtaining knowledge through cross-site comparisons. The corresponding
downside of increasing the number of sites, however, is the markedly increased
cost of doing so. Nevertheless, because of the newness, the complexity,
and the intensity of Jobs-Plus, it would seem prudent to conduct one or
more case studies to learn as much as possible about the inner workings
of the initiative.
For community conditions that can be measured at two
or more points in time, such as crime rates, the number of local business
establishments, the number of local jobs, and the amount of property tax
revenues generated, one could conduct a before-after analysis for Jobs-Plus
communities and comparison communities.
The basic approach would be the same as that described
earlier for measuring the impacts of Jobs-Plus on public housing residents
and public housing developments. The key difference here is that new outcome
measures, and their corresponding data-sources, become relevant.
In particular, measures of Jobs-Plus impacts on community
economic activities might be important. Unfortunately, previous attempts
to measure the impacts of government programs on local economic activity,
especially the effects of community economic development programs and
enterprise zones, have met with limited success (see James, 1991, and
Vidal, 1995).
One promising approach for overcoming this problem utilizes
data from the Duns Market Identifiers (DMI) file, which provides almost
a complete annual census of business establishments in major U.S. cities.77
This information has been collected for several decades, and has been
used by previous researchers to analyze patterns of industrial location.78
Available data about individual business establishments include, among
other characteristics: (1) their start-up date (which could be used to
identify new businesses); (2) their standard industrial classification
(SIC, which could be used to identify local industries that are growing
or declining); (3) their number of employees (which could be used to measure
job growth or decline), and (4) their postal zip codes (which could be
used to identify establishments within Jobs-Plus communities or comparison
communities).
DMI data are expensive to obtain, so they should be used
on a limited basis. For example, they could be obtained for the year that
Jobs-Plus begins, and for some later year (perhaps four or five years
later) for the zip code(s) which most closely approximate a Jobs-Plus
community. From this information, one could estimate the total number
of establishments and the total number of jobs (overall and by industry)
in each of the two years. One then could compute the absolute and percentage
changes in these characteristics. In addition, one could determine the
number and percentage of new establishment that appeared in the community
and former establishments that no longer exist.79
This would enable one to describe the changes in employment levels and
business establishments that occurred during the analysis period.
A simple but more expensive extension of this approach
would be to also obtain DMI data for some time in the past, say, for example,
five years before Jobs-Plus was implemented. This would provide a measure
of the change in employment and in the number of establishments
(overall and by industry) during a relatively long period before and
after Jobs-Plus was implemented. One then could examine the change
in the annualized growth rate, which would provide a limited interrupted
time-series analysis.
The primary strength of the preceding approach is its
potential for providing objective information about the community impacts
of Jobs-Plus. It main weaknesses are the expense of purchasing
and processing the necessary data, and the methodological limitations
of the before-after analyses that would be possible. For example, just
because changes are observed in the employment levels of a community,
or just because changes in growth rates are observed, does not mean that
they were caused by Jobs-Plus. This causal inference could be strengthened,
if corresponding DMI data were obtained for comparison communities. But
even with these data, it would not be possible to conclude definitively
that the change observed for the comparison communities represents what
the corresponding change would have been for the Jobs-Plus communities
without Jobs-Plus.
Hence, causal inferences about the community impacts of
Jobs-Plus will need to rely heavily on supplementary findings about
the Jobs-Plus theory of change. If one can establish that the preconditions
for Jobs-Plus to produce community impacts were met (e.g.,
positive impacts on public housing residents and developments were observed,
and the program was implemented successfully), then it might be possible
to build a case that Jobs-Plus caused the community changes observed.
If the community changes are large and occur consistently across Jobs-Plus
sites, this case could be even stronger.
V.
Concluding Thoughts
Jobs-Plus is planned to be a large-scale, saturation-level
employment program for residents of selected public housing projects.
As such, it represents a comprehensive community initiative whose impacts
cannot be estimated from a randomized experiment, the now widely accepted
best way for assessing the impacts of employment programs. Hence, a variety
of non-experimental evaluation approaches are being considered for this
project.
Given the relatively large impacts that are anticipated,
it is hoped that a careful assessment of how key outcome measures change
over time, both at Jobs-Plus "treatment developments" and at
nearby comparison developments, will provide an adequate basis for measuring
the key impacts of the program. Because many future programmatic initiatives
are likely to focus on whole groups at a time instead of individual participants,
it is hoped that the evaluation approaches being developed for Jobs-Plus
will be applicable to evaluations of these future projects as well.
NOTES:
[1] Based on information
from the 1989 American Housing Survey, Casey (1992, p. 11) estimates that
45 percent of the households in public housing received welfare or SSI.
Based on information from the 1987 American Housing Survey, Newman and
Schnare (1992, p. 56) estimate that 49 percent of the families with children
in public housing and 19 percent of the elderly households in public housing
received welfare or SSI.
[2] PHAs can increase
the percentage of public housing residents who are employed in two ways:
(1) by helping current residents find and hold jobs, and (2) by attracting
low-income adults who are employed into public housing. The present paper
focuses mainly on the first strategy.
[3] Because of
the extreme complexity of Jobs-Plus, it is especially important to base
its evaluation on an explicit theory, or set of theories, about how the
initiative is expected to work, and what outcomes it is expected to produce.
Chen (1990), Chen and Rossi (1987), and Bickman (1987), among others,
have argued that such "theory-driven" evaluations are necessary
in order to advance our understanding of social programs. More recently,
Weiss (1995), Connell and Kubisch (1995), and Connell, Aber and Walker
(1995) have argued that a well-specified theory of change is essential
for the evaluation of "comprehensive community initiatives,"
of which Jobs-Plus is a prime example (discussed later).
[4] Gueron and Pauly
(1991) provide the most extensive review of this research. Their summary
of findings (pp. 15-20) clearly indicates that employment and training
services can increase the future earnings of welfare recipients, although
the magnitudes of these increases vary substantially across different
types of programs, different types of participants, and different locations.
In addition, recent findings from a large-scale study of Job Opportunity
and Basic Skills (JOBS) programs in three sites Atlanta, Georgia,
Grand Rapids, Michigan, and Riverside, California (Freedman and Friedlander,
1995, p. ES-5) indicate that two years after single parents on
AFDC were offered labor market attachment services (to facilitate quick
employment), their average monthly earnings were 26 percent higher
than they would have been without the program. Furthermore, Goldman (1995,
Table 5) indicates that AFDC recipients from Atlanta (the only site for
which such findings are available currently) who lived in public housing
when they entered the study, and subsequently were offered labor market
attachment services, earned 56 percent more per month, two years
after they entered the study than they would have earned without the JOBS
program.
[5] The implicit tax
rate on the earnings of low-income families often exceeds 100 percent.
In other words, for each additional dollar earned, a family can lose more
than one dollar of cash plus in-kind benefits. This situation reflects
the combination of offsets to earnings that occurs if a family is receiving
more than one type of benefit (e.g., AFDC, food stamps, Medicaid,
and subsidized housing). The work disincentive is especially severe if
a familys income is near the threshold (or "notch") where
it becomes ineligible for Medicaid. Work disincentives from housing subsidies
are smaller than those from public welfare. Nevertheless, they are substantial,
and have played an important role in housing policy debates (e.g.,
see PHADA/GAHRA, 1994; Wilkins, 1993). Perhaps more important, however,
is the combined effect of work disincentives from both welfare
and housing subsidies.
[6] Danziger et
al. (1981) and Moffit (1992) survey the extensive literature on the
effect of these work disincentives, the findings of which are mixed. To
help resolve this controversy, two recent studies, the Canadian Self-Sufficiency
Project (SSP) and the Minnesota Family Investment Program (MFIP), are
using large-scale randomized experiments to test welfare reform plans
that make work pay (see Card and Robins, 1996, for a discussion of SSP,
and Knox, Brown, and Lin, 1995, for a discussion of MFIP). Initial findings
for SSP indicate that making work pay can, indeed, help to increase employment.
For example, in the fifth quarter after welfare recipients enrolled in
SSP, they earned 58 percent more than they would have without the
programs substantial financial incentive. It is too soon to know
how long these impacts will last, however. Corresponding findings are
not available yet for MFIP.
[7] Findings from
a major early study of resident management in seven public housing developments
located in six cities (Manpower Demonstration Research Corporation, 1981)
plus findings from a more recent study of 80 emerging resident management
corporations from across the U.S. (ICF, Incorporated, 1993) support this
conclusion.
[8] Building on
Wilsons (1987) seminal work, many authors have identified the absence
of a positive, work-oriented social infrastructure (positive adult role
models, well-attended churches, active neighborhood associations, etc.)
in public housing, due to the extreme concentration of poverty there,
as one of the greatest obstacles to its social and economic stability
(e.g., see Spence, 1993, and Schill, 1993).
[9] The research agenda
for Jobs-Plus includes an in-depth implementation study. However, the
present paper focuses only on estimating the impacts of Jobs-Plus.
[10] This recognition
has evolved over the past two decades based on careful methodological
research. For example, after reviewing a wide range of sophisticated quasi-experimental
estimates of the impacts of programs funded by the Comprehensive Employment
and Training Act (CETA), an expert panel convened by the U.S. Department
of Labor to help it decide how to evaluate the next generation of such
programs, funded under the Job Training Partnership Act (JTPA), concluded
that a randomized experiment was essential for this purpose (Stromsdorfer
et al., 1985). Likewise, based on an extensive review of non-experimental
research about the effectiveness of employment and training and programs
for youths, a committee of experts convened by the National Academy of
Sciences concluded that: "Future advances in field research on the
efficacy of employment and training programs will require a more conscious
commitment to research strategies using random assignment" (Betsey,
Hollister, and Papageorgiou, 1985, p. 30). These conclusions reflect two
main findings from methodological research on different methods for measuring
the impacts of employment and training programs:
- Different statistical matching and modeling techniques
produce widely varying impact estimates (sometimes ranging from statistically
significantly positive to statistically significantly negative) when
applied to the same data-sets. Unfortunately, "Data limitations
and the inability to adequately test the validity of the selection processes
assumed make it impossible to determine which studies modeled the process
correctly" (Barnow, 1987, p. 157).
- When impact estimates from a wide range of quasi-experimental
matching and modelling techniques were compared to corresponding impact
estimates from randomized experiments, the quasi-experimental estimates
varied widely and differed markedly from the experimental estimates
(for example, see Fraker and Maynard, 1987, LaLonde, 1986, and Friedlander
and Robins, 1995).
[11] More complex
experiments randomly assign eligible applicants to a control group or
to one of several different program groups. This makes it possible to
estimate the "differential" impacts of different types of programs.
[12] More precisely,
the "expected value" of every characteristic of the program
group (whether it is measurable or not) equals its "expected value"
for the control group. The larger the study sample is, the more similar
the actual treatment and control group means will be for all characteristics.
[13] Kennedy
(1988) reviews the findings from these studies. Apgar (1990) describes
the influence of these findings on housing policy debates.
[14] Feins (1993)
describes the proposed research design for this project.
[15] Impact estimates
obtained from this design would be unbiased, but would have limited statistical
power because of the small number of housing developments involved.
[16] Perhaps the
most successful quasi-experimental research in the field of housing policy
is the series of Fair Housing Audits which measured the incidence, nature,
and intensity of discrimination against minority group members by landlords
and real estate brokers. These studies provide compelling evidence about
housing discrimination, in part because of their rigorous design, and
in part because of the high levels of discrimination that exist
in rental housing markets (for a review of these studies, see Yinger,
1988). Likewise, a major quasi-experimental study of saturation-level
guaranteed jobs programs for in-school youths provided convincing findings
about the effect of such job guarantees on the short-term employment rates
of youth because these impacts were so striking and so immediate
(Gueron, 1984). Another quasi-experimental study that has had a major
impact on housing policy debates is that of the Gautreaux Program to promote
fair housing in Chicago (see, for example, Rosenbaum, 1995). This study
approximated a natural experiment whereby on a "nearly random"
basis, some families received Section 8 housing subsidies for rental units
in middle income suburban or outlying urban neighborhoods, while others
received subsidies for units in low-income, inner-city neighborhoods.
Adults who moved to middle income neighborhoods were much more likely
to become employed than were those who remained in low-income neighborhoods.
[17] The conceptual
framework used here was developed by Kubisch, Weiss, Schorr, and Connell
(1995) , pp. 3-5.
[18] See U.S. Government
Accounting Office (1992) for a discussion of this issue.
[19] See Newman and
Schnare (1992) and Schlay (1993) for an extensive discussion of the issues
surrounding the economic self-sufficiency of public housing residents
and previous attempts to address these issues.
[20] Shlay and Holupka
(1992) also based part of their analysis on a secondary comparison group
of persons from Lafayette Courts who did not participate in the Family
Development Center.
[21] This approach
was used by Bryant and Rupp (1987) to estimate the impacts of employment
and training programs funded by CETA, the Comprehensive Employment and
Training Act.
[22] This approach
was used by Dickinson, Johnson, and West (1987) to estimate the impacts
of employment and training programs funded by CETA.
[23] Evaluation
researchers refer to this as the problem of "history" (Cook
and Campbell, 1979).
[24] Evaluation
researchers refer to this problem as differential "maturation"
(Cook and Campbell, 1979).
[25] Using other
baseline characteristics as a proxy for these trends is a weak substitute
for having data which describe them.
[26] See Ashenfelter
(1978), Ashenfelter and Card (1985), Bloom (1984), and Bloom (1987) for
a discussion of this issue.
[27] See Cook and
Campbell (1979).
[28] Pre-program periods,
and post-program periods for both groups are defined as calendar periods
that occur before and after the program group is first exposed to the
program.
[29] This statement
applies to many different outcomes in many different fields (for example,
employment and earnings, receipt of welfare and housing subsidies, homelessness,
smoking, alcohol abuse, drug abuse, criminal behavior, child abuse, etc.)
Conceptually, the statement is plausible because past outcomes can serve
as a proxy for their many separate causes. Empirically, the statement
is borne out by the fact that multiple measures on past outcomes usually
provide much better predictions of future such outcomes than do data on
separate hypothesized causal factors.
[30] Bloom (1984)
used this approach and refers to it as a "time-varying, fixed-effect
model." Ashenfelter and Card (1985) used this approach and refer
to it as a "random coefficients" model. Although their computational
procedures are different, the basic models are the same.
[31] If such additional
variables are not included, then it turns out that the point estimate
of the program impact based on the micro-data for all sample members (the
difference between the mean deviation from trend for program group members
and the mean deviation from trend for comparison group members) is identical
to the point estimate that can be obtained from aggregate data on the
mean outcomes for each group for each period, by computing the difference
between the program groups deviation from its aggregate trend and
the comparison groups deviation from its aggregate trend (Bloom,
1984).
[32] This problem
applies equally to before-after, two-group comparisons.
[33] Researchers
cannot agree on how to solve this problem for quasi-experimental longitudinal
evaluations of employment and training programs (see Stromsdorfer et
al., 1985, and Barnow, 1987). Hence, randomized experiments are now
used to evaluate these programs.
[34] After the initiative
begins, some families might choose to move into a Jobs-Plus
development to increase their economic self-sufficiency. For these
families, the selection process might be more like that of a traditional
employment and training program. Nevertheless, because the choice of a
housing unit involves many considerations beyond those which motivate
enrollment in a job-training program, there probably is less room for
the kind of self-selection on labor market potential that occurs in job-training
programs.
[35] We refer to control
developments here rather than comparison developments because we are discussing
a true randomized experiment.
[36] An impact estimator
based on the random assignment of Jobs-Plus developments and control developments
is unbiased because the expected value of its sampling distribution
equals the true impact being estimated. The estimator is inefficient (imprecise
or uncertain), however, because its standard error is larger than
the standard error for a sample with the same number of subjects that
were randomly assigned individually to the program group and the control
group (see Raudenbush, 1995, and Hays, 1973).
[37] See Bloom (1995)
for a discussion of how to define and compute minimum detectable effects.
[38] For example,
the National JTPA Study used a sample of 16,000 persons from 16 sites
across the U.S. (Bloom et al., 1995).
[39] For a given total
sample size, the cluster effect increases as the size of the clusters
increase, and as the average difference between the clusters increases
(see Raudenbush, 1995).
[40] For clusters
of 200 subjects each and an intra-class correlation of 0.01 (which is
small), the standard error of a sample produced by cluster assignment
will be 1.7 times that produced by random assignment of individual subjects.
For an intra-class correlation of 0.10 (which is large), the standard
error for cluster assignment will be 4.8 times that for random assignment.
The intra-class correlation measures how much the clusters differ from
each other relative to how much individual subjects differ from each other.
[41] Other things
equal, the larger the developments, the less variation in their year-to-year
change in employment rates. We do not address this feature of the problem,
however, because information about it (specifically, the intra-class correlation)
does not exist. Instead, we try to tell a story about a range of variation
that might exist for relatively large developments. If our story understates
cross-site variation, it overstates the statistical power of Jobs-Plus
impact estimates. If it overstates cross-site variation, it understates
statistical power. We tried to construct an example with a lot of cross-site
variation, so it would not overstate statistical power. A more informed
estimate based on actual longitudinal data for earnings, employment, AFDC
receipt, and receipt of food stamps is being developed currently.
[42] This finding
represents the standard error of a difference of means, with five observations
(developments) in the sample for the program group and five observations
(developments) in the sample for the comparison group.
[43] With eight degrees
of freedom, the critical t value at the .05 significance level is 1.86
for a one-tail test and 2.31 for a two-tail test.
[44] These models
also are referred to as random coefficients models (Ashenfelter and Card,
1985).
[45] This program
consists of a two-year "remedial stage" and a five-year "transition
stage."
[46] Employment
and earnings data were collected retrospectively at baseline for a period
of several years for part of the sample in the National JTPA Study (Bloom
et al., 1990).
[47] The baseline
period will be shortest for residents who entered public housing most
recently.
[48] This information
is more useful for measuring the impacts of Jobs-Plus on public housing
developments that participate in the initiative (discussed later).
[49] Workers who
are not covered by Unemployment Insurance include mainly self-employed
persons and certain types of federal employees.
[50] For a detailed
comparison of earnings and employment data from UI wage records and from
a survey for the same sample and the same time period, see Bloom et
al., 1993a, Appendix E, pp. 345-366. Average earnings from UI wages
records were about 25 percent lower than those from the survey; program
impact estimates from the two data-sources were similar, especially when
expressed in percentage terms.
[51] See, for example,
Bloom et al. (1993b).
[52] Spence (1993)
and Schill (1993) emphasize this issue.
[53] Recall that
if random assignment is used, we will have a control group rather than
a comparison group.
[54] See the discussion
of sample size and minimum detectable effects in Section 2.
[55] Cook and Campbell
(1979) outline the conceptual basis for interrupted time-series analysis.
McCain and McCleary (1979) and McDowall, McCleary, Meidinger, and Hay
(1980) describe how to apply time-series models to estimate interrupted
time-series. Figlio (1995) provides an excellent application of this approach
to measuring the effect of lowering the drinking age on alcohol-related
traffic accidents.
[56] To do so, one
could use the rent rolls of a PHA to identify who had lived in a particular
development in a specific month each year for example, January.
One then could then obtain AFDC data for each sample member from the records
of local or state welfare agencies. For each year that a sample member
lived in the development, his/her AFDC data would be part of the time-series
for that development.
[57] This implies
that the best predictor of future behavior is long-term past behavior,
which is the case for most outcomes.
[58] If there are at
least four followup periods, it is possible, in theory, to estimate the
impact of the program on the intercept and the slope of the original trend-line.
We do not take this approach, however, because it does not focus directly
on the magnitudes of the actual annual impacts and, hence, is more difficult
to interpret.
[59] For example,
if in followup year one, the estimate of the impact of Jobs-Plus
for a single development is D, and the estimated variance of this estimate
is s 2,
then the variance of the mean estimate for all n Jobs-Plus developments
that year one is s
2/n.
[60] This simple
formulation for the variance is possible because the two developments
represent independent, non-overlapping samples.
[61] This implies
a one-tail hypothesis test.
[62] This also implies
a one-tail hypothesis test.
[63] It is also possible
to specify a pooled regression to estimate this finding directly.
[64] Using the mean
of the baseline outcomes to forecast future outcomes is more conservative
than using the annualized change rate because the mean dampens the effect
of year-to-year noise in the forecast, whereas the annualized change rate
accentuates the effect of this variation.
[65] These procedures
are generally referred to as Box-Jenkins models.
[66] By relative variance,
we mean the "noise to signal ratio," which might be measured
as the coefficient of variation.
[67] See Halpern (1995)
for a discussion of this issue.
[68] See Coulton (1995),
pp. 174-175, for a discussion of this issue.
[69] For a discussion
of problems that arise when trying to define the boundaries of a community
involved in a comprehensive community initiative, see Hollister and Hill
(1995), pp. 130-131.
[70] See Jick (1979)
for a discussion of triangulation, in general. See Brewer and Hunter (1989)
for a discussion of how to triangulate alternative measures of a construct
by testing their convergent and discriminant validity.
[71] The approaches
discussed below also could be (and probably should be) used to study the
impacts of Jobs-Plus on public housing developments. They are introduced
here, however, because they are some of the few options which exist for
measuring community impacts.
[72] Krueger (1994)
explains in detail how to use focus groups for such purposes.
[73] Patton (1987)
and Bryman (1988) discuss the importance of obtaining data from the frame
of reference of participants in a process and in their own words.
[74] See Yin (1989)
and Yin (1993) for an extensive discussion of this approach.
[75] See University
of Wisconsin-Extension (1995).
[76] Examples
of such ethnographic research include Stack and Burton (1994), Stack (1974),
and Leavitt (1994).
[77] One problem with
these data is the fact that they provide uneven coverage of very small
firms, and firms in personal services industries (Schwartz, 1987).
[78] The coverage
and quality of these data have improved markedly over time. Schwartz (1987)
discusses issues that arise when using these data. Struyk and James (1975)
apply them to the analysis of industrial location patterns.
[79] Schwartz
(1987) suggests, however, that estimates of the number of new establishments
in an area and estimates of the number of establishments that leave an
area, based on DMI data, probably overstate these outcomes.
REFERENCES:
Allison, Paul D. 1988. Event History Analysis: Regression
for Longitudinal Event Data (Beverly Hills, CA: Sage Publications).
Apgar, William. 1990. "Which Housing Policy is Best?"
Housing Policy Debate, Vol. 1, No. 1, 1-32.
Ashenfelter, Orley and David Card. 1985. "Using the
Longitudinal Structure of Earnings To Estimate the Effect of Training
Programs", Review of Economics and Statisics, Vol. 67, November,
pp. 648-660.
Ashenfelter, Orley. 1978. " Estimating the Effect
of Training Programs on Earnings", Review of Economics and Statistics,
Vol. 60, February, pp. 47-57.
Barnow, Burt S. 1987. "The Impact of CETA Programs
on Earnings: A Review of the Literature". Journal of Human Resources
22 (Spring): 157-93.
Betsey, Charles, L., Robinson G. Hollister, and Mary R.
Papageorgiou. 1985. Youth Employment and Training Programs: The YEDPA
Years. Committee on Youth Employment Programs, Commission on Behavioral
and Social Sciences and Education, National Research Council. Washington.
D.C.: National Academy Press.
Bickman, Leonard, 1987. "The Functions of
Program Theory" in Leonard Bickman ed. Using Program Theory in
Evaluation, New Directions for Program Evaluation, No. 33 (San
Francisco: Jossey-Bass).
Bloom, Howard S. 1995. "Minimum Detectable Effects:
A Simple Way to Report the Statistical Power of Experimental Designs",
Evaluation Review, Vol. 19, No. 5, pp. 547-556.
Bloom, Howard S. 1990. Back to Work: Testing Reemployment
Services for Displaced Workers (Kalamazoo, MI: W.E. Upjohn Institute
for Employment Research).
Bloom, Howard S. 1987. "What Works for Whom: CETA
Impacts for Adult Participants" Evaluation Review, Vol 11,
No. 4, pp. 510-527.
Bloom, Howard S. 1984. "Estimating the Effect of
Job-Training Programs Using Longitudinal Data: Ashenfelters Findings
Reconsidered" Journal of Human Resources, Fall, pp. 544-556.
Bloom, Howard S., Barbara Fink, Susanna Gurr and Wendy
Bancroft. 1995. "The Canadian Earnings Supplement Project: Subsidizing
Reemployment Instead of Subsidizing Unemployment" (New York: Social
Research and Demonstration Corporation, November).
Bloom, Howard S., Larry L. Orr, Stephen H. Ball, George
Cave, Fred Doolittle, Winston Lin and Johannes M. Bos. 1997. "The
Benefits and Costs of JTPA Title II-A Programs." Journal of Human
Resources, Vol. 32, No. 3 (Summer): 549-576.
Bloom, Howard S., Larry L. Orr, George Cave, Stephen H.
Bell and Fred Doolittle. 1993a. "The National JTPA Study: Title II-A
Impacts on Earnings and Employment at 18 Months" (Washington, D.C.:
U.S. Department of Labor).
Bloom, Howard S., Larry L. Orr, George Cave, Stephen H.
Bell, Fred Doolittle and Winston Lin. 1993b. "The National JTPA Study,
Overview: Impacts, Benefits and Costs of Title II-A" (Bethesda MD:
Abt Associates Inc., November).
Bloom, Howard S., Larry L. Orr, Fred Doolittle, Joseph
Hotz and Burt S. Barnow. 1990. "Design of the National JTPA Study"
(Bethesda, MD: Abt Associates Inc.).
Brewer, John and Albert Hunter. 1989. Multimethod Research:
A Synthesis of Styles (Newbury Park, CA: Sage Publications) pp. 126-137.
Brown, Prudence. 1995. "The Role of the Evaluator
in Comprehensive Community Initiatives" in James P. Connell, Anne
C. Kubisch, Lisbeth B. Schorr and Carol H. Weiss, eds. New Approaches
to Evaluating Community Initiatives: Concepts, Methods, and Contexts
(Washington, D.C.: The Aspen Institute).
Bryman, Alan. 1988. "The Nature of Qualitative Research"
in Bryman, Quantity and Quality in Social Research (London: Unwin
Hyman), pp. 45-71.
Byrant, Edward C. and Kalman Rupp. 1987. "Evaluating
the Impact of CETA on Participant Earnings", Evaluation Review,
Vol. 11, No. 4, pp. 473-492.
Campbell, Donald T. and Julian C. Stanley. 1963. "Experimental
and Quasi-Experimental Designs for Research on Teaching" in N. L.
Gage ed., Handbook of Research on Teaching (Chicago: Rand McNally).
Card, David and Philip Robins. 1996. "Do Financial
Incentives Encourage Welfare Recipients to Work? Initial 18-Month Findings
from the Self-Sufficiency Project" (Vancouver, British Columbia:
Social Research and Demonstration Corporation).
Casey, Connie H. 1992. "Characteristics of HUD-Assisted
Renters and Their Units in 1989" (Washington, D.C.: U.S. Department
of Housing and Urban Development, March).
Cave, George, Hans Bos, Fred Doolittle and Cyril Toussaint.
1993. JOBSTART: Final Report on a Program for School Dropouts. (New
York: Manpower Demonstration Research Corporation).
Chen, Huey-Tsyh. 1990. Theory-Driven Evaluations.
(Newbury Park, Calif: Sage Publications).
Chen, Huey-Tsyh and Peter H. Rossi. 1987. "The Theory-Driven
Approach to Validity". Evaluation and Program Planning 10:
pp. 95-103.
Connell, James P. and Anne C. Kubisch. 1995. "Applying
a Theories of Change Approach to the Evaluation of Comprehensive Community
Initiatives: Prospects and Problems" (Washington, D.C.: The Aspen
Institute), Draft, November.
Connell, James P., J. Lawrence Aber and Gary Walker. 1995.
"How Do Urban Communities Affect Youth? Using Social Science Research
to Inform the Design and Evaluation of Comprehensive Community Initiatives"
in Connell, James P., Anne C. Kubisch, Lisbeth B. Schorr, Carol Weiss
eds. New Approaches to Evaluating Community Initiatives: Concepts,
Methods and Contexts (Washington, D.C.: The Aspen Institute), 93-125.
Cook, Thomas D. and Donald T. Campbell. 1979. Quasi-Experimentation:
Design and Analysis Issues for Field Settings (Chicago: Rand McNally).
Corson, Walter S., Paul T. Decker, Shari M. Dunston, and
Stuart Kerachksy. 1991. Pennsylvania Reemployment Bonus Demonstration:
Final Report (Princeton, N.J.: Mathematica Policy Research).
Coulton, Claudia. 1995. "Using Community-Level Indicators
of Childrens Well-Being in a Comprehensive Community Initiative"
in Connell, James P., Anne C. Kubisch, Lisbeth B. Schorr, Carol Weiss
eds. New Approaches to Evaluating Community Initiatives: Concepts,
Methods and Contexts (Washington, D.C.: The Aspen Institute), 93-125.
Danziger, Sheldon, Robert Havemen and Robert Plotnick.
1981. "How Income Transfers Affect Work Savings and Income Distribution",
Journal of Economic Literature, 19: pp. 975-1028.
Dickinson, Katherine P., Terry R. Johnson and Richard
W. West. 1987. "An Analysis of the Sensitivity of Quasi-Experimental
Net Impact Estimates of CETA Programs" Evaluation Review,
Vol. 11, No. 4, pp. 452 - 472.
Feins, Judith. 1993. "Provide Implementation Assistance
and Evaluation for the Moving to Opportunity Demonstration: Draft Research
Design Plan" (Cambridge, MA: Abt Associates Inc., December).
Figlio, David N. 1995. "The Effect of Drinking Age
Laws and Alcohol-Related Crashes: Time-Series Evidence from Wisconsin",
Journal of Policy Analysis and Management, Vol. 14, No. 4, pp.
555-566.
Fraker, Thomas M. and Rebecca A. Maynard. 1987. "The
Adequacy of Comparison Group Designs for Evaluations of Employment-Related
Programs". Journal of Human Resources 22 (Spring): 194-227.
Freedman, Stephen and Daniel Friedlander. 1995. "The
JOBS Evaluation: Early Findings on Program Impacts in Three Sites"
(Washington, D.C.: U.S. Department of Health and Human Services and U.S.
Department of Education, September).
Friedlander, Daniel and Philip Robins. 1995. "Evaluating
Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods."
American Economic Review, Vol. 85, No. 4 (September): 923-937.
Friedlander, Daniel and Philip K. Robins. 1995. "Evaluating
Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods."
American Economic Review 85 (September): 923-37.
Goldman, Barbara. 1995. "JOBS Program Impacts for
Residents of Public Housing" (New York: Manpower Demonstration Research
Corporation, memorandum, December 6).
Gueron, Judith M. 1984. Lessons from a Job Guarantee:
The Youth Incentive Entitlement Pilot Projects (New York: Manpower
Demonstration Research Corporation, June).
Gueron, Judith M. and Edward Pauly. 1991. From Welfare
to Work (New York: Russell Sage Foundation).
Halpern, Robert. 1995. Rebuilding the Inner-City: A
History of Neighborhood Initiatives to Address Poverty in the United States
(New York: Columbia University Press: 1995) p. 233.
Hays, William L. 1973. Statistics for the Social Sciences
(New York: Holt, Rinehart and Winston, Inc.)
Hollister, Robinson G. and Jennifer Hill. 1995. "Problems
in the Evaluation of Community-Wide Initiatives" in Connell, James
P., Anne C. Kubisch, Lisbeth B. Schorr, Carol Weiss eds. New Approaches
to Evaluating Community Initiatives: Concepts, Methods and Contexts
(Washington, D.C.: The Aspen Institute), pp. 127-172.
ICF Incorporated. 1993. "Report on Emerging Resident
Management Corporations in Public Housing" (Washington, D.C.: U.S.
Department of Housing and Urban Development).
James, Franklin J. 1991. "The Evaluation of Enterprise
Zone Programs" in Roy E. Green ed. Enterprise Zones: New Directions
in Economic Development (Newbury Park, CA: Sage Publications).
Jick, Todd D. 1979. "Mixing Qualitative and Quantitative
Methods: Triangulation in Action" Administrative Science Quarterly,
Vol. 24, December, pp. 602-611.
Kennedy, Stephen D. 1988. "Direct Cash Low-Income
Housing Assistance" in Howard S. Bloom, David S. Cordray and Richard
J. Light, eds. Lessons from Selected Program and Policy Areas (San
Francisco: Jossey-Bass Inc.).
Knox, Virginia, Amy Brown and Winston Lin. 1995. "MFIP:
An Early Report on Minnesotas Approach to Welfare Reform" (New
York: Manpower Demonstration Research Corporation, November).
Kotloff, Lauren, Phoebe Roaf, and Michelle Alberti Gambone.
1995. "The Plain Talk Planning Year: Mobilizing Communities to Change
(New York: Public/Private Ventures)
Kubisch, Anne C., Carol H. Weiss, Lisbeth B. Schorr and
James P. Connell. 1995. "Introduction" in James P. Connell,
Anne C. Kubisch, Lisbeth B. Schorr and Carol Weiss, eds. New Approaches
to Evaluating Community Initiatives: Concepts, Methods and Contexts (Washington,
D.C.: The Aspen Institute), pp. 1-21.
Krueger, Richard A. 1994. Focus Groups: A Practical
Guide for Applied Research (Thousand Oaks, CA: Sage Publications).
Lalonde, Robert J. 1986. "Evaluating the Econometric
Evaluations of Training Programs with Experimental Data" American
Economic Review 76 (September): 604-20.
Leavitt, Jacqueline. 1994. "Defining Cultural Differences
in Space: Public Housing as a Microcosm" (College Park, MD: Urban
Studies and Planning Program, University of Maryland).
Lesser, Cara. 1993. "Doesnt Pay to Work: The
Financial Disincentive to Work for Residents in Public Housing" (San
Francisco: San Francisco Housing Authority).
Manpower Demonstration Research Corporation, Board of
Directors. 1980. Summary and Findings of the National Supported Work
Demonstration (Cambridge, MA: Ballinger Publishing Company).
Manpower Demonstration Research Corporation. 1981. Tenant
Management: Findings from a Three-Year Experiment in Public Housing
(Cambridge, MA: Ballinger Publishing Company).
McCain, Leslie J. and Richard McCleary. 1979. "The
Statistical Analysis of the Simple Interrupted Time-Series Quasi-Experiment"
in Thomas D. Cook and Donald T. Campbell, Quasi-Experimentation: Design
and Analysis Issues for Field Settings (Chicago: Rand McNally), pp.
233-293.
McDowall, David, Richard McCleary, Errol E. Meidinger
and Richard A. Hay, Jr. 1980. "Interrupted Time-Series Analysis",
Sage University Papers (Newbury Park, CA: Sage Publications).
Moffitt, Robert. 1992. "Incentive Effects of the
U.S. Welfare System: A Review", Journal of Economic Literature,
30: pp. 1-61.
Newman, Sandra J. and Ann B. Schnare. 1992. Beyond
Bricks and Mortar: Reexamining the Purpose and Effects of Housing Assistance
(Washington D.C.: The Urban Institute Press).
Patton, Michael Q. 1988. "The Nature of Qualitative
Data" in Patton, Qualitative Evaluation Methods (Beverly Hills,
CA: Sage Publications), pp. 21-37.
Public Housing Authorities Directors Association and Georgia
Association of Housing and Redevelopment Authorities, Inc. 1994. "Rent
Reform: The Missing Link in Upward Mobility for Public Housing Residents"
(Washington, D.C.: Public Housing Authorities Directors Association,
Winter).
Raudenbush, Stephen W. 1995. "Statistical Analysis
and Optimal Design in Cluster Randomization Studies" (East Lansing,
MI: Michigan State University, January).
Riccio, James A. and Alan Orenstein. 1996. "Understanding
Best Practices for Operating Welfare-to-Work Programs", Evaluation
Review, Vol. 20, No. 1, February, pp. 3-28.
Rohe, William M. 1993. "From Public Housing to Home
Ownership: An Evaluation of an Innovative Self-Sufficiency Program"
(presented to the Association of Collegiate Schools of Planning, October
28-31).
Rosenbaum, James E. 1995. "Changing the Geography
of Opportunity by Expanding Residential Choice: Lessons from the Gautreaux
Program", Housing Policy Debate, Vol. 6, Issue 1, pp. 231-269.
Schill, Michael H. 1993. "Distressed Public Housing:
Where Do We Go From Here?" University of Chicago Law Review,
60:2 (Spring): pp. 497-554.
Schwartz, Alex. 1987. "Dun and Bradstreets
Market Identifiers and Economic Geography". Professional Geographer,
Vol. 39, No. 2.
Shlay, Anne B. 1993. "Family Self-Sufficiency and
Housing", Housing Policy Debate, Vol. 4, Issue 3, pp. 457-495.
Shlay, Anne B. and C. Scott Holupka. 1992. "Steps
Toward Independence: Evaluating an Integrated Service Program for Public
Housing Residents", Evaluation Review, Vol. 16, No. 5, pp.
508-533.
Spence, Lewis H. 1993. "Rethinking the Social Role
of Public Housing", Housing Policy Debate, Vol. 4, Issue 3,
pp. 355-368.
Spiegelman, Robert G., Christopher J. OLeary and
Kenneth J. Kline. 1992. The Washington Reemployment Bonus Experiment:
Final Report (Kalamazoo, MI: W.E. Upjohn Institute for Employment
Research, January).
Stack, Carol B. 1974. All Our Kin (New York: Harper
and Row Publishers).
Stack, Carol B. and Linda M. Burton. 1994. "Kinscripts:
Reflections on Family, Generation and Culture" in Glenn, Chang and
Forcey, eds. Mothering: Ideology, Experience and Agency (Routledge).
Stromsdorfer, E., H. Bloom, R. Boruch, M. Borus, J. Gueron,
A. Gustman, P. Rossi, F. Scheuren, M. Smith, and F. Stafford. 1985. "Recommendations
of the Job Training Longitudinal Survey Research Advisory Panel".
Washington, D.C.: Employment and Training Administration, U.S. Department
of Labor, November.
Struyk, Raymond J. and Franklin J. James. 1975. Intrametropolitan
Industrial Location (Lexington, MA: Lexington Books).
University of Wisconsin-Extension. 1995. "Hillside
Terrace Baseline Case Study: Hope VI Program Assessment" (Milwaukee,
WI: University of Wisconsin-Extension, June).
U.S. Government Accounting Office. 1992. "Public
and Assisted Housing: Linking Housing and Supportive Service to Promote
Self-Sufficiency" (Washington, D.C.: USGAO, April).
Vidal, Avis C. 1995. "Reintegrating Disadvantaged
Communities into the Fabric of Urban Life: The Role of Community Development".
Housing Policy Debate, Vol. 6, Issue 1, pp. 169-230.
Walker, Gary and Frances Vilella-Velez. 1992. Anatomy
of a Demonstration: The Summer Training and Education Program (STEP) from
Pilot through Replication and Postprogram Impacts (Philadelphia: Public/Private
Ventures, Winter).
Weiss, Carol Hirschon. 1995. "Nothing as Practical
as Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community
Initiatives for Children and Families" in Connell, James P., Anne
C. Kubisch, Lisbeth B. Schorr, Carol Weiss eds. New Approaches to Evaluating
Community Initiatives: Concepts, Methods and Contexts (Washington,
D.C.: The Aspen Institute), pp. 65-92.
Wilson, William Julius. 1987. The Truly Disadvantaged:
The Inner City, the Underclass, and Public Policy (Chicago: University
of Chicago Press).
Yin, Robert K. 1989. Case Study Research: Design and
Methods (Newbury Park, CA: Sage Publications).
Yin, Robert K. 1993. Applications of Case Study Research
(Newbury Park, CA: Sage Publications).
Yinger, John. 1988. "Examining Racial Discrimination
with Fair Housing Audits" in Howard S. Bloom, David S. Cordray and
Richard J. Light, eds. Lessons from Selected Program and Policy Areas
(San Francisco: Jossey-Bass).
|