Abstract

This paper examines issues and options for the design of a major non-experimental study to measure the impacts of a large-scale, saturation-level demonstration program to promote employment among residents of selected public housing developments. The program, Jobs-Plus, is being launched by the Manpower Demonstration Research Corporation in cooperation with the U.S. Department of Housing and Urban Development and The Rockefeller Foundation. [An updated, full list of the Jobs-Plus funding partners is provided at the front of this paper.] Because Jobs-Plus will be a comprehensive community initiative, available to all residents of the several public housing developments where it is implemented, the program cannot be evaluated using a randomized experiment, the now-standard method for measuring the impacts of employment and training programs. However, because community-wide initiatives are becoming an increasingly important component of social policy, it is essential to develop methods for determining their success. It is the purpose of this paper, therefore, to explore possibilities for doing so.

I. Introduction
Jobs-Plus

Anticipated sea-changes in welfare policy and housing policy are likely to place an enormous financial burden on U.S. public housing. In particular, welfare reform initiatives that limit the duration of benefits received by families will reduce the ability of public housing residents, almost half of whom are welfare recipients, to pay their rents.¹ In addition, proposed measures to consolidate federal housing subsidies probably will reduce the overall level of such funding. In response to these financial threats to the survival of public housing, it is imperative that local Public Housing Authorities (PHAs) find effective ways to increase the share of public housing residents who are employed.²

The benefits of resident employment go beyond financial and economic gains. Holding a job provides social opportunities, linkages to a larger community beyond public housing, positive role models for children in the development, and personal satisfaction. Creating a mixed-income community may reinvigorate the development, change public perceptions about public housing, and revitalize the surrounding neighborhood.

The Jobs-Plus Initiative being launched by the Manpower Demonstration Research Corporation (MDRC), in partnership with the U.S. Department of Housing and Urban Development (HUD) and The Rockefeller Foundation, is intended to help policy-makers and program managers learn some important lessons about what is required to increase the employment of public housing residents. To do so, MDRC will work closely with six PHAs that will develop locally-based approaches to providing saturation-level employment opportunities for working-age residents from at least one family housing development in each PHA.

Although the design of Jobs-Plus will vary across sites, reflecting their local resources, preferences and constraints, each site will rely on a combination of three strategies:

• implementing state-of-the-art employment and training services that have proved effective for welfare recipients and public housing residents;
• developing financial incentives that promote work through state welfare reform efforts and PHA modifications to rent rules and eligibility requirements;
• building a public housing community that actively supports work through resident groups and other local organizations.

Because of the enormous challenges facing public housing at his time, Jobs-Plus cannot afford merely to modify "business as usual." Instead it must provide a bold departure from previous attempts to increase the economic self-sufficiency of public housing residents.

The "theory of change"³ which underlies Jobs-Plus, and which gives rise to the expectation that it can, indeed, produce the major changes needed, derives from the premise that the synergy created by combining the initiative’s three main strategies will be substantial.

Previous studies have shown that state-of-the-art employment and training services can markedly increase the earnings of welfare recipients, in general, and welfare recipients who are public housing residents, in particular.⁴ Especially promising are services with direct and immediate links to jobs.

Previous research also has shown that public assistance programs and housing subsidy programs contain powerful financial incentives that inhibit work effort.⁵ If a poor family earns "too much" it can lose its Medicaid health benefits. In addition, its housing subsidy might be reduced, its AFDC benefits might be reduced, and its food stamp benefits might be reduced. If local welfare agencies and local PHAs can combine their efforts and find new ways to mitigate these disincentives, at least temporally, this could help motivate families on welfare and in public housing to work harder toward economic self-sufficiency.⁶

Previous research also has shown that in some cases, public housing residents can build effective organizations and take more control over their lives.⁷ Although such organizations are difficult to create, and require substantial time and resources to sustain, they are a potentially important vehicle for building the "social capital" necessary to foster a work-oriented environment in public housing.⁸

There is no evidence, however, about the impacts of combining these strategies in a concentrated way in a specific location. Nevertheless, given the considerable demonstrated potential of each separate strategy, and the likelihood that each would reinforce the others if they were implemented together, the impacts of a joint strategy could be far greater than the sum of the impacts of the separate strategies.

Research Questions

Research on Jobs-Plus should address the following questions:

• To what extent was it possible for the study sites to implement the three Jobs-Plus strategies: high-quality employment and training services, financial incentives that reward work, and a community culture that promotes a positive work ethic? To what extent were these strategies implemented together, and to what extent were they mutually reinforcing?
• What institutions, resident groups, local civic groups, and individuals played key roles in the development of Jobs-Plus at each site? In what ways and how well did these groups and individuals work together? How did they influence the design and the evolution of Jobs-Plus? In particular, what role was played by public housing residents?
• How long did it take for each study site to develop a Jobs-Plus initiative? What resources were necessary to make this to happen? How much did these resources cost? How, and from where, were these resources obtained?
• To what extent did public housing residents participate in the Jobs-Plus initiative? What leadership roles did they play in designing, implementing, and sustaining the initiative? What services did they receive, and in what activities did they participate? What were the main barriers to their participation? How, and to what extent, were these barriers overcome? What were the main forces that helped stimulate participation in the initiative?
• In what ways, and to what extent, did Jobs-Plus affect: (1) public housing residents in the study sites; (2) public housing developments where the initiative was implemented; and (3) communities within which these developments were located? How long did it take to produce these impacts, and to what extent were they sustained over time?
• Was Jobs-Plus cost-effective: (1) from the perspective of public housing residents who were involved; (2) from the perspective of local PHAs that were involved; (3) from the perspective of taxpayers; and (4) from the perspective of society overall?
• How did the evolution of Jobs-Plus differ across sites? How did its structure differ across sites? How did its results differ across sites? To what can we attribute this variation? What can be learned from this variation about the conditions necessary for an initiative like Jobs-Plus to be successful elsewhere?

Measuring Jobs-Plus Impacts⁹

To assess Jobs-Plus properly requires measuring its impacts. By definition, a Jobs-Plus impact is:

the difference between what happened when Jobs-Plus was implemented (its outcome) and what would have happened without Jobs-Plus (its counterfactual).

Consider the following situation. What if one year after Jobs-Plus was implemented in a public housing development, 25 percent of its households had an adult who was employed? Some of these adults probably would have been employed without Jobs-Plus. Thus, to estimate what Jobs-Plus actually caused to happen — its impact — one must estimate what the employment rate would have been without the initiative. If, for example, this rate would have been 10 percent without Jobs-Plus, then the impact of the initiative would be 15 percentage points. If, on the other hand, the employment rate would have been 25 percent without Jobs-Plus, then its impact would be zero.

In the field of employment and training research, randomized experiments are now widely regarded as the best way to estimate program impacts.¹⁰ In the words of one prominent researcher, randomized experiments are "a bit like the nectar of the gods: once you’ve had a taste of the pure stuff it is hard to settle for the flawed alternatives" (Hollister and Hill, 1995, p. 134).

Randomized experiments are like a lottery. They randomly assign eligible program applicants to either a program group, which is allowed to participate, or a control group which is excluded from the program, at least temporarily.¹¹ This process ensures that the two groups are similar in all ways, except for their access to the program.¹² Hence, the subsequent experience of the control group provides a valid estimate of the counterfactual for the program group. Differences between the experiences of the two groups therefore provide valid program impact estimates.

During the past two decades, this approach has been used widely to estimate the impacts of programs for economically disadvantaged adults (for example, see Bloom et al., 1997; and Manpower Demonstration Research Corporation, Board of Directors, 1980), programs for displaced workers (for example, see Bloom, 1990, Corson et al., 1991, and Speigelman et al., 1992), programs for welfare recipients (for an extensive review of these studies, see Gueron and Pauly, 1991), programs for "at risk" youths in school (for example, see Walker and Vilella-Velez, 1992), and programs for young school dropouts (for example, see Cave, Bos, Doolittle, and Toussaint, 1993).

Likewise, housing policy debates have relied extensively on findings from the two major randomized housing experiments conducted to date, the Experimental Housing Allowance Program (EHAP) and the Freestanding Housing Voucher Demonstration Program.¹³ In addition, HUD recently commissioned a large-scale randomized experiment, the Moving to Opportunity Demonstration, to study the effect of helping poor families move from inner-city ghettoes to moderate-income neighborhoods.¹⁴

Unfortunately (from a methodological perspective), Jobs-Plus is not a program to which individuals or households can be assigned randomly. Instead, it is a "comprehensive community initiative" to which all residents of the public housing developments involved will be "exposed," and in which all residents who desire can participate. Hence, it will not be possible to measure Jobs-Plus impacts by randomly assigning individuals.

It might be possible, however, to randomly select a program group and a control group of public housing developments at each site from among those willing and able to participate. This approach cannot provide the methodological protection available from standard large-scale randomized experiments, however, because of the small number of housing developments involved.¹⁵ Nevertheless, because Jobs-Plus impacts are expected to be large, the approach might provide enough statistical power to identify them. Furthermore, because these impacts are expected to be large, quasi-experimental methods might be able to provide plausible causal inferences about them (discussed later).¹⁶

Jobs-Plus is intended for a broad range of stakeholders who have very different priorities for the types of impacts that might be produced. To provide a coherent structure for reporting these impacts, we recommend examining them from three perspectives:

• impacts on public housing residents,
• impacts on public housing developments, and
• impacts on the surrounding community.

The first perspective represents the experience of persons who live in a public housing development where Jobs-Plus is implemented, regardless of whether or not they stay there. From this perspective, researchers will study the same people over time. The second perspective represents the experience of public housing developments where Jobs-Plus is implemented, recognizing that residents will move into and out of the developments. From this perspective, researchers will study the same developments over time. The third perspective represents the experience of a larger residential and business community, the composition of which will change over time. From this perspective, researchers will study the same communities over time.

Replicability of the Research Findings

As mentioned earlier, because Jobs-Plus will represent a major departure from previous attempts to increase the economic self-sufficiency of public housing residents, very little is known about this approach. Hence, to give Jobs-Plus a fair test at this early stage in its development will require selecting sites that are committed to making the concept work and judged likely to be able to do so. Thus, the goal of the initiative probably will be to measure what Jobs-Plus can achieve under conditions that are more favorable than average. Nevertheless, it would be unwise to choose sites for Jobs-Plus that are so unusual that their findings are not replicable. Therefore, a careful balance will be maintained between the expected commitment and capability of the sites selected for the demonstration and the likelihood that other sites, but not necessarily all sites, could replicate their experience in the future.

Developing New Methods to Assess Comprehensive Community Initiatives

• A horizontally complex intervention. Jobs-Plus will integrate the efforts of numerous social service systems (especially welfare and housing agencies) plus the activities of many civic, business, and community groups. Therefore, it will have many facets, whose separate impacts will be difficult to distinguish.
• A flexible and evolving intervention. Jobs-Plus sites will have wide latitude to mold the initiative. Furthermore, each site’s vision for the project will change over time. Hence, Jobs-Plus projects probably will differ from their initial plans, and probably will take on different forms at different times. This makes it especially important, and difficult, to document precisely what the intervention was at each site, and how it changed over time.
• A broad range of outcomes to examine. Jobs-Plus is intended to produce changes in many different dimensions (economic, psychological, social, political, and physical) and changes that will occur over widely varying time-frames (immediate, intermediate, and long-run). Hence, the number of outcome measures to consider is substantial, and the task of summarizing them will be difficult.
• Vertically complex impacts. Jobs-Plus is intended to produce impacts that are experienced at different levels and that therefore should be examined from different perspectives. As indicated above, Jobs-Plus is expected to produce impacts on public housing residents, on public housing developments, and on the communities surrounding these developments.
• Important contextual issues. Although Jobs-Plus will be implemented in local communities, its evolution, its final form, and its results will be influenced by many forces which are outside of its control. For example, federal regulations (and possible waivers) which govern welfare and housing programs will determine the range of available programmatic options. Economic conditions in the community, in the region, and nationally will influence the job opportunities through which economic self-sufficiency can be achieved. Barriers due to racial and ethnic discrimination will limit the success and influence the direction of the initiative. Therefore, to understand why certain impacts did or did not occur requires a detailed knowledge of the environment within which Jobs-Plus operated.
• Absence of an adequate experimental control group. For reasons outlined above, it will not be possible to randomly assign individuals to Jobs-Plus or a control group. Thus, it will not be possible to estimate Jobs-Plus impacts using the approach now relied on by evaluations of most employment and training programs. Nevertheless, it will be crucial to determine what, in fact, Jobs-Plus caused to happen.

The preceding problems, inherent in evaluations of comprehensive community initiatives, have been the focus of current work by the Roundtable on Comprehensive Community Initiatives for Children and Families, sponsored by the Aspen Institute. This group has: (1) identified and clarified the many serious problems experienced by such evaluations (Hollister and Hill, 1995); (2) made a forceful case for the need to find ways to overcome these problems (Kubisch, Weiss, Schorr, and Connell, 1995, and Brown, 1995), and (3) begun to sketch an alternative evaluation framework, based on "theories of change" (Weiss, 1995, and Connell and Kubisch, 1995).

This work is in its very early stages, however, and no new evaluation methods or models have been developed or tested. Therefore, research designs for Jobs-Plus will have to rely on combining existing methodologies in creative ways and developing new approaches.

This Paper

This present paper focuses on estimating the impacts of Jobs-Plus from the three perspectives described above. Section 2 discusses impacts from the perspective of the residents of Jobs-Plus developments. Section 3 discusses impacts from the perspective of the developments themselves. Lastly, Section 4 discusses impacts from the perspective of the community surrounding a Jobs-Plus development. All plans for the Jobs-Plus evaluation are tentative at this time. The final design will evolve during the pilot phase of the project.

II. Jobs-Plus Impacts on Public Housing Residents

Introduction

This section explores ways to measure the impacts of Jobs-Plus on residents of public housing developments. To measure these impacts requires observing the same people over time, regardless of whether they stay in the development where Jobs-Plus was implemented or move away.

The section begins with a simple model of the process by which Jobs-Plus is expected to produce its intended effects. This model provides a conceptual framework for defining program outcome measures. Three research designs for measuring Jobs-Plus impacts are then considered, followed by a discussion of sample size and minimum detectable effects. The section concludes with a discussion of data issues.

A Model of Jobs-Plus Impacts and Measures of Jobs-Plus Outcomes

Exhibit 2.1 illustrates a four-stage process which describes how Jobs-Plus is expected to improve the lives of public housing residents. The process starts with the Jobs-Plus program and baseline characteristics of the residents, their public housing development, and its surrounding community. The three main components of Jobs-Plus — program services, financial incentives, and community support — will provide the impetus for improvements in the lives of public housing residents. The baseline characteristics of these residents, their development, and its surrounding community will influence their ability to benefit from Jobs-Plus.

For example, personal characteristics (such as residents’ education levels, labor market experiences, dependence on public assistance, and attitudes toward life, work, and welfare) will play a large role in determining their value to potential employers. Likewise, characteristics of public housing developments (such as their perceived safety, the presence of working adults, the availability of child-care services, the existence of supportive organizations, and their proximity to job opportunities) can affect residents’ willingness and ability to seek employment. Similarly, characteristics of the community surrounding a public housing development (such as its perceived safety, the existence of nearby jobs, access to public transportation, and the presence of retail facilities) can facilitate or impede the labor market success of public housing residents. This constellation of baseline characteristics defines the setting in which Jobs-Plus will take place and will be documented carefully.

The second stage in the process involves some initial Jobs-Plus outcomes that must occur in order for the initiative to influence the decisions and behavior of public housing residents. First, residents must understand how Jobs-Plus changes their incentives to work, through modifications of welfare eligibility requirements and benefit determination, and through modifications of public housing eligibility requirements and rent calculations. If residents understand these provisions, they can respond to them accordingly; if they don’t understand them, they cannot respond systematically. In addition, public housing residents will need a wide variety of services to help them overcome their many serious barriers to employment. If they receive these services, they should be better able to compete effectively for jobs. Without these services, they will not be as competitive. Finally, Jobs-Plus will help tenant organizations and other neighborhood groups to create a personally-supportive, work-focused environment. If such a climate is created, and if public housing residents perceive it to exist, then it could provide an important boost to their self-improvement efforts.

These initial outcomes are anticipated prerequisites for the success of Jobs-Plus and will be examined as part of the implementation research for the project. Hence, they will provide an early indication of whether or not Jobs-Plus is likely to be successful.

The third stage in the process involves changes produced by Jobs-Plus in the attitudes, the human capital (knowledge, skills, and credentials), and the behavior of public housing residents, plus changes in the extent and nature of their interactions and their level of organization. It is directly through these changes that Jobs-Plus is expected to produce its intended effects on economic self-sufficiency. For example, to the extent that exposure to Jobs-Plus increases the self-confidence of public housing residents about finding and holding a job, they are likely to try harder to do so. To the extent that Jobs-Plus increases the knowledge and skills of public housing residents, and helps them obtain educational and vocational credentials, it will increase their potential productivity, and thereby increase their value to employers. To the extent that Jobs-Plus increases the amount of time that public housing residents spend in productive activities (such as study, job-search, volunteer work, or helping children) and to the extent that Jobs-Plus decreases the incidence of negative behaviors (such as alcohol abuse, drug abuse, and criminal activities), it will help to foster life skills and behavior patterns that support, instead of impede, gainful employment. Likewise, to the extent that public housing residents interact more frequently with each other, and develop more cohesive organizations, they are more likely to reinforce each other’s attempts to become and remain employed.

Only if these changes occur, can Jobs-Plus improve the lives of public housing residents. Hence, measuring these intermediate impacts provides an interim test of whether Jobs-Plus is working as it was intended.

The fourth and final stage in the process involves actual measures of the progress made by public housing residents toward economic self-sufficiency. There is disagreement, however, about what constitutes economic self-sufficiency.¹⁸ At a minimum, it requires gainful employment. In addition, most people would agree that it involves a lack of dependence on AFDC and food stamps. Agreement is less uniform, however, about whether or not persons who receive housing subsidies (which have higher income eligibility levels than AFDC) should be considered self-sufficient. It is even less clear whether or not persons who receive subsidized health care, through Medicaid, should be considered self-sufficient. Lastly, there are many families that have a full-time worker and receive no form of public assistance, but nevertheless remain in poverty. Should these families be considered self-sufficient? Given the complexity of the concept of self-sufficiency, and the widely varying views of what the concept means, the impact of Jobs-Plus should be measured in ways that reflect these different views.

Exhibit 2.2 lists outcome measures that reflect the preceding model, and thereby could serve as the basis for assessing the initial, intermediate, and long-term impacts of Jobs-Plus.

Impact Study Design #1: A Two-Group, Before-After Analysis

The Approach

• the program outcome is represented by the change in an outcome measure from before the program was implemented (usually at baseline) to after it was implemented,
• the counterfactual is represented by the corresponding change for the comparison group,
• the impact estimate is the difference between the observed change for the program group and the observed change for the comparison group.

In theory, this approach could be used to estimate impacts with regard to any of the outcome measures listed in Exhibit 2.2. To the extent that the observed change in the measure for the comparison group accurately reflects what the change for the program group would have been without the program, the approach provides a valid estimate of the impact for the sample.

The key to making this approach work is finding the right comparison group. Shlay and Holupka (1992), in their study of the Family Development Center at the Lafayette Courts public housing development in Baltimore, Maryland, chose their principle comparison group from families at a nearby public housing development, Murphy Homes.²⁰ Rohe (1993), in his study of the Gateway Transitional Families Program, in Charlotte, North Carolina, chose his comparison group from persons who applied to the program but did not participate.

To increase the validity of findings from a two-group, before-after design, it is sometimes possible to match comparison group members to program group members in terms of measured baseline characteristics. There are two basic versions of this approach:

• cell matching, whereby: (1) subgroups (cells) are defined according to combinations of baseline characteristics (e.g., age category, gender, race, and welfare status), (2) each program group member is placed in the cell to which he or she belongs, and (3) comparison group members are chosen to match the number of program group members in each cell;²¹
• distance function matching, whereby: (1) a weighted function of baseline characteristics is computed for each program group member and for each potential member of the comparison group; and (2) a comparison group member is chosen to match each program group member in a way that minimizes the difference (or "distance") between their values of the weighted function.²²

In addition to statistical matching procedures, researchers usually estimate program impacts from a multiple regression equation, of the following form, which attempts to control for differences between program group and comparison group baseline characteristics which are included in the model

where:

Y₂ = the follow-up value of the outcome measure,

Y₁ = the baseline value of the outcome measure,

T = one for program group members, and zero for comparison group members,

Xj = a baseline characteristic,

a = an intercept term,

B ₀ = the program impact,

B ₁ and B _j = regression coefficients, and

e = a random error term.

Strengths and Weaknesses

The principal strengths of the approach are its relative simplicity, and the fact that it could be implemented readily for Jobs-Plus. The main weaknesses of the approach are several important "threats" to the internal validity of its program impact estimates. These threats represent specific reasons why the change in the comparison group outcome might not accurately reflect what the change would have been for the program group without Jobs-Plus.

First, is the possibility that the comparison group and the program group will experience different events that affect their willingness and ability to work.²³ Hence, it is probably advisable to choose comparison group members who are from the same local labor market and are served by the same welfare agency and public housing authority.

Several versions of this problem are likely to occur. For example, while the study is underway, other government programs probably will change in ways that affect the Jobs-Plus target population. In particular, welfare reform measures are likely to be taken by some, if not all, of the jurisdictions where Jobs-Plus is implemented. If the comparison group experiences local welfare reform and the program group experiences Jobs-Plus, then impact estimates will represent the differential effect of Jobs-Plus versus local welfare reform. If welfare reform in the Jobs-Plus PHAs is sufficiently dramatic, its effects could over-shadow those of Jobs-Plus, and thereby make it very difficult to measure Jobs-Plus impacts.

A second potential problem of this type is represented by the fact that HUD will be conducting another major demonstration project in many of the PHAs where Jobs-Plus might take place. This new project, the Moving to Work Demonstration, will test increases in the flexibility of federal regulations that control PHA operations. The increased flexibility might enable PHAs to modify their eligibility standards for public housing and their rent determination procedures in ways that also will stimulate work effort. Furthermore, changes in federal regulations might enable PHAs to better coordinate services and activities with other organizations. In short, the Moving to Work Demonstration could provide a competing "treatment" to Jobs-Plus. If it is not possible to withhold this treatment from enough public housing developments to maintain an untreated comparison group, then impact estimates obtained by comparing Jobs-Plus sites and comparison sites will represent the "differential" impacts of the two programs.

A second common problem with two-group, before/after designs is the fact that the program group and the comparison group are often on two different long-term trends before they were chosen for the study.²⁴ Hence, even if the study had not been conducted, subsequent changes observed for the two groups would be different. If, for example, tenants at one public housing development were more likely to find jobs than were the tenants at another development, the subsequent changes in their employment rates would be different in the absence of Jobs-Plus. With only one baseline outcome measure for each group, there is no way to identify their underlying trends. Hence, there is no effective way to compensate for them.²⁵

A third common problem with two-group, before/after designs also arises from the fact that only one baseline outcome measure exists. This problem results not from the existence of underlying trends, but from a temporary random departure from this trend that occurs at baseline.

For example, previous studies of employment and training programs have identified a "pre-program dip" whereby average earnings in the year before participants enter a program are well below their preceding trend.²⁶ This phenomenon has been the subject of extensive debate, and there is still no agreement about the extent to which it represents a temporary aberration (due to illness, a business failure, or other "bad luck"), or the extent to which it represents the onset of a permanent decline in earnings (which happens to economically displaced workers).

Individuals are probably more likely to enroll in a job-training program when they have just experienced an unusually bad period than when they have just experienced an unusually good period. In other words, their motivation to enroll is probably negatively correlated with random fluctuations in their earnings prospects. If so, then their average earnings in the next period are likely to increase regardless of whether or not they enroll in a program. This phenomenon is a statistical artifact called regression to the mean.²⁷

With two groups, the problem becomes one of different changes in earnings due to different regression artifacts. Hence, the observed change for the comparison group will not accurately represent what this change would have been for the treatment group, without the treatment.

The second and third problems mentioned above represent two different reasons why the program group and the comparison group differ initially in ways that would cause them to experience different outcomes, even in the absence of Jobs-Plus. Hence, they represent two different forms of "selection bias."

Impact Study Design #2: Longitudinal Data

The Approach

The second approach to estimating the impacts of Jobs-Plus is a logical extension of the first, which is intended to deal with: (1) program groups and comparison groups that have different underlying outcome trends, and (2) program groups and comparison groups which have different regression artifacts. To address these issues requires data for both groups on outcome measures for a number of years (four to five) before Jobs-Plus begins, and as long as needed to identify impacts which materialize thereafter. The longer it takes for impacts to materialize, the longer the followup period that is required. This type of information is generally referred to as longitudinal data or panel data.

From these data, one can compute a "pre-program" trend for each program group member and each comparison group member. It is then possible to compute a separate post-program deviation from each person’s pre-program trend.²⁸ The difference between the mean deviation from trend for the Jobs-Plus group and the mean deviation from trend for the comparison group provides an estimate of the impact of Jobs-Plus. This estimate represents: the extent to which Jobs-Plus caused participants to change their long-term labor market behavior beyond the change that would have occurred without Jobs-Plus.

Because past behavior, especially past behavior for a number of years, is usually the best predictor of future behavior, comparing sample members’ post-program outcomes to their pre-program trend (thereby allowing each sample member to serve as his or her comparison group) is probably the best way, absent a randomized experiment, to control for underlying differences in their likely future behavior.²⁹

Giving individuals their own trend implies estimating a separate intercept and a separate slope for each. This is a straightforward extension of standard "fixed-effect" models for longitudinal data, which specify a separate intercept for each sample member.³⁰ It is possible to include additional explanatory variables in such models, but this generally does not increase their explanatory power appreciably because the additional variables are usually reflected in the past outcome trends for each sample member.³¹

Strengths and Weaknesses

The principal strength of this approach is its ability to control for differences in the underlying trends of the program group and the comparison group. In addition, because the approach provides an explicit description of the behavior of sample members over time, it can identify aberrations that might have occurred before an intervention took place. If no such aberration exists, then regression artifacts should not be a problem. If, on the other hand, an aberration appears in the data for either the program group or the comparison group, or both, then the potential for differential regression artifacts exists. Nevertheless, impact estimates based on longitudinal data are less susceptible to the influence of regression artifacts than are simple before/after comparisons.

One potential weakness of longitudinal impact estimators is the possibility that the program group and the comparison group are subject to different major local events (history). If this occurs, then differences in the changes observed for the two groups might reflect differences in the events to which they were exposed, not the impact of Jobs-Plus. Only by choosing program and comparison groups that are likely to be exposed to the same events, and carefully documenting the events to which they were exposed, can one properly interpret the impact estimates obtained.

A second potential weakness of longitudinal impact estimators is the possibility that past behavior will not adequately reflect potential future for program groups members because they have reached a point in their lives where they will make a major change.³² To see this point, consider the following situation.

Enrolling in a job-training program might signal a desire to change one’s situation in life. Even if someone has been out of the labor force for a long time, or has been in and out of low-pay jobs for many years, he or she might be ready, able, and willing to make a change with or without participating in a program. If so, then data over time on prior employment, earnings, welfare receipt, or housing subsidies will not provide an adequate statistical control for likely future changes in these outcomes. Indeed, nothing short of a randomized experiment will do so.³³

Fortunately, because of a fundamental difference between the participant selection process for employment and training programs, and that for Jobs-Plus, longitudinal impact methods (even two-group, before-after methods) might be more suitable for evaluating Jobs-Plus than for evaluating employment and training programs.

Individuals will become members of the Jobs-Plus program group if they happen to live in a public housing development where Jobs-Plus is implemented. They will not become members of the program group if they do not live in a Jobs-Plus development. Hence, at least when the initiative begins, Jobs-Plus "comes to the individuals," the individuals do not "come to Jobs-Plus."³⁴ Therefore, the program group should not exhibit a pre-program dip. Consequently, it should not exhibit a regression artifact.

Furthermore, there is no reason to expect that the proportion of program group members who are planning to change their lives differs appreciably from the proportion of residents at other similar developments who plan to do so.

Impact Study Design #3: Random Assignment of
Public Housing Developments³⁵

A third approach for estimating the impacts of Jobs-Plus, which builds on the preceding two approaches, is a randomized experiment, in which: (1) several public housing developments are chosen as candidate Jobs-Plus sites from each PHA that participates, (2) these developments are matched as closely as possible, and (3) a lottery is used to determine which development implements Jobs-Plus and which become members of a control group.

If this approach is feasible, it could be used in conjunction with either or both of the preceding approaches. By randomly deciding which candidate developments become Jobs-Plus sites and which become control sites, one can eliminate systematic forces that might produce initial differences between the two groups. Only by chance would such differences occur. Hence, this procedure would eliminates bias in program impact estimates. Nevertheless, a substantial margin for random sampling error would remain, because residents would be randomly assigned by development, not independently as individuals.³⁶

Note that this random sampling error will be present whether public housing developments are chosen randomly for the program and control groups or whether they are chosen in some other way (which would not eliminate bias). The problem with respect to random sampling error arises because Jobs-Plus applies to whole public housing developments, not to specific residents.

Sample Size and Minimum Detectable Effects³⁷

Sample size is key to the Jobs-Plus impact analysis. To place this issue in perspective, first consider the magnitude of impacts produced by successful welfare-to-work programs that have some of the planned features of Jobs-Plus (Exhibit 2.3).

Column one in the exhibit reports some of the first available findings from demonstration programs being conducted in Atlanta, Georgia; Grand Rapids, Michigan; and Riverside, California, under the Job Opportunities and Basic Skills (JOBS) Program. These findings reflect the impacts on employment rates of a "Labor Force Attachment" approach to stimulating employment, through job-search assistance, followed by work experience or short-term education and training activities, if needed. The second column presents unpublished findings from the same study for a subsample of public housing residents in Atlanta. The third column reports recently published findings from a major Canadian study of an earnings supplement designed to "make work pay" by guaranteeing (for up to three years) a substantial minimum earnings to welfare recipients who find a full-time job within a year and remain employed full-time thereafter.

In each case, the employment rate for program group members observed eighteen months to two years after enrollment was 8 to 9 percentage points higher than it would have been without the program. These findings were statistically significant.

Furthermore, these findings represent substantial impacts on employment. To see this, it is useful to express each impact as a percentage of the corresponding employment rate for the control group. Hence, the 8.1 percentage point impact in the first column represents a 23.5 percent increase from the 34.4 percentage point employment rate for control group members. In other words, 23.5 percent more of the program group members were employed than would have occurred without the program. Corresponding impacts in these terms are a 28.0 percent increase in employment for column two and a 28.4 percent increase in employment for column three.

The impacts of Jobs-Plus should exceed those in Exhibit 2.3, because it will combine the separate approaches used by the programs represented in the exhibit, and add a third important component to the mix. As discussed earlier, Jobs-Plus will include state-of-the-art employment and training services; financial and other incentives that promote work; and vigorous efforts to build a community that supports work. Hence, Jobs-Plus should increase employment rates by at least 10 percentage points. Therefore, the impact study design for Jobs-Plus must have enough statistical power to detect impacts of roughly this size.

Findings in the exhibit were based on total samples that ranged from just under 800 persons to just over 1,900 persons, which is typical of many recent experiments, but is considerably smaller than the largest ones.³⁸ These samples could detect impacts on employment rates of 8 to 9 percentage points. Thus, if the Jobs-Plus sample were selected in a similar way (by randomly assigning individuals to program or control status), 1,000 to 2,000 public housing residents would be adequate to detect the types of impacts anticipated if Jobs-Plus is successful.

However, Jobs-Plus sample members will enter the study in groups, or "clusters," either as residents in public housing developments selected for the Jobs-Plus program, or as residents in public housing developments selected for the comparison group. This "cluster assignment" will increase the standard errors of Jobs-Plus impact estimates, thereby reducing their statistical power. This loss of power could be substantial, because the clusters (developments) will be large (containing 200 to 400 families each).³⁹ Indeed, the standard errors for impact estimates from a Jobs-Plus sample might be several times as large as those from a sample of the same size produced by random assignment of individual public housing residents.⁴⁰

Without information about the "intra-class correlation" for the outcome measures to be used, it is not possible to determine whether any given number of public housing residents will be adequate for the study. Nevertheless, to provide some insight about the likely statistical power of Jobs-Plus impact estimates, Exhibit 2.4 presents a hypothetical example.

The exhibit lists employment rates for five hypothetical housing developments in two consecutive years, without Jobs-Plus. The size of these developments is not specified, but I assume that they are relatively large, and focus on aggregate employment rates.⁴¹ The magnitudes of these employment rates are similar to those observed by the studies represented in Exhibit 2.3.

What is more important, however, is the variation in the changes in employment rates. These changes range from an increase of 10 percentage points to a decrease of 10 percentage points (which is substantial). Their standard deviation is 8 percentage points.

If this situation approximates the actual variability of changes in employment rates for public housing developments like those in the Jobs-Plus sample, then one can approximate the minimum detectable effects of Jobs-Plus using the public housing development as the unit of analysis.

For example, what if one Jobs-Plus development and one comparison development are chosen from each of five PHAs? Estimates of impacts on changes in employment rates obtained from this sample of 10 developments would have a standard error of 5 percentage points.⁴² This means that an impact estimate greater than roughly 10 percentage points would be statistically significant at the .05 level.⁴³

If instead, we chose one Jobs-Plus development for each PHA and two developments for the comparison group, to increase statistical power, the standard error of the impact estimate would be 4.4 percentage points. Thus, an impact estimate greater than about 9 percentage points would be statistically significant at the .05 level.

Given the preceding assumptions, it therefore should be possible to detect impacts on employment rates that are comparable to those produced by state-of-the-art employment and training programs and targeted financial incentives.

I next briefly discuss some important data issues and options.

• Whom should we observe?
• What types of outcomes should we measure?
• For what period should we measure these outcomes?
• Where can we get information about these outcomes?

To measure the impacts of Jobs-Plus on public housing residents will require defining the target population as all persons living in a Jobs-Plus development when the initiative begins. It also probably will require drawing a sample at the same time from a comparison development for each site. During the followup period, it will be necessary to track these individuals wherever they go.

Earlier I presented a model of Jobs-Plus impacts which implies a series of initial, intermediate, and long-term outcomes. Exhibit 2.2 lists these measures, which are not repeated here. Instead, I consider aspects of these measures which determine the types of data collection methods they will require. In addition, I briefly indicate how different outcome measures will provide different types of information about changes over time and, consequently, will require different statistical analysis methods.

First consider measures of differences in levels or differences in states at two points in time, say, for example, at baseline and at the end of followup. This focus applies to outcomes such as residents’ attitudes about themselves, about welfare, and about work, their understanding of the incentives of Jobs-Plus, and whether or not they are employed, on welfare, or still living in public housing.

The emphasis here is on describing how the outcome differed at two points in time, not on identifying when it might have changed, or on describing fully its pattern of change over time (if it changed more than once). Hence, one can measure such outcomes at two specific times and summarize them as mean changes — a simple before-after analysis.

True longitudinal data are different. They provide continuous information over time about a level, a flow, or a state, and thereby make it possible to summarize a pattern over time, which describes both how an outcome changed and when it changed.

To analyze longitudinal data for continuous outcomes, such as earnings or welfare benefits, one can use conventional fixed-effect models for panel data, or extensions of these models, such as time-varying, fixed-effect models (Bloom, 1984).⁴⁴

To analyze such data for discrete outcomes, such as employed or not, on welfare or not, or living in public housing or not, requires statistical methods for "event history" analysis, such as "hazard rate" models (Allison, 1988).

Two questions arise here: (1) how long should the followup period be? and (2) how long should the baseline period be? Different answers to these questions will be necessary for different outcome measures obtained from different data sources with different time limitations.

The overall followup period for the project probably should be at least five years, because of the time that it will take to implement Jobs-Plus fully, and because of the likely long time required for the initiative to increase the economic self-sufficiency of public housing residents. For example, Rohe (1993) describes a seven-year process for participants in the Gateway Transitional Families Program sponsored by the Charlotte, North Carolina Housing Authority.⁴⁵

We also recommend that the baseline period for some of the study’s key outcome measures include four to five years before sample members are exposed to Jobs-Plus. Shorter baseline periods would enable researchers to control for differences in the initial conditions of sample members, but longer periods are required to control for differences in their underlying trends.

Where to Get the Data?

• sample surveys,
• PHA administrative records,
• UI wage records, and
• AFDC and food stamps records.

Sample surveys could be conducted at baseline and at one or more times thereafter. The first survey could measure the baseline characteristics listed earlier. It also could obtain retrospective longitudinal information about employment, earnings, welfare receipt, subsidized housing, and other key outcomes during the previous two-to-five years. The time period for this information would be limited by the ability of survey respondents to recall past events.⁴⁶

PHA administrative records, especially those based on the HUD Form 50058, which is used to certify a family’s eligibility for public housing when it applies, and is used each year thereafter to recertify eligibility and set family rents, provide information about selected socioeconomic characteristics. If these forms are completed properly, and if they are accessible, they could provide an annual "snapshot" of a household’s composition, its income by source, and its net income (minus allowances and expenses). Examination of the quality and availability of these data for sites that are selected to participate in Jobs-Plus will be necessary to determine whether or not they should be used.

As a source of longitudinal baseline data, the 50058 Form is limited by the period for which it is available, and the fact that it covers different periods for different residents.⁴⁷ Nevertheless, it might provide a good description of each household in the study at the time Jobs-Plus begins. As a source of longitudinal followup data, the 50058 Form is limited by the fact that it is not collected after a family leaves a PHA’s housing programs.⁴⁸

UI wage records contain data on earnings that are reported quarterly to each state by employers, as required by law, for all workers covered by Unemployment Insurance.⁴⁹ This information is reported for well over 90 percent of all employees in most states. It is used to determine eligibility for Unemployment Insurance and weekly benefit rates for persons who file a UI claim.

By matching sample members’ Social Security numbers to their UI wage records, one can measure their total quarterly earnings and their quarterly employment rates. This approach has been used by many studies of employment and training programs. As a source of longitudinal followup data, UI wage records are excellent, because they can be obtained for an indefinite period. Furthermore, they have been found to produce program impact estimates that are similar to those based on more costly followup survey data (Bloom et al., 1993).⁵⁰ As a source of pre-program longitudinal data, this information might be limited to four to six quarters, which is all that is usually kept in active records by state UI agencies because they only maintain active data for the period used to establish UI eligibility and weekly benefit rates. It might be possible, however, to increase the coverage of this information substantially, if state agencies agree to provide information from their archives.

AFDC and food stamps records can be obtained by matching the Social Security numbers of sample members to the administrative records of local and state welfare agencies. This information also has been used by past evaluations of employment and training programs. When available, it represents an effective way to produce an accurate long-term, monthly history.⁵¹

III. Jobs-Plus Impacts on Public Housing Developments

Introduction

This section explores ways to measure the impacts of Jobs-Plus on public housing developments. The goal here is to study what happens to a particular residential environment because of the initiative. I first present a model of how Jobs-Plus might affect a public housing development and corresponding measures of its success. I then consider research designs for measuring the impacts of Jobs-Plus.

Exhibit 3.1 presents a model of how Jobs-Plus might affect a public housing development. Just as the model for individual residents was an oversimplification, so is the model for public housing developments. Nevertheless, it provides a conceptual basis for specifying program outcome measures and a logical structure for organizing research findings.

The structure of the model is as follows. Jobs-Plus is designed to directly influence the characteristics of public housing residents and the prevailing culture of their development. Changes in resident characteristics, in turn, produce further changes in local culture, and vice versa. These changes increase the success of the development as a place to live and as a financial enterprise.

Changes in the resident characteristics of a public housing development can occur both because individuals change over time and because different households move out of and into the development. These changes are related to residents’ attitudes, human capital, labor market success, welfare dependence, mobility, and level of organization and interaction.

As residents begin to change, improvements in the local culture become possible. If Jobs-Plus is successful, we will begin to see the evolution of a safe, personally-supportive and work-oriented environment. Factors to consider in this regard, include the absence of crime, drug, and gang activities, and the presence of organizations that promote social capital, such as resident groups and civic organizations.⁵²

The final stage of the model represents the overall success of the public housing development, as a place to live, and as a financial enterprise. To the extent that the lives of individual residents are improved, and the prevailing culture becomes more positive, one should expect to see improved physical conditions, more favorable perceptions of the safety and desirability of the development, and improved financial conditions.

Exhibit 3.2 lists some measures of a public housing development’s overall success. Measures of changes in resident characteristics and changes in local culture were presented in Exhibit 2.2.

To estimate impacts from this design requires comparing the mean change in an outcome variable for Jobs-Plus developments to the mean change for comparison or control developments.⁵³ For example, to estimate impacts on vacancy rates, one would compute the mean change in vacancy rates for a Jobs-Plus sample, repeat this procedure for a comparison sample, and take the difference. A standard t test can assess the statistical significance of this difference.

The primary strengths of this approach are its simplicity and feasibility. Its main weaknesses are:

• the Jobs-Plus developments and the comparison developments might be exposed to different events which affect their outcomes differently (they might experience different histories),
• the margin for random sampling error will be high (the statistical power of program impact estimates will be low) because of the small number of developments in the sample,
• Jobs-Plus developments and comparison developments might have a different average underlying trend (they might be maturing differently),
• random factors might affect the baseline conditions of the two groups differently (they might exhibit different regression artifacts).

To address the first problem requires selecting Jobs-Plus developments and comparison developments that are as likely as possible to experience similar events in the near future. As discussed in Section 2, they probably should be selected from the same PHAs, and one should make an attempt to match their past histories, resident characteristics, and political situations. Furthermore, it will be crucial to document events which transpire in each development to determine whether different events affected their future outcomes in important ways.

Aside from choosing more project sites or choosing more developments from each participating site, there is little that can be done to increase the statistical power of impact estimates for Jobs-Plus. Nevertheless, if these impacts are as large as those documented by past rigorous studies of key Jobs-Plus components, impact estimates for Jobs-Plus could be statistically significant.⁵⁴

The best way to address the last two problems listed above is to use longitudinal data. Designs of this type, when applied to aggregate units, such as public housing developments, are referred to in the evaluation research literature as "interrupted time-series" analyses.⁵⁵

The Approach

The simplest interrupted time-series analysis involves a single site with multiple years of data before an intervention and multiple years of data during its followup period. This analysis can be applied to outcome measures for public housing, such as vacancy rates, the percentage of households on welfare, total rents paid, the percentage of households with a working adult, and so on. Exhibit 3.3 illustrates how to use the analysis to estimate the impact of Jobs-Plus on the percentage of households in a development who are on welfare.

If four or five years of baseline data on this measure can be obtained, then one can fit a pre-Jobs-Plus trend line.⁵⁶ A linear trend probably will be adequate for most outcomes, but a curvilinear trend (discussed later) can be estimated if the curvature of the baseline trend is pronounced. Extrapolation (extension) of the baseline trend provides the best available estimate of what the outcome would have been without Jobs-Plus (the counterfactual).⁵⁷

The deviation from the baseline trend in the first year after Jobs-Plus begins (line D₁ in the exhibit) provides an estimate of the impact of Jobs-Plus for that year. Deviations from trend in subsequent years (lines D₂ and D₃ ) provide impact estimates for these years. Estimating the impact for each followup year provides an easy way to describe the pattern of the impact over time (whether it is constant, it decays, or it grows). As a way to summarize this information, one can then average these estimates to determine the mean annual impact during the followup period.⁵⁸

For example, the hypothetical results in the bottom panel of Exhibit 3.3 indicate that there was a 3 point reduction in the percentage of residents who were on welfare in the first year after Jobs-Plus began, an 11 point reduction in the second year, and a 10 point reduction in the third year. The outcome was about 9 percentage points lower than predicted, on average, during the followup period.

The following regression model can be used to estimate the impacts in the exhibit:

where:

Y_t = the value of the outcome variable in year t,

T_t = one if year t is after Jobs-Plus began and zero otherwise,

t_t = the value for year t of the annual counter,

B ₀ = the impact of Jobs-Plus (the deviation from trend),

B ₁ = the slope of the baseline trend,

a = the intercept of the baseline trend.

If only years zero through six are included in the analysis, the coefficient, B ₀, equals the deviation from trend in year six (line D₁ in the exhibit) and the t statistic for this coefficient provides a test of its statistical significance. To include all three followup years in the analysis, and allow each to have a separate deviation from trend, one can replace T with a separate dummy variable for each year. The coefficient for each dummy variable equals the deviation from trend for the year that it represents (lines D₁, D₂, and D₃), and the t statistic for each coefficient provides a test of its statistical significance. If years zero through eight are included in the analysis, and one dummy variable is used to represent all followup years, then its coefficient is the mean deviation from trend for years six through 8 (9 percentage points in the example above).

For an interrupted time-series analysis to be effective there must be a stable baseline trend and a pronounced deviation from this trend during the followup period. The more stable the baseline trend is (the less the points vary around the trend-line), the more confidence one can place in the forecast of the counterfactual for the followup period. The larger and more abrupt the post-Jobs-Plus deviation from trend is, the easier it will be to identify this deviation statistically.

Thus, for public housing characteristics that do not change appreciably before Jobs-Plus, or that change gradually over time, there might be enough statistical power to produce meaningful estimates of Jobs-Plus impacts. For outcomes that change erratically before Jobs-Plus, the statistical power of impact estimates probably will be limited.

Because we anticipate the impacts of Jobs-Plus to be large, we are hopeful that an interrupted time-series analysis can identify them. However, because these impacts might appear gradually over several years, we are somewhat concerned that it will be difficult to identify them with confidence. On balance, then, we are cautiously optimistic.

A separate interrupted time-series analysis could be conducted for each outcome measure for each Jobs-Plus development. One then could pool findings for a specific outcome measure across developments by taking the mean of their impact estimates and computing the standard error of this mean.⁵⁹ Pooling these findings across sites will increase their statistical power.

One could account for non-linear trends, if the need arises, by replacing the year counter (t) with its logarithm, by replacing the outcome measure (Y) with its logarithm, or both. This will allow for curvature without reducing the number of degrees of freedom.

Impact Study Design #3: Interrupted Time-Series Analysis
for Jobs-Plus Developments and Comparison Developments

A logical extension of the preceding analysis is to construct a separate interrupted time-series for a comparison development where Jobs-Plus is not implemented. Exhibit 3.4 illustrates how this time-series can be used to improve estimates of the impacts of Jobs-Plus. The approach is applicable regardless of how the comparison development was chosen (whether random assignment was used or not, and whether matching was used or not).

The top panel of Exhibit 3.4 repeats the time-series for the hypothetical Jobs-Plus development in Exhibit 3.3. The bottom panel presents findings during the same period for a comparison development. The interrupted time-series analysis for the comparison development yields deviations from trend in years 6, 7 and 8 equal to E₁, E₂, and E₃, respectively. If the comparison development and the Jobs-Plus development were chosen from the same local environment, then the comparison development deviation from its trend provides an estimate of what the deviation would have been for the Jobs-Plus development without Jobs-Plus (the counterfactual).

Hence, D_t - E_t provides an estimate of the impact of Jobs-Plus in year t. The variance of this difference equals the sum of the variances of D_t and E_t, so one can readily test the statistical significance of the difference.⁶⁰

It is useful to proceed with this analysis in two steps. First, compute and test the deviation from trend, D_t, for the Jobs-Plus development. This step asks the question: "Was there a statistically significant improvement in the outcome for the Jobs-Plus development?"⁶¹ Next compute and test the difference between the two deviations from trend, D_t - E_t,. This step asks the question: "Did the outcome improve by an amount that was statistically significantly greater than its likely change without Jobs-Plus?"⁶²

There are many ways to pool these findings for a sample of Jobs-Plus developments and comparison developments. The simplest way to do so is to subtract the mean deviation for the comparison developments from the mean deviation for the Jobs-Plus developments and compute the standard error for this difference of means.⁶³

One important variation on the interrupted time-series theme occurs when only two or three baseline observations exist. This will apply to Jobs-Plus outcomes that have limited consistent past data because: (1) their collection began only recently, (2) the way they are collected changed recently, or (3) they are only kept for short periods. Two or three baseline observations are better than only one (for before-after analyses), and are less effective than more baseline observations (for interrupted time-series analyses). But how should this information be used?

A conservative approach is to assume that the underlying system is not changing rapidly over time. Under this assumption, one would use the mean value of the baseline outcome to forecast the future outcome without Jobs-Plus (the counterfactual). The observed difference between the actual future outcome and this forecast provides an estimate of the impact of Jobs-Plus. Comparing this estimated difference with the corresponding difference for a comparison development could further strengthen the finding. Pooling these findings across developments could strengthen the findings even further. In all cases, simple tests for statistical significance are possible.

A less conservative approach is to assume that the observed annual change in the outcome for the baseline period will continue into the future.⁶⁴ Hence, this annual change rate becomes the counterfactual. One then could compare the annualized change rate for the baseline period with the annualized change rate for the followup period.

Instead of annual data for an outcome, it might be possible to obtain information for shorter units of time — for example, monthly crime rates. Data for shorter units of time provide more observations, which in turn, make it possible to use more sophisticated time-series models to estimate program impacts (see McCain and McCleary, 1979, and McDowall, McCleary, Meidinger, and Hay, 1980).⁶⁵

Such models can enable researchers to accommodate the serial correlation of observations over time and account explicitly for the effects of seasonality. There are probably very few Jobs-Plus outcomes for which such data are available, however. In addition, the ability of these data to represent long-term trends is not necessarily greater than the ability of annual data to do so. Furthermore, data for shorter time periods have a larger relative variance and greater serial correlation.⁶⁶ Hence, procedures based on such data probably will have limited application to the Jobs-Plus impact study. Nevertheless, they will be considered.

IV. Jobs-Plus Impacts on the Community

Introduction

This section explores ways to measure the impacts of Jobs-Plus on the surrounding community of a public housing development. The focus here is on changes in an area over time caused by changes in people and firms who remain in the area plus differences in the people and firms who move in and out of the area.

I first present a simple model of how Jobs-Plus might affect a local community and identify some outcome measures that could be used to examine these effects. I then briefly discuss issues that must be addressed when trying to identify the relevant community for a public housing development. I next examine approaches for estimating Jobs-Plus impacts. Because of the elusive and indirect nature of these impacts, and because of the limited data available for estimating them, the approaches available for this purpose are less well-structured than those discussed in earlier sections of this paper.

A Model of Jobs-Plus Impacts and Measures of Community Outcomes

Exhibit 4.1 presents a simple model of how Jobs-Plus might affect a community that is located near a public housing development where the initiative is implemented. It begins with the three basic elements of the Jobs-Plus program, which are expected to produce positive changes in the lives of public housing residents and improvements in the living conditions at their developments. As the environment in the development improves, it can start to become a "positive anchor tenant" for the surrounding community. Much like a major department store draws other establishments to a shopping center, a newly-transformed public housing development could attract residential and commercial development to its surrounding community. Similarly, it might stimulate physical improvements, increased public services, healthier and more stable social conditions, increased civic engagement, and increased community empowerment. Exhibit 4.2 lists a number of possible outcome measures that could be used to test for community improvements.

It should be noted, however, that while it is possible for improved conditions at a public housing development to spill over into its surrounding community, and thereby leverage improvements to a wider area, these spillover effects are indirect and secondary, relative to the direct effects of Jobs-Plus on public housing residents and the developments in which they live. In addition, these effects might take a long time to materialize, given the many forces that must converge in order for them to occur. Hence, of all the impacts hypothesized for Jobs-Plus, community effects are the most speculative. This is especially true given the many outside forces — such as macroeconomic conditions, racial conflict, government funding decisions, and other public policies — that play a large role in determining the condition of an urban community.⁶⁷

Defining the Appropriate Community

A community usually refers to a social, rather than a geographic, unit.⁶⁸ However, to assess the impacts of Jobs-Plus will require defining local communities in spatially distinct terms. To do so, will present a major challenge, because there is little agreement about what constitutes a community.⁶⁹ Nevertheless, four factors should be considered when trying to delineate community boundaries:

• physical barriers, such as rivers, highways, parks, railroad tracks, and large-scale industrial, commercial or residential development;
• social interactions, such as shopping patterns, visiting patterns, church attendance, and recreational patterns;
• political units, such as voting wards or districts, and school districts;
• statistical units for which data are reported, such as census blocks or tracts, police precincts, neighborhood planning districts, and zip codes.

Local residents should be involved in defining community boundaries. Their input will help to ensure that the relationship between the public housing development and the residential, commercial, and industrial development adjacent to it is considered when the analytic boundaries of a development’s community are established. The key here is to designate an area to which the public housing development is, or could be, genuinely related when the barriers which tend to isolate public housing are removed. If the designated area is too small, however, estimates of the impacts of Jobs-Plus on it might omit some important spillover effects. If the area is too large, estimates also will understate the true impacts of Jobs-Plus, because they will be diffused too widely to be identifiable. In the end, pragmatic considerations of data availability will play a major role in determining community boundaries for analysis purposes. One important issue in this regard is the fact that boundaries for reporting areas shift over time; hence, the comparability of such data should be examined very carefully.

Issues to Consider When Measuring Community Impacts

Before discussing specific approaches for estimating the community impacts of Jobs-Plus, it is important to acknowledge some key issues that will arise when attempting to measure these impacts.

First, it probably will be necessary to use different methods to measure impacts on different outcomes. Some outcomes will be measurable through quantitative methods, and some will be measurable through qualitative methods.

Second, due to limited data and resources, it might be necessary to restrict certain impact analyses to a few Jobs-Plus sites, and limit their generalizability accordingly.

Third, because of the indirect, diffuse, and somewhat speculative nature of the potential community impacts of Jobs-Plus, resources probably should be limited to measuring only those impacts judged most likely to occur, and most likely to be observable. In particular, the primary emphasis of the impact analysis for Jobs-Plus should be on its direct effects on public housing residents and public housing development residents, discussed earlier.

Fourth, an explicit emphasis should be placed on "triangulating" impact estimates.⁷⁰ This means trying to gauge the effects of Jobs-Plus in as many different ways as possible. Each method will be subject to different limitations, but if the findings produced by different methods tell the same basic story, a plausible empirical case can be made.

Fifth, one should expect the findings from any analysis of the community impacts of Jobs-Plus to be "messy." Hence, the story represented by these findings will be difficult to tell, and will require a high degree of craftsmanship from its authors.

Sixth and last, one should not proceed with a detailed analysis of the community impacts of Jobs-Plus, unless the initiative has been successful in terms of its implementation, and its impacts on public housing residents and developments.

Having acknowledged these conditions, the following are some potential approaches that might be used to assess the community impacts of Jobs-Plus.⁷¹

This approach is mainly subjective and, hence, relies heavily on qualitative methods. The basic idea is quite simple. In communities where Jobs-Plus is implemented, one could ask public housing residents, other community residents, community merchants, and community leaders how, if at all, Jobs-Plus improved their community. Through a combination of pre-coded and open-ended questions, one could elicit responses that provide specific examples of community improvements that were believed to be caused by Jobs-Plus and detailed explanations for why these outcomes should be attributed to Jobs-Plus.

These questions could be asked through sample surveys, in-depth personal interviews, focus groups,⁷² or some combination thereof. They could be asked at one point in time after Jobs-Plus had begun (presumably long enough after it had started to allow for community effects to occur), or at several points in time. The basic approach would be to look backwards in time, try to identify community improvements that had occurred during the period of interest, and try to articulate how Jobs-Plus caused these improvements.

Two strengths of this approach are its feasibility and its relative simplicity. More important, perhaps, is the fact that it would enable participants in the Jobs-Plus process to express in their own words and from their own perspectives how the initiative improved their community. This feature satisfies a key requirement of qualitative research methods and can, in some cases, produce highly convincing findings.⁷³

A potential weakness of the approach is the possibility that enthusiastic Jobs-Plus stakeholders will overstate its community impacts. Requiring specific examples of impacts and detailed explanations for how these impacts were created by Jobs-Plus, might restrain this tendency somewhat, and might provide a factual basis for gauging the validity of the approach. Nevertheless, there always will be a margin for overstating the initiative’s accomplishments. Hence, the approach should be used only in conjunction with other, more objective, although perhaps less rich, methods.

A detailed case study, conducted at one or more of the Jobs-Plus projects, could provide important insights into how and why the initiative did or did not produce community impacts. For this approach to be most effective, it should focus on "telling the Jobs-Plus story" in a structured way, that helps to develop and test a theory of change for the initiative (discussed earlier). This theory would trace the key intervening steps through which the initiative changes attitudes and behavior, and thereby produces community improvements.

A detailed case study would require extensive and continual presence by research staff at a Jobs-Plus site. It could include, among other activities: (1) extensive repeated interviews with key participants in Jobs-Plus; (2) attendance at Jobs-Plus meetings, planning sessions, and events; (3) focus groups with community residents, merchants, and leaders; (4) physical inspections of community conditions through windshield surveys and other "on-the-ground" approaches; and (5) access to written records and newspaper accounts that document key Jobs-Plus events, decisions, and actions. These activities could form part of the basis for the research on Jobs-Plus planning and implementation discussed earlier, as well as analyses of program impacts. Such extensive on-site presence might require a local researcher, much like that used to study the HOPE VI Program at the Hillside Terrace housing development in Milwaukee, Wisconsin.⁷⁵

Case studies can utilize both qualitative methods and quantitative methods. In addition, they can combine in-depth analyses of how Jobs-Plus evolved over time and the key forces which shaped its development, with descriptive analyses of existing conditions at one or more points in time (for example, the results of a sample survey of existing attitudes or the results of a windshield survey of existing physical conditions). As indicated above, the "glue" that will be needed to hold these disparate analyses together is a well-specified theory of how Jobs-Plus produced community impacts. Unfortunately, it is simple to state this requirement, but difficult to meet it. Nevertheless, without doing so, one cannot properly interpret the findings from a case study.

The primary strength of a case study is its richness of detail because of its depth of analysis. In particular, case studies that incorporate ethnographic methods⁷⁶ might be able to capture the cultural context of Jobs-Plus in a way that will help explain community residents’ responses to it. Such observations can provide a framework and perspective within which to understand some of the more quantitative measures that are obtained.

The primary weakness of a case study is its limited generalizability because of the limited number of cases that can be studied in this way. Hence, the more Jobs-Plus sites to which a case study approach is applied, the greater the generalizability of its findings will be. In addition, the more sites that are studied in this way, the greater the opportunities will be for obtaining knowledge through cross-site comparisons. The corresponding downside of increasing the number of sites, however, is the markedly increased cost of doing so. Nevertheless, because of the newness, the complexity, and the intensity of Jobs-Plus, it would seem prudent to conduct one or more case studies to learn as much as possible about the inner workings of the initiative.

For community conditions that can be measured at two or more points in time, such as crime rates, the number of local business establishments, the number of local jobs, and the amount of property tax revenues generated, one could conduct a before-after analysis for Jobs-Plus communities and comparison communities.

The basic approach would be the same as that described earlier for measuring the impacts of Jobs-Plus on public housing residents and public housing developments. The key difference here is that new outcome measures, and their corresponding data-sources, become relevant.

In particular, measures of Jobs-Plus impacts on community economic activities might be important. Unfortunately, previous attempts to measure the impacts of government programs on local economic activity, especially the effects of community economic development programs and enterprise zones, have met with limited success (see James, 1991, and Vidal, 1995).

One promising approach for overcoming this problem utilizes data from the Duns Market Identifiers (DMI) file, which provides almost a complete annual census of business establishments in major U.S. cities.⁷⁷ This information has been collected for several decades, and has been used by previous researchers to analyze patterns of industrial location.⁷⁸ Available data about individual business establishments include, among other characteristics: (1) their start-up date (which could be used to identify new businesses); (2) their standard industrial classification (SIC, which could be used to identify local industries that are growing or declining); (3) their number of employees (which could be used to measure job growth or decline), and (4) their postal zip codes (which could be used to identify establishments within Jobs-Plus communities or comparison communities).

DMI data are expensive to obtain, so they should be used on a limited basis. For example, they could be obtained for the year that Jobs-Plus begins, and for some later year (perhaps four or five years later) for the zip code(s) which most closely approximate a Jobs-Plus community. From this information, one could estimate the total number of establishments and the total number of jobs (overall and by industry) in each of the two years. One then could compute the absolute and percentage changes in these characteristics. In addition, one could determine the number and percentage of new establishment that appeared in the community and former establishments that no longer exist.⁷⁹ This would enable one to describe the changes in employment levels and business establishments that occurred during the analysis period.

A simple but more expensive extension of this approach would be to also obtain DMI data for some time in the past, say, for example, five years before Jobs-Plus was implemented. This would provide a measure of the change in employment and in the number of establishments (overall and by industry) during a relatively long period before and after Jobs-Plus was implemented. One then could examine the change in the annualized growth rate, which would provide a limited interrupted time-series analysis.

The primary strength of the preceding approach is its potential for providing objective information about the community impacts of Jobs-Plus. It main weaknesses are the expense of purchasing and processing the necessary data, and the methodological limitations of the before-after analyses that would be possible. For example, just because changes are observed in the employment levels of a community, or just because changes in growth rates are observed, does not mean that they were caused by Jobs-Plus. This causal inference could be strengthened, if corresponding DMI data were obtained for comparison communities. But even with these data, it would not be possible to conclude definitively that the change observed for the comparison communities represents what the corresponding change would have been for the Jobs-Plus communities without Jobs-Plus.

Hence, causal inferences about the community impacts of Jobs-Plus will need to rely heavily on supplementary findings about the Jobs-Plus theory of change. If one can establish that the preconditions for Jobs-Plus to produce community impacts were met (e.g., positive impacts on public housing residents and developments were observed, and the program was implemented successfully), then it might be possible to build a case that Jobs-Plus caused the community changes observed. If the community changes are large and occur consistently across Jobs-Plus sites, this case could be even stronger.

V. Concluding Thoughts

Jobs-Plus is planned to be a large-scale, saturation-level employment program for residents of selected public housing projects. As such, it represents a comprehensive community initiative whose impacts cannot be estimated from a randomized experiment, the now widely accepted best way for assessing the impacts of employment programs. Hence, a variety of non-experimental evaluation approaches are being considered for this project.

Given the relatively large impacts that are anticipated, it is hoped that a careful assessment of how key outcome measures change over time, both at Jobs-Plus "treatment developments" and at nearby comparison developments, will provide an adequate basis for measuring the key impacts of the program. Because many future programmatic initiatives are likely to focus on whole groups at a time instead of individual participants, it is hoped that the evaluation approaches being developed for Jobs-Plus will be applicable to evaluations of these future projects as well.

NOTES:

[1] Based on information from the 1989 American Housing Survey, Casey (1992, p. 11) estimates that 45 percent of the households in public housing received welfare or SSI. Based on information from the 1987 American Housing Survey, Newman and Schnare (1992, p. 56) estimate that 49 percent of the families with children in public housing and 19 percent of the elderly households in public housing received welfare or SSI.

[2] PHAs can increase the percentage of public housing residents who are employed in two ways: (1) by helping current residents find and hold jobs, and (2) by attracting low-income adults who are employed into public housing. The present paper focuses mainly on the first strategy.

[3] Because of the extreme complexity of Jobs-Plus, it is especially important to base its evaluation on an explicit theory, or set of theories, about how the initiative is expected to work, and what outcomes it is expected to produce. Chen (1990), Chen and Rossi (1987), and Bickman (1987), among others, have argued that such "theory-driven" evaluations are necessary in order to advance our understanding of social programs. More recently, Weiss (1995), Connell and Kubisch (1995), and Connell, Aber and Walker (1995) have argued that a well-specified theory of change is essential for the evaluation of "comprehensive community initiatives," of which Jobs-Plus is a prime example (discussed later).

[4] Gueron and Pauly (1991) provide the most extensive review of this research. Their summary of findings (pp. 15-20) clearly indicates that employment and training services can increase the future earnings of welfare recipients, although the magnitudes of these increases vary substantially across different types of programs, different types of participants, and different locations. In addition, recent findings from a large-scale study of Job Opportunity and Basic Skills (JOBS) programs in three sites — Atlanta, Georgia, Grand Rapids, Michigan, and Riverside, California (Freedman and Friedlander, 1995, p. ES-5) — indicate that two years after single parents on AFDC were offered labor market attachment services (to facilitate quick employment), their average monthly earnings were 26 percent higher than they would have been without the program. Furthermore, Goldman (1995, Table 5) indicates that AFDC recipients from Atlanta (the only site for which such findings are available currently) who lived in public housing when they entered the study, and subsequently were offered labor market attachment services, earned 56 percent more per month, two years after they entered the study than they would have earned without the JOBS program.

[5] The implicit tax rate on the earnings of low-income families often exceeds 100 percent. In other words, for each additional dollar earned, a family can lose more than one dollar of cash plus in-kind benefits. This situation reflects the combination of offsets to earnings that occurs if a family is receiving more than one type of benefit (e.g., AFDC, food stamps, Medicaid, and subsidized housing). The work disincentive is especially severe if a family’s income is near the threshold (or "notch") where it becomes ineligible for Medicaid. Work disincentives from housing subsidies are smaller than those from public welfare. Nevertheless, they are substantial, and have played an important role in housing policy debates (e.g., see PHADA/GAHRA, 1994; Wilkins, 1993). Perhaps more important, however, is the combined effect of work disincentives from both welfare and housing subsidies.

[6] Danziger et al. (1981) and Moffit (1992) survey the extensive literature on the effect of these work disincentives, the findings of which are mixed. To help resolve this controversy, two recent studies, the Canadian Self-Sufficiency Project (SSP) and the Minnesota Family Investment Program (MFIP), are using large-scale randomized experiments to test welfare reform plans that make work pay (see Card and Robins, 1996, for a discussion of SSP, and Knox, Brown, and Lin, 1995, for a discussion of MFIP). Initial findings for SSP indicate that making work pay can, indeed, help to increase employment. For example, in the fifth quarter after welfare recipients enrolled in SSP, they earned 58 percent more than they would have without the program’s substantial financial incentive. It is too soon to know how long these impacts will last, however. Corresponding findings are not available yet for MFIP.

[7] Findings from a major early study of resident management in seven public housing developments located in six cities (Manpower Demonstration Research Corporation, 1981) plus findings from a more recent study of 80 emerging resident management corporations from across the U.S. (ICF, Incorporated, 1993) support this conclusion.

[8] Building on Wilson’s (1987) seminal work, many authors have identified the absence of a positive, work-oriented social infrastructure (positive adult role models, well-attended churches, active neighborhood associations, etc.) in public housing, due to the extreme concentration of poverty there, as one of the greatest obstacles to its social and economic stability (e.g., see Spence, 1993, and Schill, 1993).

[9] The research agenda for Jobs-Plus includes an in-depth implementation study. However, the present paper focuses only on estimating the impacts of Jobs-Plus.

[10] This recognition has evolved over the past two decades based on careful methodological research. For example, after reviewing a wide range of sophisticated quasi-experimental estimates of the impacts of programs funded by the Comprehensive Employment and Training Act (CETA), an expert panel convened by the U.S. Department of Labor to help it decide how to evaluate the next generation of such programs, funded under the Job Training Partnership Act (JTPA), concluded that a randomized experiment was essential for this purpose (Stromsdorfer et al., 1985). Likewise, based on an extensive review of non-experimental research about the effectiveness of employment and training and programs for youths, a committee of experts convened by the National Academy of Sciences concluded that: "Future advances in field research on the efficacy of employment and training programs will require a more conscious commitment to research strategies using random assignment" (Betsey, Hollister, and Papageorgiou, 1985, p. 30). These conclusions reflect two main findings from methodological research on different methods for measuring the impacts of employment and training programs:

1. Different statistical matching and modeling techniques produce widely varying impact estimates (sometimes ranging from statistically significantly positive to statistically significantly negative) when applied to the same data-sets. Unfortunately, "Data limitations and the inability to adequately test the validity of the selection processes assumed make it impossible to determine which studies modeled the process correctly" (Barnow, 1987, p. 157).
2. When impact estimates from a wide range of quasi-experimental matching and modelling techniques were compared to corresponding impact estimates from randomized experiments, the quasi-experimental estimates varied widely and differed markedly from the experimental estimates (for example, see Fraker and Maynard, 1987, LaLonde, 1986, and Friedlander and Robins, 1995).

[11] More complex experiments randomly assign eligible applicants to a control group or to one of several different program groups. This makes it possible to estimate the "differential" impacts of different types of programs.

[12] More precisely, the "expected value" of every characteristic of the program group (whether it is measurable or not) equals its "expected value" for the control group. The larger the study sample is, the more similar the actual treatment and control group means will be for all characteristics.

[13] Kennedy (1988) reviews the findings from these studies. Apgar (1990) describes the influence of these findings on housing policy debates.

[14] Feins (1993) describes the proposed research design for this project.

[15] Impact estimates obtained from this design would be unbiased, but would have limited statistical power because of the small number of housing developments involved.

[16] Perhaps the most successful quasi-experimental research in the field of housing policy is the series of Fair Housing Audits which measured the incidence, nature, and intensity of discrimination against minority group members by landlords and real estate brokers. These studies provide compelling evidence about housing discrimination, in part because of their rigorous design, and in part because of the high levels of discrimination that exist in rental housing markets (for a review of these studies, see Yinger, 1988). Likewise, a major quasi-experimental study of saturation-level guaranteed jobs programs for in-school youths provided convincing findings about the effect of such job guarantees on the short-term employment rates of youth because these impacts were so striking and so immediate (Gueron, 1984). Another quasi-experimental study that has had a major impact on housing policy debates is that of the Gautreaux Program to promote fair housing in Chicago (see, for example, Rosenbaum, 1995). This study approximated a natural experiment whereby on a "nearly random" basis, some families received Section 8 housing subsidies for rental units in middle income suburban or outlying urban neighborhoods, while others received subsidies for units in low-income, inner-city neighborhoods. Adults who moved to middle income neighborhoods were much more likely to become employed than were those who remained in low-income neighborhoods.

[17] The conceptual framework used here was developed by Kubisch, Weiss, Schorr, and Connell (1995) , pp. 3-5.

[18] See U.S. Government Accounting Office (1992) for a discussion of this issue.

[19] See Newman and Schnare (1992) and Schlay (1993) for an extensive discussion of the issues surrounding the economic self-sufficiency of public housing residents and previous attempts to address these issues.

[20] Shlay and Holupka (1992) also based part of their analysis on a secondary comparison group of persons from Lafayette Courts who did not participate in the Family Development Center.

[21] This approach was used by Bryant and Rupp (1987) to estimate the impacts of employment and training programs funded by CETA, the Comprehensive Employment and Training Act.

[22] This approach was used by Dickinson, Johnson, and West (1987) to estimate the impacts of employment and training programs funded by CETA.

[23] Evaluation researchers refer to this as the problem of "history" (Cook and Campbell, 1979).

[24] Evaluation researchers refer to this problem as differential "maturation" (Cook and Campbell, 1979).

[25] Using other baseline characteristics as a proxy for these trends is a weak substitute for having data which describe them.

[26] See Ashenfelter (1978), Ashenfelter and Card (1985), Bloom (1984), and Bloom (1987) for a discussion of this issue.

[27] See Cook and Campbell (1979).

[28] Pre-program periods, and post-program periods for both groups are defined as calendar periods that occur before and after the program group is first exposed to the program.

[29] This statement applies to many different outcomes in many different fields (for example, employment and earnings, receipt of welfare and housing subsidies, homelessness, smoking, alcohol abuse, drug abuse, criminal behavior, child abuse, etc.) Conceptually, the statement is plausible because past outcomes can serve as a proxy for their many separate causes. Empirically, the statement is borne out by the fact that multiple measures on past outcomes usually provide much better predictions of future such outcomes than do data on separate hypothesized causal factors.

[30] Bloom (1984) used this approach and refers to it as a "time-varying, fixed-effect model." Ashenfelter and Card (1985) used this approach and refer to it as a "random coefficients" model. Although their computational procedures are different, the basic models are the same.

[31] If such additional variables are not included, then it turns out that the point estimate of the program impact based on the micro-data for all sample members (the difference between the mean deviation from trend for program group members and the mean deviation from trend for comparison group members) is identical to the point estimate that can be obtained from aggregate data on the mean outcomes for each group for each period, by computing the difference between the program group’s deviation from its aggregate trend and the comparison group’s deviation from its aggregate trend (Bloom, 1984).

[32] This problem applies equally to before-after, two-group comparisons.

[33] Researchers cannot agree on how to solve this problem for quasi-experimental longitudinal evaluations of employment and training programs (see Stromsdorfer et al., 1985, and Barnow, 1987). Hence, randomized experiments are now used to evaluate these programs.

[34] After the initiative begins, some families might choose to move into a Jobs-Plus development to increase their economic self-sufficiency. For these families, the selection process might be more like that of a traditional employment and training program. Nevertheless, because the choice of a housing unit involves many considerations beyond those which motivate enrollment in a job-training program, there probably is less room for the kind of self-selection on labor market potential that occurs in job-training programs.

[35] We refer to control developments here rather than comparison developments because we are discussing a true randomized experiment.

[36] An impact estimator based on the random assignment of Jobs-Plus developments and control developments is unbiased because the expected value of its sampling distribution equals the true impact being estimated. The estimator is inefficient (imprecise or uncertain), however, because its standard error is larger than the standard error for a sample with the same number of subjects that were randomly assigned individually to the program group and the control group (see Raudenbush, 1995, and Hays, 1973).

[37] See Bloom (1995) for a discussion of how to define and compute minimum detectable effects.

[38] For example, the National JTPA Study used a sample of 16,000 persons from 16 sites across the U.S. (Bloom et al., 1995).

[39] For a given total sample size, the cluster effect increases as the size of the clusters increase, and as the average difference between the clusters increases (see Raudenbush, 1995).

[40] For clusters of 200 subjects each and an intra-class correlation of 0.01 (which is small), the standard error of a sample produced by cluster assignment will be 1.7 times that produced by random assignment of individual subjects. For an intra-class correlation of 0.10 (which is large), the standard error for cluster assignment will be 4.8 times that for random assignment. The intra-class correlation measures how much the clusters differ from each other relative to how much individual subjects differ from each other.

[41] Other things equal, the larger the developments, the less variation in their year-to-year change in employment rates. We do not address this feature of the problem, however, because information about it (specifically, the intra-class correlation) does not exist. Instead, we try to tell a story about a range of variation that might exist for relatively large developments. If our story understates cross-site variation, it overstates the statistical power of Jobs-Plus impact estimates. If it overstates cross-site variation, it understates statistical power. We tried to construct an example with a lot of cross-site variation, so it would not overstate statistical power. A more informed estimate based on actual longitudinal data for earnings, employment, AFDC receipt, and receipt of food stamps is being developed currently.

[42] This finding represents the standard error of a difference of means, with five observations (developments) in the sample for the program group and five observations (developments) in the sample for the comparison group.

[43] With eight degrees of freedom, the critical t value at the .05 significance level is 1.86 for a one-tail test and 2.31 for a two-tail test.

[44] These models also are referred to as random coefficients models (Ashenfelter and Card, 1985).

[45] This program consists of a two-year "remedial stage" and a five-year "transition stage."

[46] Employment and earnings data were collected retrospectively at baseline for a period of several years for part of the sample in the National JTPA Study (Bloom et al., 1990).

[47] The baseline period will be shortest for residents who entered public housing most recently.

[48] This information is more useful for measuring the impacts of Jobs-Plus on public housing developments that participate in the initiative (discussed later).

[49] Workers who are not covered by Unemployment Insurance include mainly self-employed persons and certain types of federal employees.

[50] For a detailed comparison of earnings and employment data from UI wage records and from a survey for the same sample and the same time period, see Bloom et al., 1993a, Appendix E, pp. 345-366. Average earnings from UI wages records were about 25 percent lower than those from the survey; program impact estimates from the two data-sources were similar, especially when expressed in percentage terms.

[51] See, for example, Bloom et al. (1993b).

[52] Spence (1993) and Schill (1993) emphasize this issue.

[53] Recall that if random assignment is used, we will have a control group rather than a comparison group.

[54] See the discussion of sample size and minimum detectable effects in Section 2.

[55] Cook and Campbell (1979) outline the conceptual basis for interrupted time-series analysis. McCain and McCleary (1979) and McDowall, McCleary, Meidinger, and Hay (1980) describe how to apply time-series models to estimate interrupted time-series. Figlio (1995) provides an excellent application of this approach to measuring the effect of lowering the drinking age on alcohol-related traffic accidents.

[56] To do so, one could use the rent rolls of a PHA to identify who had lived in a particular development in a specific month each year — for example, January. One then could then obtain AFDC data for each sample member from the records of local or state welfare agencies. For each year that a sample member lived in the development, his/her AFDC data would be part of the time-series for that development.

[57] This implies that the best predictor of future behavior is long-term past behavior, which is the case for most outcomes.

[58] If there are at least four followup periods, it is possible, in theory, to estimate the impact of the program on the intercept and the slope of the original trend-line. We do not take this approach, however, because it does not focus directly on the magnitudes of the actual annual impacts and, hence, is more difficult to interpret.

[59] For example, if in followup year one, the estimate of the impact of Jobs-Plus for a single development is D, and the estimated variance of this estimate is s ², then the variance of the mean estimate for all n Jobs-Plus developments that year one is s ²/n.

[60] This simple formulation for the variance is possible because the two developments represent independent, non-overlapping samples.

[61] This implies a one-tail hypothesis test.

[62] This also implies a one-tail hypothesis test.

[63] It is also possible to specify a pooled regression to estimate this finding directly.

[64] Using the mean of the baseline outcomes to forecast future outcomes is more conservative than using the annualized change rate because the mean dampens the effect of year-to-year noise in the forecast, whereas the annualized change rate accentuates the effect of this variation.

[65] These procedures are generally referred to as Box-Jenkins models.

[66] By relative variance, we mean the "noise to signal ratio," which might be measured as the coefficient of variation.

[67] See Halpern (1995) for a discussion of this issue.

[68] See Coulton (1995), pp. 174-175, for a discussion of this issue.

[69] For a discussion of problems that arise when trying to define the boundaries of a community involved in a comprehensive community initiative, see Hollister and Hill (1995), pp. 130-131.

[70] See Jick (1979) for a discussion of triangulation, in general. See Brewer and Hunter (1989) for a discussion of how to triangulate alternative measures of a construct by testing their convergent and discriminant validity.

[71] The approaches discussed below also could be (and probably should be) used to study the impacts of Jobs-Plus on public housing developments. They are introduced here, however, because they are some of the few options which exist for measuring community impacts.

[72] Krueger (1994) explains in detail how to use focus groups for such purposes.

[73] Patton (1987) and Bryman (1988) discuss the importance of obtaining data from the frame of reference of participants in a process and in their own words.

[74] See Yin (1989) and Yin (1993) for an extensive discussion of this approach.

[75] See University of Wisconsin-Extension (1995).

[76] Examples of such ethnographic research include Stack and Burton (1994), Stack (1974), and Leavitt (1994).

[77] One problem with these data is the fact that they provide uneven coverage of very small firms, and firms in personal services industries (Schwartz, 1987).

[78] The coverage and quality of these data have improved markedly over time. Schwartz (1987) discusses issues that arise when using these data. Struyk and James (1975) apply them to the analysis of industrial location patterns.

[79] Schwartz (1987) suggests, however, that estimates of the number of new establishments in an area and estimates of the number of establishments that leave an area, based on DMI data, probably overstate these outcomes.

REFERENCES:

Allison, Paul D. 1988. Event History Analysis: Regression for Longitudinal Event Data (Beverly Hills, CA: Sage Publications).

Apgar, William. 1990. "Which Housing Policy is Best?" Housing Policy Debate, Vol. 1, No. 1, 1-32.

Ashenfelter, Orley and David Card. 1985. "Using the Longitudinal Structure of Earnings To Estimate the Effect of Training Programs", Review of Economics and Statisics, Vol. 67, November, pp. 648-660.

Ashenfelter, Orley. 1978. " Estimating the Effect of Training Programs on Earnings", Review of Economics and Statistics, Vol. 60, February, pp. 47-57.

Barnow, Burt S. 1987. "The Impact of CETA Programs on Earnings: A Review of the Literature". Journal of Human Resources 22 (Spring): 157-93.

Betsey, Charles, L., Robinson G. Hollister, and Mary R. Papageorgiou. 1985. Youth Employment and Training Programs: The YEDPA Years. Committee on Youth Employment Programs, Commission on Behavioral and Social Sciences and Education, National Research Council. Washington. D.C.: National Academy Press.

Bickman, Leonard, 1987. "The Functions of Program Theory" in Leonard Bickman ed. Using Program Theory in Evaluation, New Directions for Program Evaluation, No. 33 (San Francisco: Jossey-Bass).

Bloom, Howard S. 1995. "Minimum Detectable Effects: A Simple Way to Report the Statistical Power of Experimental Designs", Evaluation Review, Vol. 19, No. 5, pp. 547-556.

Bloom, Howard S. 1990. Back to Work: Testing Reemployment Services for Displaced Workers (Kalamazoo, MI: W.E. Upjohn Institute for Employment Research).

Bloom, Howard S. 1987. "What Works for Whom: CETA Impacts for Adult Participants" Evaluation Review, Vol 11, No. 4, pp. 510-527.

Bloom, Howard S. 1984. "Estimating the Effect of Job-Training Programs Using Longitudinal Data: Ashenfelter’s Findings Reconsidered" Journal of Human Resources, Fall, pp. 544-556.

Bloom, Howard S., Barbara Fink, Susanna Gurr and Wendy Bancroft. 1995. "The Canadian Earnings Supplement Project: Subsidizing Reemployment Instead of Subsidizing Unemployment" (New York: Social Research and Demonstration Corporation, November).

Bloom, Howard S., Larry L. Orr, Stephen H. Ball, George Cave, Fred Doolittle, Winston Lin and Johannes M. Bos. 1997. "The Benefits and Costs of JTPA Title II-A Programs." Journal of Human Resources, Vol. 32, No. 3 (Summer): 549-576.

Bloom, Howard S., Larry L. Orr, George Cave, Stephen H. Bell and Fred Doolittle. 1993a. "The National JTPA Study: Title II-A Impacts on Earnings and Employment at 18 Months" (Washington, D.C.: U.S. Department of Labor).

Bloom, Howard S., Larry L. Orr, George Cave, Stephen H. Bell, Fred Doolittle and Winston Lin. 1993b. "The National JTPA Study, Overview: Impacts, Benefits and Costs of Title II-A" (Bethesda MD: Abt Associates Inc., November).

Bloom, Howard S., Larry L. Orr, Fred Doolittle, Joseph Hotz and Burt S. Barnow. 1990. "Design of the National JTPA Study" (Bethesda, MD: Abt Associates Inc.).

Brewer, John and Albert Hunter. 1989. Multimethod Research: A Synthesis of Styles (Newbury Park, CA: Sage Publications) pp. 126-137.

Brown, Prudence. 1995. "The Role of the Evaluator in Comprehensive Community Initiatives" in James P. Connell, Anne C. Kubisch, Lisbeth B. Schorr and Carol H. Weiss, eds. New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts (Washington, D.C.: The Aspen Institute).

Bryman, Alan. 1988. "The Nature of Qualitative Research" in Bryman, Quantity and Quality in Social Research (London: Unwin Hyman), pp. 45-71.

Byrant, Edward C. and Kalman Rupp. 1987. "Evaluating the Impact of CETA on Participant Earnings", Evaluation Review, Vol. 11, No. 4, pp. 473-492.

Campbell, Donald T. and Julian C. Stanley. 1963. "Experimental and Quasi-Experimental Designs for Research on Teaching" in N. L. Gage ed., Handbook of Research on Teaching (Chicago: Rand McNally).

Card, David and Philip Robins. 1996. "Do Financial Incentives Encourage Welfare Recipients to Work? Initial 18-Month Findings from the Self-Sufficiency Project" (Vancouver, British Columbia: Social Research and Demonstration Corporation).

Casey, Connie H. 1992. "Characteristics of HUD-Assisted Renters and Their Units in 1989" (Washington, D.C.: U.S. Department of Housing and Urban Development, March).

Cave, George, Hans Bos, Fred Doolittle and Cyril Toussaint. 1993. JOBSTART: Final Report on a Program for School Dropouts. (New York: Manpower Demonstration Research Corporation).

Chen, Huey-Tsyh. 1990. Theory-Driven Evaluations. (Newbury Park, Calif: Sage Publications).

Chen, Huey-Tsyh and Peter H. Rossi. 1987. "The Theory-Driven Approach to Validity". Evaluation and Program Planning 10: pp. 95-103.

Connell, James P. and Anne C. Kubisch. 1995. "Applying a Theories of Change Approach to the Evaluation of Comprehensive Community Initiatives: Prospects and Problems" (Washington, D.C.: The Aspen Institute), Draft, November.

Connell, James P., J. Lawrence Aber and Gary Walker. 1995. "How Do Urban Communities Affect Youth? Using Social Science Research to Inform the Design and Evaluation of Comprehensive Community Initiatives" in Connell, James P., Anne C. Kubisch, Lisbeth B. Schorr, Carol Weiss eds. New Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts (Washington, D.C.: The Aspen Institute), 93-125.

Cook, Thomas D. and Donald T. Campbell. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings (Chicago: Rand McNally).

Corson, Walter S., Paul T. Decker, Shari M. Dunston, and Stuart Kerachksy. 1991. Pennsylvania Reemployment Bonus Demonstration: Final Report (Princeton, N.J.: Mathematica Policy Research).

Coulton, Claudia. 1995. "Using Community-Level Indicators of Children’s Well-Being in a Comprehensive Community Initiative" in Connell, James P., Anne C. Kubisch, Lisbeth B. Schorr, Carol Weiss eds. New Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts (Washington, D.C.: The Aspen Institute), 93-125.

Danziger, Sheldon, Robert Havemen and Robert Plotnick. 1981. "How Income Transfers Affect Work Savings and Income Distribution", Journal of Economic Literature, 19: pp. 975-1028.

Dickinson, Katherine P., Terry R. Johnson and Richard W. West. 1987. "An Analysis of the Sensitivity of Quasi-Experimental Net Impact Estimates of CETA Programs" Evaluation Review, Vol. 11, No. 4, pp. 452 - 472.

Feins, Judith. 1993. "Provide Implementation Assistance and Evaluation for the Moving to Opportunity Demonstration: Draft Research Design Plan" (Cambridge, MA: Abt Associates Inc., December).

Figlio, David N. 1995. "The Effect of Drinking Age Laws and Alcohol-Related Crashes: Time-Series Evidence from Wisconsin", Journal of Policy Analysis and Management, Vol. 14, No. 4, pp. 555-566.

Fraker, Thomas M. and Rebecca A. Maynard. 1987. "The Adequacy of Comparison Group Designs for Evaluations of Employment-Related Programs". Journal of Human Resources 22 (Spring): 194-227.

Freedman, Stephen and Daniel Friedlander. 1995. "The JOBS Evaluation: Early Findings on Program Impacts in Three Sites" (Washington, D.C.: U.S. Department of Health and Human Services and U.S. Department of Education, September).

Friedlander, Daniel and Philip Robins. 1995. "Evaluating Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods." American Economic Review, Vol. 85, No. 4 (September): 923-937.

Friedlander, Daniel and Philip K. Robins. 1995. "Evaluating Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods." American Economic Review 85 (September): 923-37.

Goldman, Barbara. 1995. "JOBS Program Impacts for Residents of Public Housing" (New York: Manpower Demonstration Research Corporation, memorandum, December 6).

Gueron, Judith M. 1984. Lessons from a Job Guarantee: The Youth Incentive Entitlement Pilot Projects (New York: Manpower Demonstration Research Corporation, June).

Gueron, Judith M. and Edward Pauly. 1991. From Welfare to Work (New York: Russell Sage Foundation).

Halpern, Robert. 1995. Rebuilding the Inner-City: A History of Neighborhood Initiatives to Address Poverty in the United States (New York: Columbia University Press: 1995) p. 233.

Hays, William L. 1973. Statistics for the Social Sciences (New York: Holt, Rinehart and Winston, Inc.)

Hollister, Robinson G. and Jennifer Hill. 1995. "Problems in the Evaluation of Community-Wide Initiatives" in Connell, James P., Anne C. Kubisch, Lisbeth B. Schorr, Carol Weiss eds. New Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts (Washington, D.C.: The Aspen Institute), pp. 127-172.

ICF Incorporated. 1993. "Report on Emerging Resident Management Corporations in Public Housing" (Washington, D.C.: U.S. Department of Housing and Urban Development).

James, Franklin J. 1991. "The Evaluation of Enterprise Zone Programs" in Roy E. Green ed. Enterprise Zones: New Directions in Economic Development (Newbury Park, CA: Sage Publications).

Jick, Todd D. 1979. "Mixing Qualitative and Quantitative Methods: Triangulation in Action" Administrative Science Quarterly, Vol. 24, December, pp. 602-611.

Kennedy, Stephen D. 1988. "Direct Cash Low-Income Housing Assistance" in Howard S. Bloom, David S. Cordray and Richard J. Light, eds. Lessons from Selected Program and Policy Areas (San Francisco: Jossey-Bass Inc.).

Knox, Virginia, Amy Brown and Winston Lin. 1995. "MFIP: An Early Report on Minnesota’s Approach to Welfare Reform" (New York: Manpower Demonstration Research Corporation, November).

Kotloff, Lauren, Phoebe Roaf, and Michelle Alberti Gambone. 1995. "The Plain Talk Planning Year: Mobilizing Communities to Change (New York: Public/Private Ventures)

Kubisch, Anne C., Carol H. Weiss, Lisbeth B. Schorr and James P. Connell. 1995. "Introduction" in James P. Connell, Anne C. Kubisch, Lisbeth B. Schorr and Carol Weiss, eds. New Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts (Washington, D.C.: The Aspen Institute), pp. 1-21.

Krueger, Richard A. 1994. Focus Groups: A Practical Guide for Applied Research (Thousand Oaks, CA: Sage Publications).

Lalonde, Robert J. 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data" American Economic Review 76 (September): 604-20.

Leavitt, Jacqueline. 1994. "Defining Cultural Differences in Space: Public Housing as a Microcosm" (College Park, MD: Urban Studies and Planning Program, University of Maryland).

Lesser, Cara. 1993. "Doesn’t Pay to Work: The Financial Disincentive to Work for Residents in Public Housing" (San Francisco: San Francisco Housing Authority).

Manpower Demonstration Research Corporation, Board of Directors. 1980. Summary and Findings of the National Supported Work Demonstration (Cambridge, MA: Ballinger Publishing Company).

Manpower Demonstration Research Corporation. 1981. Tenant Management: Findings from a Three-Year Experiment in Public Housing (Cambridge, MA: Ballinger Publishing Company).

McCain, Leslie J. and Richard McCleary. 1979. "The Statistical Analysis of the Simple Interrupted Time-Series Quasi-Experiment" in Thomas D. Cook and Donald T. Campbell, Quasi-Experimentation: Design and Analysis Issues for Field Settings (Chicago: Rand McNally), pp. 233-293.

McDowall, David, Richard McCleary, Errol E. Meidinger and Richard A. Hay, Jr. 1980. "Interrupted Time-Series Analysis", Sage University Papers (Newbury Park, CA: Sage Publications).

Moffitt, Robert. 1992. "Incentive Effects of the U.S. Welfare System: A Review", Journal of Economic Literature, 30: pp. 1-61.

Newman, Sandra J. and Ann B. Schnare. 1992. Beyond Bricks and Mortar: Reexamining the Purpose and Effects of Housing Assistance (Washington D.C.: The Urban Institute Press).

Patton, Michael Q. 1988. "The Nature of Qualitative Data" in Patton, Qualitative Evaluation Methods (Beverly Hills, CA: Sage Publications), pp. 21-37.

Public Housing Authorities Directors Association and Georgia Association of Housing and Redevelopment Authorities, Inc. 1994. "Rent Reform: The Missing Link in Upward Mobility for Public Housing Residents" (Washington, D.C.: Public Housing Authorities Director’s Association, Winter).

Raudenbush, Stephen W. 1995. "Statistical Analysis and Optimal Design in Cluster Randomization Studies" (East Lansing, MI: Michigan State University, January).

Riccio, James A. and Alan Orenstein. 1996. "Understanding Best Practices for Operating Welfare-to-Work Programs", Evaluation Review, Vol. 20, No. 1, February, pp. 3-28.

Rohe, William M. 1993. "From Public Housing to Home Ownership: An Evaluation of an Innovative Self-Sufficiency Program" (presented to the Association of Collegiate Schools of Planning, October 28-31).

Rosenbaum, James E. 1995. "Changing the Geography of Opportunity by Expanding Residential Choice: Lessons from the Gautreaux Program", Housing Policy Debate, Vol. 6, Issue 1, pp. 231-269.

Schill, Michael H. 1993. "Distressed Public Housing: Where Do We Go From Here?" University of Chicago Law Review, 60:2 (Spring): pp. 497-554.

Schwartz, Alex. 1987. "Dun and Bradstreet’s Market Identifiers and Economic Geography". Professional Geographer, Vol. 39, No. 2.

Shlay, Anne B. 1993. "Family Self-Sufficiency and Housing", Housing Policy Debate, Vol. 4, Issue 3, pp. 457-495.

Shlay, Anne B. and C. Scott Holupka. 1992. "Steps Toward Independence: Evaluating an Integrated Service Program for Public Housing Residents", Evaluation Review, Vol. 16, No. 5, pp. 508-533.

Spence, Lewis H. 1993. "Rethinking the Social Role of Public Housing", Housing Policy Debate, Vol. 4, Issue 3, pp. 355-368.

Spiegelman, Robert G., Christopher J. O’Leary and Kenneth J. Kline. 1992. The Washington Reemployment Bonus Experiment: Final Report (Kalamazoo, MI: W.E. Upjohn Institute for Employment Research, January).

Stack, Carol B. 1974. All Our Kin (New York: Harper and Row Publishers).

Stack, Carol B. and Linda M. Burton. 1994. "Kinscripts: Reflections on Family, Generation and Culture" in Glenn, Chang and Forcey, eds. Mothering: Ideology, Experience and Agency (Routledge).

Stromsdorfer, E., H. Bloom, R. Boruch, M. Borus, J. Gueron, A. Gustman, P. Rossi, F. Scheuren, M. Smith, and F. Stafford. 1985. "Recommendations of the Job Training Longitudinal Survey Research Advisory Panel". Washington, D.C.: Employment and Training Administration, U.S. Department of Labor, November.

Struyk, Raymond J. and Franklin J. James. 1975. Intrametropolitan Industrial Location (Lexington, MA: Lexington Books).

University of Wisconsin-Extension. 1995. "Hillside Terrace Baseline Case Study: Hope VI Program Assessment" (Milwaukee, WI: University of Wisconsin-Extension, June).

U.S. Government Accounting Office. 1992. "Public and Assisted Housing: Linking Housing and Supportive Service to Promote Self-Sufficiency" (Washington, D.C.: USGAO, April).

Vidal, Avis C. 1995. "Reintegrating Disadvantaged Communities into the Fabric of Urban Life: The Role of Community Development". Housing Policy Debate, Vol. 6, Issue 1, pp. 169-230.

Walker, Gary and Frances Vilella-Velez. 1992. Anatomy of a Demonstration: The Summer Training and Education Program (STEP) from Pilot through Replication and Postprogram Impacts (Philadelphia: Public/Private Ventures, Winter).

Weiss, Carol Hirschon. 1995. "Nothing as Practical as Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families" in Connell, James P., Anne C. Kubisch, Lisbeth B. Schorr, Carol Weiss eds. New Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts (Washington, D.C.: The Aspen Institute), pp. 65-92.

Wilson, William Julius. 1987. The Truly Disadvantaged: The Inner City, the Underclass, and Public Policy (Chicago: University of Chicago Press).

Yin, Robert K. 1989. Case Study Research: Design and Methods (Newbury Park, CA: Sage Publications).

Yin, Robert K. 1993. Applications of Case Study Research (Newbury Park, CA: Sage Publications).

Yinger, John. 1988. "Examining Racial Discrimination with Fair Housing Audits" in Howard S. Bloom, David S. Cordray and Richard J. Light, eds. Lessons from Selected Program and Policy Areas (San Francisco: Jossey-Bass).

U.S. Department of Housing and Urban Development, The Rockefeller Foundation, U.S. Department of Health and Human Services, The James Irvine Foundation, The Joyce Foundation, Northwest Area Foundation, Surdna Foundation, Inc., U.S. Department of Labor, The Annie E. Casey Foundation