Abstract

This paper examines issues and options for the design of a major non-experimental study to measure the impacts of a large-scale, saturation-level demonstration program to promote employment among residents of selected public housing developments. The program, Jobs-Plus, is being launched by the Manpower Demonstration Research Corporation in cooperation with the U.S. Department of Housing and Urban Development and The Rockefeller Foundation. [An updated, full list of the Jobs-Plus funding partners is provided at the front of this paper.] Because Jobs-Plus will be a comprehensive community initiative, available to all residents of the several public housing developments where it is implemented, the program cannot be evaluated using a randomized experiment, the now-standard method for measuring the impacts of employment and training programs. However, because community-wide initiatives are becoming an increasingly important component of social policy, it is essential to develop methods for determining their success. It is the purpose of this paper, therefore, to explore possibilities for doing so.

I. Introduction
Jobs-Plus

Anticipated sea-changes in welfare policy and housing policy are likely to place an enormous financial burden on U.S. public housing. In particular, welfare reform initiatives that limit the duration of benefits received by families will reduce the ability of public housing residents, almost half of whom are welfare recipients, to pay their rents.¹ In addition, proposed measures to consolidate federal housing subsidies probably will reduce the overall level of such funding. In response to these financial threats to the survival of public housing, it is imperative that local Public Housing Authorities (PHAs) find effective ways to increase the share of public housing residents who are employed.²

The benefits of resident employment go beyond financial and economic gains. Holding a job provides social opportunities, linkages to a larger community beyond public housing, positive role models for children in the development, and personal satisfaction. Creating a mixed-income community may reinvigorate the development, change public perceptions about public housing, and revitalize the surrounding neighborhood.

The Jobs-Plus Initiative being launched by the Manpower Demonstration Research Corporation (MDRC), in partnership with the U.S. Department of Housing and Urban Development (HUD) and The Rockefeller Foundation, is intended to help policy-makers and program managers learn some important lessons about what is required to increase the employment of public housing residents. To do so, MDRC will work closely with six PHAs that will develop locally-based approaches to providing saturation-level employment opportunities for working-age residents from at least one family housing development in each PHA.

Although the design of Jobs-Plus will vary across sites, reflecting their local resources, preferences and constraints, each site will rely on a combination of three strategies:

• implementing state-of-the-art employment and training services that have proved effective for welfare recipients and public housing residents;
• developing financial incentives that promote work through state welfare reform efforts and PHA modifications to rent rules and eligibility requirements;
• building a public housing community that actively supports work through resident groups and other local organizations.

Because of the enormous challenges facing public housing at his time, Jobs-Plus cannot afford merely to modify "business as usual." Instead it must provide a bold departure from previous attempts to increase the economic self-sufficiency of public housing residents.

The "theory of change"³ which underlies Jobs-Plus, and which gives rise to the expectation that it can, indeed, produce the major changes needed, derives from the premise that the synergy created by combining the initiative’s three main strategies will be substantial.

Previous studies have shown that state-of-the-art employment and training services can markedly increase the earnings of welfare recipients, in general, and welfare recipients who are public housing residents, in particular.⁴ Especially promising are services with direct and immediate links to jobs.

Previous research also has shown that public assistance programs and housing subsidy programs contain powerful financial incentives that inhibit work effort.⁵ If a poor family earns "too much" it can lose its Medicaid health benefits. In addition, its housing subsidy might be reduced, its AFDC benefits might be reduced, and its food stamp benefits might be reduced. If local welfare agencies and local PHAs can combine their efforts and find new ways to mitigate these disincentives, at least temporally, this could help motivate families on welfare and in public housing to work harder toward economic self-sufficiency.⁶

Previous research also has shown that in some cases, public housing residents can build effective organizations and take more control over their lives.⁷ Although such organizations are difficult to create, and require substantial time and resources to sustain, they are a potentially important vehicle for building the "social capital" necessary to foster a work-oriented environment in public housing.⁸

There is no evidence, however, about the impacts of combining these strategies in a concentrated way in a specific location. Nevertheless, given the considerable demonstrated potential of each separate strategy, and the likelihood that each would reinforce the others if they were implemented together, the impacts of a joint strategy could be far greater than the sum of the impacts of the separate strategies.

Research Questions

Research on Jobs-Plus should address the following questions:

• To what extent was it possible for the study sites to implement the three Jobs-Plus strategies: high-quality employment and training services, financial incentives that reward work, and a community culture that promotes a positive work ethic? To what extent were these strategies implemented together, and to what extent were they mutually reinforcing?
• What institutions, resident groups, local civic groups, and individuals played key roles in the development of Jobs-Plus at each site? In what ways and how well did these groups and individuals work together? How did they influence the design and the evolution of Jobs-Plus? In particular, what role was played by public housing residents?
• How long did it take for each study site to develop a Jobs-Plus initiative? What resources were necessary to make this to happen? How much did these resources cost? How, and from where, were these resources obtained?
• To what extent did public housing residents participate in the Jobs-Plus initiative? What leadership roles did they play in designing, implementing, and sustaining the initiative? What services did they receive, and in what activities did they participate? What were the main barriers to their participation? How, and to what extent, were these barriers overcome? What were the main forces that helped stimulate participation in the initiative?
• In what ways, and to what extent, did Jobs-Plus affect: (1) public housing residents in the study sites; (2) public housing developments where the initiative was implemented; and (3) communities within which these developments were located? How long did it take to produce these impacts, and to what extent were they sustained over time?
• Was Jobs-Plus cost-effective: (1) from the perspective of public housing residents who were involved; (2) from the perspective of local PHAs that were involved; (3) from the perspective of taxpayers; and (4) from the perspective of society overall?
• How did the evolution of Jobs-Plus differ across sites? How did its structure differ across sites? How did its results differ across sites? To what can we attribute this variation? What can be learned from this variation about the conditions necessary for an initiative like Jobs-Plus to be successful elsewhere?

Measuring Jobs-Plus Impacts⁹

To assess Jobs-Plus properly requires measuring its impacts. By definition, a Jobs-Plus impact is:

the difference between what happened when Jobs-Plus was implemented (its outcome) and what would have happened without Jobs-Plus (its counterfactual).

Consider the following situation. What if one year after Jobs-Plus was implemented in a public housing development, 25 percent of its households had an adult who was employed? Some of these adults probably would have been employed without Jobs-Plus. Thus, to estimate what Jobs-Plus actually caused to happen — its impact — one must estimate what the employment rate would have been without the initiative. If, for example, this rate would have been 10 percent without Jobs-Plus, then the impact of the initiative would be 15 percentage points. If, on the other hand, the employment rate would have been 25 percent without Jobs-Plus, then its impact would be zero.

In the field of employment and training research, randomized experiments are now widely regarded as the best way to estimate program impacts.¹⁰ In the words of one prominent researcher, randomized experiments are "a bit like the nectar of the gods: once you’ve had a taste of the pure stuff it is hard to settle for the flawed alternatives" (Hollister and Hill, 1995, p. 134).

Randomized experiments are like a lottery. They randomly assign eligible program applicants to either a program group, which is allowed to participate, or a control group which is excluded from the program, at least temporarily.¹¹ This process ensures that the two groups are similar in all ways, except for their access to the program.¹² Hence, the subsequent experience of the control group provides a valid estimate of the counterfactual for the program group. Differences between the experiences of the two groups therefore provide valid program impact estimates.

During the past two decades, this approach has been used widely to estimate the impacts of programs for economically disadvantaged adults (for example, see Bloom et al., 1997; and Manpower Demonstration Research Corporation, Board of Directors, 1980), programs for displaced workers (for example, see Bloom, 1990, Corson et al., 1991, and Speigelman et al., 1992), programs for welfare recipients (for an extensive review of these studies, see Gueron and Pauly, 1991), programs for "at risk" youths in school (for example, see Walker and Vilella-Velez, 1992), and programs for young school dropouts (for example, see Cave, Bos, Doolittle, and Toussaint, 1993).

Likewise, housing policy debates have relied extensively on findings from the two major randomized housing experiments conducted to date, the Experimental Housing Allowance Program (EHAP) and the Freestanding Housing Voucher Demonstration Program.¹³ In addition, HUD recently commissioned a large-scale randomized experiment, the Moving to Opportunity Demonstration, to study the effect of helping poor families move from inner-city ghettoes to moderate-income neighborhoods.¹⁴

Unfortunately (from a methodological perspective), Jobs-Plus is not a program to which individuals or households can be assigned randomly. Instead, it is a "comprehensive community initiative" to which all residents of the public housing developments involved will be "exposed," and in which all residents who desire can participate. Hence, it will not be possible to measure Jobs-Plus impacts by randomly assigning individuals.

It might be possible, however, to randomly select a program group and a control group of public housing developments at each site from among those willing and able to participate. This approach cannot provide the methodological protection available from standard large-scale randomized experiments, however, because of the small number of housing developments involved.¹⁵ Nevertheless, because Jobs-Plus impacts are expected to be large, the approach might provide enough statistical power to identify them. Furthermore, because these impacts are expected to be large, quasi-experimental methods might be able to provide plausible causal inferences about them (discussed later).¹⁶

Jobs-Plus is intended for a broad range of stakeholders who have very different priorities for the types of impacts that might be produced. To provide a coherent structure for reporting these impacts, we recommend examining them from three perspectives:

• impacts on public housing residents,
• impacts on public housing developments, and
• impacts on the surrounding community.

The first perspective represents the experience of persons who live in a public housing development where Jobs-Plus is implemented, regardless of whether or not they stay there. From this perspective, researchers will study the same people over time. The second perspective represents the experience of public housing developments where Jobs-Plus is implemented, recognizing that residents will move into and out of the developments. From this perspective, researchers will study the same developments over time. The third perspective represents the experience of a larger residential and business community, the composition of which will change over time. From this perspective, researchers will study the same communities over time.

Replicability of the Research Findings

As mentioned earlier, because Jobs-Plus will represent a major departure from previous attempts to increase the economic self-sufficiency of public housing residents, very little is known about this approach. Hence, to give Jobs-Plus a fair test at this early stage in its development will require selecting sites that are committed to making the concept work and judged likely to be able to do so. Thus, the goal of the initiative probably will be to measure what Jobs-Plus can achieve under conditions that are more favorable than average. Nevertheless, it would be unwise to choose sites for Jobs-Plus that are so unusual that their findings are not replicable. Therefore, a careful balance will be maintained between the expected commitment and capability of the sites selected for the demonstration and the likelihood that other sites, but not necessarily all sites, could replicate their experience in the future.

Developing New Methods to Assess Comprehensive Community Initiatives

• A horizontally complex intervention. Jobs-Plus will integrate the efforts of numerous social service systems (especially welfare and housing agencies) plus the activities of many civic, business, and community groups. Therefore, it will have many facets, whose separate impacts will be difficult to distinguish.
• A flexible and evolving intervention. Jobs-Plus sites will have wide latitude to mold the initiative. Furthermore, each site’s vision for the project will change over time. Hence, Jobs-Plus projects probably will differ from their initial plans, and probably will take on different forms at different times. This makes it especially important, and difficult, to document precisely what the intervention was at each site, and how it changed over time.
• A broad range of outcomes to examine. Jobs-Plus is intended to produce changes in many different dimensions (economic, psychological, social, political, and physical) and changes that will occur over widely varying time-frames (immediate, intermediate, and long-run). Hence, the number of outcome measures to consider is substantial, and the task of summarizing them will be difficult.
• Vertically complex impacts. Jobs-Plus is intended to produce impacts that are experienced at different levels and that therefore should be examined from different perspectives. As indicated above, Jobs-Plus is expected to produce impacts on public housing residents, on public housing developments, and on the communities surrounding these developments.
• Important contextual issues. Although Jobs-Plus will be implemented in local communities, its evolution, its final form, and its results will be influenced by many forces which are outside of its control. For example, federal regulations (and possible waivers) which govern welfare and housing programs will determine the range of available programmatic options. Economic conditions in the community, in the region, and nationally will influence the job opportunities through which economic self-sufficiency can be achieved. Barriers due to racial and ethnic discrimination will limit the success and influence the direction of the initiative. Therefore, to understand why certain impacts did or did not occur requires a detailed knowledge of the environment within which Jobs-Plus operated.
• Absence of an adequate experimental control group. For reasons outlined above, it will not be possible to randomly assign individuals to Jobs-Plus or a control group. Thus, it will not be possible to estimate Jobs-Plus impacts using the approach now relied on by evaluations of most employment and training programs. Nevertheless, it will be crucial to determine what, in fact, Jobs-Plus caused to happen.

The preceding problems, inherent in evaluations of comprehensive community initiatives, have been the focus of current work by the Roundtable on Comprehensive Community Initiatives for Children and Families, sponsored by the Aspen Institute. This group has: (1) identified and clarified the many serious problems experienced by such evaluations (Hollister and Hill, 1995); (2) made a forceful case for the need to find ways to overcome these problems (Kubisch, Weiss, Schorr, and Connell, 1995, and Brown, 1995), and (3) begun to sketch an alternative evaluation framework, based on "theories of change" (Weiss, 1995, and Connell and Kubisch, 1995).

This work is in its very early stages, however, and no new evaluation methods or models have been developed or tested. Therefore, research designs for Jobs-Plus will have to rely on combining existing methodologies in creative ways and developing new approaches.

This Paper

This present paper focuses on estimating the impacts of Jobs-Plus from the three perspectives described above. Section 2 discusses impacts from the perspective of the residents of Jobs-Plus developments. Section 3 discusses impacts from the perspective of the developments themselves. Lastly, Section 4 discusses impacts from the perspective of the community surrounding a Jobs-Plus development. All plans for the Jobs-Plus evaluation are tentative at this time. The final design will evolve during the pilot phase of the project.

II. Jobs-Plus Impacts on Public Housing Residents

Introduction

This section explores ways to measure the impacts of Jobs-Plus on residents of public housing developments. To measure these impacts requires observing the same people over time, regardless of whether they stay in the development where Jobs-Plus was implemented or move away.

The section begins with a simple model of the process by which Jobs-Plus is expected to produce its intended effects. This model provides a conceptual framework for defining program outcome measures. Three research designs for measuring Jobs-Plus impacts are then considered, followed by a discussion of sample size and minimum detectable effects. The section concludes with a discussion of data issues.

A Model of Jobs-Plus Impacts and Measures of Jobs-Plus Outcomes

Exhibit 2.1 illustrates a four-stage process which describes how Jobs-Plus is expected to improve the lives of public housing residents. The process starts with the Jobs-Plus program and baseline characteristics of the residents, their public housing development, and its surrounding community. The three main components of Jobs-Plus — program services, financial incentives, and community support — will provide the impetus for improvements in the lives of public housing residents. The baseline characteristics of these residents, their development, and its surrounding community will influence their ability to benefit from Jobs-Plus.

For example, personal characteristics (such as residents’ education levels, labor market experiences, dependence on public assistance, and attitudes toward life, work, and welfare) will play a large role in determining their value to potential employers. Likewise, characteristics of public housing developments (such as their perceived safety, the presence of working adults, the availability of child-care services, the existence of supportive organizations, and their proximity to job opportunities) can affect residents’ willingness and ability to seek employment. Similarly, characteristics of the community surrounding a public housing development (such as its perceived safety, the existence of nearby jobs, access to public transportation, and the presence of retail facilities) can facilitate or impede the labor market success of public housing residents. This constellation of baseline characteristics defines the setting in which Jobs-Plus will take place and will be documented carefully.

The second stage in the process involves some initial Jobs-Plus outcomes that must occur in order for the initiative to influence the decisions and behavior of public housing residents. First, residents must understand how Jobs-Plus changes their incentives to work, through modifications of welfare eligibility requirements and benefit determination, and through modifications of public housing eligibility requirements and rent calculations. If residents understand these provisions, they can respond to them accordingly; if they don’t understand them, they cannot respond systematically. In addition, public housing residents will need a wide variety of services to help them overcome their many serious barriers to employment. If they receive these services, they should be better able to compete effectively for jobs. Without these services, they will not be as competitive. Finally, Jobs-Plus will help tenant organizations and other neighborhood groups to create a personally-supportive, work-focused environment. If such a climate is created, and if public housing residents perceive it to exist, then it could provide an important boost to their self-improvement efforts.

These initial outcomes are anticipated prerequisites for the success of Jobs-Plus and will be examined as part of the implementation research for the project. Hence, they will provide an early indication of whether or not Jobs-Plus is likely to be successful.

The third stage in the process involves changes produced by Jobs-Plus in the attitudes, the human capital (knowledge, skills, and credentials), and the behavior of public housing residents, plus changes in the extent and nature of their interactions and their level of organization. It is directly through these changes that Jobs-Plus is expected to produce its intended effects on economic self-sufficiency. For example, to the extent that exposure to Jobs-Plus increases the self-confidence of public housing residents about finding and holding a job, they are likely to try harder to do so. To the extent that Jobs-Plus increases the knowledge and skills of public housing residents, and helps them obtain educational and vocational credentials, it will increase their potential productivity, and thereby increase their value to employers. To the extent that Jobs-Plus increases the amount of time that public housing residents spend in productive activities (such as study, job-search, volunteer work, or helping children) and to the extent that Jobs-Plus decreases the incidence of negative behaviors (such as alcohol abuse, drug abuse, and criminal activities), it will help to foster life skills and behavior patterns that support, instead of impede, gainful employment. Likewise, to the extent that public housing residents interact more frequently with each other, and develop more cohesive organizations, they are more likely to reinforce each other’s attempts to become and remain employed.

Only if these changes occur, can Jobs-Plus improve the lives of public housing residents. Hence, measuring these intermediate impacts provides an interim test of whether Jobs-Plus is working as it was intended.

The fourth and final stage in the process involves actual measures of the progress made by public housing residents toward economic self-sufficiency. There is disagreement, however, about what constitutes economic self-sufficiency.¹⁸ At a minimum, it requires gainful employment. In addition, most people would agree that it involves a lack of dependence on AFDC and food stamps. Agreement is less uniform, however, about whether or not persons who receive housing subsidies (which have higher income eligibility levels than AFDC) should be considered self-sufficient. It is even less clear whether or not persons who receive subsidized health care, through Medicaid, should be considered self-sufficient. Lastly, there are many families that have a full-time worker and receive no form of public assistance, but nevertheless remain in poverty. Should these families be considered self-sufficient? Given the complexity of the concept of self-sufficiency, and the widely varying views of what the concept means, the impact of Jobs-Plus should be measured in ways that reflect these different views.

Exhibit 2.2 lists outcome measures that reflect the preceding model, and thereby could serve as the basis for assessing the initial, intermediate, and long-term impacts of Jobs-Plus.

Impact Study Design #1: A Two-Group, Before-After Analysis

The Approach

• the program outcome is represented by the change in an outcome measure from before the program was implemented (usually at baseline) to after it was implemented,
• the counterfactual is represented by the corresponding change for the comparison group,
• the impact estimate is the difference between the observed change for the program group and the observed change for the comparison group.

In theory, this approach could be used to estimate impacts with regard to any of the outcome measures listed in Exhibit 2.2. To the extent that the observed change in the measure for the comparison group accurately reflects what the change for the program group would have been without the program, the approach provides a valid estimate of the impact for the sample.

The key to making this approach work is finding the right comparison group. Shlay and Holupka (1992), in their study of the Family Development Center at the Lafayette Courts public housing development in Baltimore, Maryland, chose their principle comparison group from families at a nearby public housing development, Murphy Homes.²⁰ Rohe (1993), in his study of the Gateway Transitional Families Program, in Charlotte, North Carolina, chose his comparison group from persons who applied to the program but did not participate.

To increase the validity of findings from a two-group, before-after design, it is sometimes possible to match comparison group members to program group members in terms of measured baseline characteristics. There are two basic versions of this approach:

• cell matching, whereby: (1) subgroups (cells) are defined according to combinations of baseline characteristics (e.g., age category, gender, race, and welfare status), (2) each program group member is placed in the cell to which he or she belongs, and (3) comparison group members are chosen to match the number of program group members in each cell;²¹
• distance function matching, whereby: (1) a weighted function of baseline characteristics is computed for each program group member and for each potential member of the comparison group; and (2) a comparison group member is chosen to match each program group member in a way that minimizes the difference (or "distance") between their values of the weighted function.²²

In addition to statistical matching procedures, researchers usually estimate program impacts from a multiple regression equation, of the following form, which attempts to control for differences between program group and comparison group baseline characteristics which are included in the model

where:

Y₂ = the follow-up value of the outcome measure,

Y₁ = the baseline value of the outcome measure,

T = one for program group members, and zero for comparison group members,

Xj = a baseline characteristic,

a = an intercept term,

B ₀ = the program impact,

B ₁ and B _j = regression coefficients, and

e = a random error term.

Strengths and Weaknesses

The principal strengths of the approach are its relative simplicity, and the fact that it could be implemented readily for Jobs-Plus. The main weaknesses of the approach are several important "threats" to the internal validity of its program impact estimates. These threats represent specific reasons why the change in the comparison group outcome might not accurately reflect what the change would have been for the program group without Jobs-Plus.

First, is the possibility that the comparison group and the program group will experience different events that affect their willingness and ability to work.²³ Hence, it is probably advisable to choose comparison group members who are from the same local labor market and are served by the same welfare agency and public housing authority.

Several versions of this problem are likely to occur. For example, while the study is underway, other government programs probably will change in ways that affect the Jobs-Plus target population. In particular, welfare reform measures are likely to be taken by some, if not all, of the jurisdictions where Jobs-Plus is implemented. If the comparison group experiences local welfare reform and the program group experiences Jobs-Plus, then impact estimates will represent the differential effect of Jobs-Plus versus local welfare reform. If welfare reform in the Jobs-Plus PHAs is sufficiently dramatic, its effects could over-shadow those of Jobs-Plus, and thereby make it very difficult to measure Jobs-Plus impacts.

A second potential problem of this type is represented by the fact that HUD will be conducting another major demonstration project in many of the PHAs where Jobs-Plus might take place. This new project, the Moving to Work Demonstration, will test increases in the flexibility of federal regulations that control PHA operations. The increased flexibility might enable PHAs to modify their eligibility standards for public housing and their rent determination procedures in ways that also will stimulate work effort. Furthermore, changes in federal regulations might enable PHAs to better coordinate services and activities with other organizations. In short, the Moving to Work Demonstration could provide a competing "treatment" to Jobs-Plus. If it is not possible to withhold this treatment from enough public housing developments to maintain an untreated comparison group, then impact estimates obtained by comparing Jobs-Plus sites and comparison sites will represent the "differential" impacts of the two programs.

A second common problem with two-group, before/after designs is the fact that the program group and the comparison group are often on two different long-term trends before they were chosen for the study.²⁴ Hence, even if the study had not been conducted, subsequent changes observed for the two groups would be different. If, for example, tenants at one public housing development were more likely to find jobs than were the tenants at another development, the subsequent changes in their employment rates would be different in the absence of Jobs-Plus. With only one baseline outcome measure for each group, there is no way to identify their underlying trends. Hence, there is no effective way to compensate for them.²⁵

A third common problem with two-group, before/after designs also arises from the fact that only one baseline outcome measure exists. This problem results not from the existence of underlying trends, but from a temporary random departure from this trend that occurs at baseline.

For example, previous studies of employment and training programs have identified a "pre-program dip" whereby average earnings in the year before participants enter a program are well below their preceding trend.²⁶ This phenomenon has been the subject of extensive debate, and there is still no agreement about the extent to which it represents a temporary aberration (due to illness, a business failure, or other "bad luck"), or the extent to which it represents the onset of a permanent decline in earnings (which happens to economically displaced workers).

Individuals are probably more likely to enroll in a job-training program when they have just experienced an unusually bad period than when they have just experienced an unusually good period. In other words, their motivation to enroll is probably negatively correlated with random fluctuations in their earnings prospects. If so, then their average earnings in the next period are likely to increase regardless of whether or not they enroll in a program. This phenomenon is a statistical artifact called regression to the mean.²⁷

With two groups, the problem becomes one of different changes in earnings due to different regression artifacts. Hence, the observed change for the comparison group will not accurately represent what this change would have been for the treatment group, without the treatment.

The second and third problems mentioned above represent two different reasons why the program group and the comparison group differ initially in ways that would cause them to experience different outcomes, even in the absence of Jobs-Plus. Hence, they represent two different forms of "selection bias."

Impact Study Design #2: Longitudinal Data

The Approach

The second approach to estimating the impacts of Jobs-Plus is a logical extension of the first, which is intended to deal with: (1) program groups and comparison groups that have different underlying outcome trends, and (2) program groups and comparison groups which have different regression artifacts. To address these issues requires data for both groups on outcome measures for a number of years (four to five) before Jobs-Plus begins, and as long as needed to identify impacts which materialize thereafter. The longer it takes for impacts to materialize, the longer the followup period that is required. This type of information is generally referred to as longitudinal data or panel data.

From these data, one can compute a "pre-program" trend for each program group member and each comparison group member. It is then possible to compute a separate post-program deviation from each person’s pre-program trend.²⁸ The difference between the mean deviation from trend for the Jobs-Plus group and the mean deviation from trend for the comparison group provides an estimate of the impact of Jobs-Plus. This estimate represents: the extent to which Jobs-Plus caused participants to change their long-term labor market behavior beyond the change that would have occurred without Jobs-Plus.

Because past behavior, especially past behavior for a number of years, is usually the best predictor of future behavior, comparing sample members’ post-program outcomes to their pre-program trend (thereby allowing each sample member to serve as his or her comparison group) is probably the best way, absent a randomized experiment, to control for underlying differences in their likely future behavior.²⁹

Giving individuals their own trend implies estimating a separate intercept and a separate slope for each. This is a straightforward extension of standard "fixed-effect" models for longitudinal data, which specify a separate intercept for each sample member.³⁰ It is possible to include additional explanatory variables in such models, but this generally does not increase their explanatory power appreciably because the additional variables are usually reflected in the past outcome trends for each sample member.³¹

Strengths and Weaknesses

The principal strength of this approach is its ability to control for differences in the underlying trends of the program group and the comparison group. In addition, because the approach provides an explicit description of the behavior of sample members over time, it can identify aberrations that might have occurred before an intervention took place. If no such aberration exists, then regression artifacts should not be a problem. If, on the other hand, an aberration appears in the data for either the program group or the comparison group, or both, then the potential for differential regression artifacts exists. Nevertheless, impact estimates based on longitudinal data are less susceptible to the influence of regression artifacts than are simple before/after comparisons.

One potential weakness of longitudinal impact estimators is the possibility that the program group and the comparison group are subject to different major local events (history). If this occurs, then differences in the changes observed for the two groups might reflect differences in the events to which they were exposed, not the impact of Jobs-Plus. Only by choosing program and comparison groups that are likely to be exposed to the same events, and carefully documenting the events to which they were exposed, can one properly interpret the impact estimates obtained.

A second potential weakness of longitudinal impact estimators is the possibility that past behavior will not adequately reflect potential future for program groups members because they have reached a point in their lives where they will make a major change.³² To see this point, consider the following situation.

Enrolling in a job-training program might signal a desire to change one’s situation in life. Even if someone has been out of the labor force for a long time, or has been in and out of low-pay jobs for many years, he or she might be ready, able, and willing to make a change with or without participating in a program. If so, then data over time on prior employment, earnings, welfare receipt, or housing subsidies will not provide an adequate statistical control for likely future changes in these outcomes. Indeed, nothing short of a randomized experiment will do so.³³

Fortunately, because of a fundamental difference between the participant selection process for employment and training programs, and that for Jobs-Plus, longitudinal impact methods (even two-group, before-after methods) might be more suitable for evaluating Jobs-Plus than for evaluating employment and training programs.

Individuals will become members of the Jobs-Plus program group if they happen to live in a public housing development where Jobs-Plus is implemented. They will not become members of the program group if they do not live in a Jobs-Plus development. Hence, at least when the initiative begins, Jobs-Plus "comes to the individuals," the individuals do not "come to Jobs-Plus."³⁴ Therefore, the program group should not exhibit a pre-program dip. Consequently, it should not exhibit a regression artifact.

Furthermore, there is no reason to expect that the proportion of program group members who are planning to change their lives differs appreciably from the proportion of residents at other similar developments who plan to do so.

Impact Study Design #3: Random Assignment of
Public Housing Developments³⁵

A third approach for estimating the impacts of Jobs-Plus, which builds on the preceding two approaches, is a randomized experiment, in which: (1) several public housing developments are chosen as candidate Jobs-Plus sites from each PHA that participates, (2) these developments are matched as closely as possible, and (3) a lottery is used to determine which development implements Jobs-Plus and which become members of a control group.

If this approach is feasible, it could be used in conjunction with either or both of the preceding approaches. By randomly deciding which candidate developments become Jobs-Plus sites and which become control sites, one can eliminate systematic forces that might produce initial differences between the two groups. Only by chance would such differences occur. Hence, this procedure would eliminates bias in program impact estimates. Nevertheless, a substantial margin for random sampling error would remain, because residents would be randomly assigned by development, not independently as individuals.³⁶

Note that this random sampling error will be present whether public housing developments are chosen randomly for the program and control groups or whether they are chosen in some other way (which would not eliminate bias). The problem with respect to random sampling error arises because Jobs-Plus applies to whole public housing developments, not to specific residents.

Sample Size and Minimum Detectable Effects³⁷

Sample size is key to the Jobs-Plus impact analysis. To place this issue in perspective, first consider the magnitude of impacts produced by successful welfare-to-work programs that have some of the planned features of Jobs-Plus (Exhibit 2.3).

Column one in the exhibit reports some of the first available findings from demonstration programs being conducted in Atlanta, Georgia; Grand Rapids, Michigan; and Riverside, California, under the Job Opportunities and Basic Skills (JOBS) Program. These findings reflect the impacts on employment rates of a "Labor Force Attachment" approach to stimulating employment, through job-search assistance, followed by work experience or short-term education and training activities, if needed. The second column presents unpublished findings from the same study for a subsample of public housing residents in Atlanta. The third column reports recently published findings from a major Canadian study of an earnings supplement designed to "make work pay" by guaranteeing (for up to three years) a substantial minimum earnings to welfare recipients who find a full-time job within a year and remain employed full-time thereafter.

In each case, the employment rate for program group members observed eighteen months to two years after enrollment was 8 to 9 percentage points higher than it would have been without the program. These findings were statistically significant.

Furthermore, these findings represent substantial impacts on employment. To see this, it is useful to express each impact as a percentage of the corresponding employment rate for the control group. Hence, the 8.1 percentage point impact in the first column represents a 23.5 percent increase from the 34.4 percentage point employment rate for control group members. In other words, 23.5 percent more of the program group members were employed than would have occurred without the program. Corresponding impacts in these terms are a 28.0 percent increase in employment for column two and a 28.4 percent increase in employment for column three.

The impacts of Jobs-Plus should exceed those in Exhibit 2.3, because it will combine the separate approaches used by the programs represented in the exhibit, and add a third important component to the mix. As discussed earlier, Jobs-Plus will include state-of-the-art employment and training services; financial and other incentives that promote work; and vigorous efforts to build a community that supports work. Hence, Jobs-Plus should increase employment rates by at least 10 percentage points. Therefore, the impact study design for Jobs-Plus must have enough statistical power to detect impacts of roughly this size.

Findings in the exhibit were based on total samples that ranged from just under 800 persons to just over 1,900 persons, which is typical of many recent experiments, but is considerably smaller than the largest ones.³⁸ These samples could detect impacts on employment rates of 8 to 9 percentage points. Thus, if the Jobs-Plus sample were selected in a similar way (by randomly assigning individuals to program or control status), 1,000 to 2,000 public housing residents would be adequate to detect the types of impacts anticipated if Jobs-Plus is successful.

However, Jobs-Plus sample members will enter the study in groups, or "clusters," either as residents in public housing developments selected for the Jobs-Plus program, or as residents in public housing developments selected for the comparison group. This "cluster assignment" will increase the standard errors of Jobs-Plus impact estimates, thereby reducing their statistical power. This loss of power could be substantial, because the clusters (developments) will be large (containing 200 to 400 families each).³⁹ Indeed, the standard errors for impact estimates from a Jobs-Plus sample might be several times as large as those from a sample of the same size produced by random assignment of individual public housing residents.⁴⁰

Without information about the "intra-class correlation" for the outcome measures to be used, it is not possible to determine whether any given number of public housing residents will be adequate for the study. Nevertheless, to provide some insight about the likely statistical power of Jobs-Plus impact estimates, Exhibit 2.4 presents a hypothetical example.

The exhibit lists employment rates for five hypothetical housing developments in two consecutive years, without Jobs-Plus. The size of these developments is not specified, but I assume that they are relatively large, and focus on aggregate employment rates.⁴¹ The magnitudes of these employment rates are similar to those observed by the studies represented in Exhibit 2.3.

What is more important, however, is the variation in the changes in employment rates. These changes range from an increase of 10 percentage points to a decrease of 10 percentage points (which is substantial). Their standard deviation is 8 percentage points.

If this situation approximates the actual variability of changes in employment rates for public housing developments like those in the Jobs-Plus sample, then one can approximate the minimum detectable effects of Jobs-Plus using the public housing development as the unit of analysis.

For example, what if one Jobs-Plus development and one comparison development are chosen from each of five PHAs? Estimates of impacts on changes in employment rates obtained from this sample of 10 developments would have a standard error of 5 percentage points.⁴² This means that an impact estimate greater than roughly 10 percentage points would be statistically significant at the .05 level.⁴³

If instead, we chose one Jobs-Plus development for each PHA and two developments for the comparison group, to increase statistical power, the standard error of the impact estimate would be 4.4 percentage points. Thus, an impact estimate greater than about 9 percentage points would be statistically significant at the .05 level.

Given the preceding assumptions, it therefore should be possible to detect impacts on employment rates that are comparable to those produced by state-of-the-art employment and training programs and targeted financial incentives.

I next briefly discuss some important data issues and options.

• Whom should we observe?
• What types of outcomes should we measure?
• For what period should we measure these outcomes?
• Where can we get information about these outcomes?

To measure the impacts of Jobs-Plus on public housing residents will require defining the target population as all persons living in a Jobs-Plus development when the initiative begins. It also probably will require drawing a sample at the same time from a comparison development for each site. During the followup period, it will be necessary to track these individuals wherever they go.

Earlier I presented a model of Jobs-Plus impacts which implies a series of initial, intermediate, and long-term outcomes. Exhibit 2.2 lists these measures, which are not repeated here. Instead, I consider aspects of these measures which determine the types of data collection methods they will require. In addition, I briefly indicate how different outcome measures will provide different types of information about changes over time and, consequently, will require different statistical analysis methods.

First consider measures of differences in levels or differences in states at two points in time, say, for example, at baseline and at the end of followup. This focus applies to outcomes such as residents’ attitudes about themselves, about welfare, and about work, their understanding of the incentives of Jobs-Plus, and whether or not they are employed, on welfare, or still living in public housing.

The emphasis here is on describing how the outcome differed at two points in time, not on identifying when it might have changed, or on describing fully its pattern of change over time (if it changed more than once). Hence, one can measure such outcomes at two specific times and summarize them as mean changes — a simple before-after analysis.

True longitudinal data are different. They provide continuous information over time about a level, a flow, or a state, and thereby make it possible to summarize a pattern over time, which describes both how an outcome changed and when it changed.

To analyze longitudinal data for continuous outcomes, such as earnings or welfare benefits, one can use conventional fixed-effect models for panel data, or extensions of these models, such as time-varying, fixed-effect models (Bloom, 1984).⁴⁴

To analyze such data for discrete outcomes, such as employed or not, on welfare or not, or living in public housing or not, requires statistical methods for "event history" analysis, such as "hazard rate" models (Allison, 1988).

Two questions arise here: (1) how long should the followup period be? and (2) how long should the baseline period be? Different answers to these questions will be necessary for different outcome measures obtained from different data sources with different time limitations.

The overall followup period for the project probably should be at least five years, because of the time that it will take to implement Jobs-Plus fully, and because of the likely long time required for the initiative to increase the economic self-sufficiency of public housing residents. For example, Rohe (1993) describes a seven-year process for participants in the Gateway Transitional Families Program sponsored by the Charlotte, North Carolina Housing Authority.⁴⁵

We also recommend that the baseline period for some of the study’s key outcome measures include four to five years before sample members are exposed to Jobs-Plus. Shorter baseline periods would enable researchers to control for differences in the initial conditions of sample members, but longer periods are required to control for differences in their underlying trends.

Where to Get the Data?

• sample surveys,
• PHA administrative records,
• UI wage records, and
• AFDC and food stamps records.

Sample surveys could be conducted at baseline and at one or more times thereafter. The first survey could measure the baseline characteristics listed earlier. It also could obtain retrospective longitudinal information about employment, earnings, welfare receipt, subsidized housing, and other key outcomes during the previous two-to-five years. The time period for this information would be limited by the ability of survey respondents to recall past events.⁴⁶

PHA administrative records, especially those based on the HUD Form 50058, which is used to certify a family’s eligibility for public housing when it applies, and is used each year thereafter to recertify eligibility and set family rents, provide information about selected socioeconomic characteristics. If these forms are completed properly, and if they are accessible, they could provide an annual "snapshot" of a household’s composition, its income by source, and its net income (minus allowances and expenses). Examination of the quality and availability of these data for sites that are selected to participate in Jobs-Plus will be necessary to determine whether or not they should be used.

As a source of longitudinal baseline data, the 50058 Form is limited by the period for which it is available, and the fact that it covers different periods for different residents.⁴⁷ Nevertheless, it might provide a good description of each household in the study at the time Jobs-Plus begins. As a source of longitudinal followup data, the 50058 Form is limited by the fact that it is not collected after a family leaves a PHA’s housing programs.⁴⁸

UI wage records contain data on earnings that are reported quarterly to each state by employers, as required by law, for all workers covered by Unemployment Insurance.⁴⁹ This information is reported for well over 90 percent of all employees in most states. It is used to determine eligibility for Unemployment Insurance and weekly benefit rates for persons who file a UI claim.

By matching sample members’ Social Security numbers to their UI wage records, one can measure their total quarterly earnings and their quarterly employment rates. This approach has been used by many studies of employment and training programs. As a source of longitudinal followup data, UI wage records are excellent, because they can be obtained for an indefinite period. Furthermore, they have been found to produce program impact estimates that are similar to those based on more costly followup survey data (Bloom et al., 1993).⁵⁰ As a source of pre-program longitudinal data, this information might be limited to four to six quarters, which is all that is usually kept in active records by state UI agencies because they only maintain active data for the period used to establish UI eligibility and weekly benefit rates. It might be possible, however, to increase the coverage of this information substantially, if state agencies agree to provide information from their archives.

AFDC and food stamps records can be obtained by matching the Social Security numbers of sample members to the administrative records of local and state welfare agencies. This information also has been used by past evaluations of employment and training programs. When available, it represents an effective way to produce an accurate long-term, monthly history.⁵¹

III. Jobs-Plus Impacts on Public Housing Developments

Introduction

This section explores ways to measure the impacts of Jobs-Plus on public housing developments. The goal here is to study what happens to a particular residential environment because of the initiative. I first present a model of how Jobs-Plus might affect a public housing development and corresponding measures of its success. I then consider research designs for measuring the impacts of Jobs-Plus.

Exhibit 3.1 presents a model of how Jobs-Plus might affect a public housing development. Just as the model for individual residents was an oversimplification, so is the model for public housing developments. Nevertheless, it provides a conceptual basis for specifying program outcome measures and a logical structure for organizing research findings.

The structure of the model is as follows. Jobs-Plus is designed to directly influence the characteristics of public housing residents and the prevailing culture of their development. Changes in resident characteristics, in turn, produce further changes in local culture, and vice versa. These changes increase the success of the development as a place to live and as a financial enterprise.

Changes in the resident characteristics of a public housing development can occur both because individuals change over time and because different households move out of and into the development. These changes are related to residents’ attitudes, human capital, labor market success, welfare dependence, mobility, and level of organization and interaction.

As residents begin to change, improvements in the local culture become possible. If Jobs-Plus is successful, we will begin to see the evolution of a safe, personally-supportive and work-oriented environment. Factors to consider in this regard, include the absence of crime, drug, and gang activities, and the presence of organizations that promote social capital, such as resident groups and civic organizations.⁵²

The final stage of the model represents the overall success of the public housing development, as a place to live, and as a financial enterprise. To the extent that the lives of individual residents are improved, and the prevailing culture becomes more positive, one should expect to see improved physical conditions, more favorable perceptions of the safety and desirability of the development, and improved financial conditions.

Exhibit 3.2 lists some measures of a public housing development’s overall success. Measures of changes in resident characteristics and changes in local culture were presented in Exhibit 2.2.

To estimate impacts from this design requires comparing the mean change in an outcome variable for Jobs-Plus developments to the mean change for comparison or control developments.⁵³ For example, to estimate impacts on vacancy rates, one would compute the mean change in vacancy rates for a Jobs-Plus sample, repeat this procedure for a comparison sample, and take the difference. A standard t test can assess the statistical significance of this difference.

The primary strengths of this approach are its simplicity and feasibility. Its main weaknesses are:

• the Jobs-Plus developments and the comparison developments might be exposed to different events which affect their outcomes differently (they might experience different histories),
• the margin for random sampling error will be high (the statistical power of program impact estimates will be low) because of the small number of developments in the sample,
• Jobs-Plus developments and comparison developments might have a different average underlying trend (they might be maturing differently),
• random factors might affect the baseline conditions of the two groups differently (they might exhibit different regression artifacts).

To address the first problem requires selecting Jobs-Plus developments and comparison developments that are as likely as possible to experience similar events in the near future. As discussed in Section 2, they probably should be selected from the same PHAs, and one should make an attempt to match their past histories, resident characteristics, and political situations. Furthermore, it will be crucial to document events which transpire in each development to determine whether different events affected their future outcomes in important ways.

Aside from choosing more project sites or choosing more developments from each participating site, there is little that can be done to increase the statistical power of impact estimates for Jobs-Plus. Nevertheless, if these impacts are as large as those documented by past rigorous studies of key Jobs-Plus components, impact estimates for Jobs-Plus could be statistically significant.⁵⁴

The best way to address the last two problems listed above is to use longitudinal data. Designs of this type, when applied to aggregate units, such as public housing developments, are referred to in the evaluation research literature as "interrupted time-series" analyses.⁵⁵

The Approach

The simplest interrupted time-series analysis involves a single site with multiple years of data before an intervention and multiple years of data during its followup period. This analysis can be applied to outcome measures for public housing, such as vacancy rates, the percentage of households on welfare, total rents paid, the percentage of households with a working adult, and so on. Exhibit 3.3 illustrates how to use the analysis to estimate the impact of Jobs-Plus on the percentage of households in a development who are on welfare.

If four or five years of baseline data on this measure can be obtained, then one can fit a pre-Jobs-Plus trend line.⁵⁶ A linear trend probably will be adequate for most outcomes, but a curvilinear trend (discussed later) can be estimated if the curvature of the baseline trend is pronounced. Extrapolation (extension) of the baseline trend provides the best available estimate of what the outcome would have been without Jobs-Plus (the counterfactual).⁵⁷

The deviation from the baseline trend in the first year after Jobs-Plus begins (line D₁ in the exhibit) provides an estimate of the impact of Jobs-Plus for that year. Deviations from trend in subsequent years (lines D₂ and D₃ ) provide impact estimates for these years. Estimating the impact for each followup year provides an easy way to describe the pattern of the impact over time (whether it is constant, it decays, or it grows). As a way to summarize this information, one can then average these estimates to determine the mean annual impact during the followup period.⁵⁸

For example, the hypothetical results in the bottom panel of Exhibit 3.3 indicate that there was a 3 point reduction in the percentage of residents who were on welfare in the first year after Jobs-Plus began, an 11 point reduction in the second year, and a 10 point reduction in the third year. The outcome was about 9 percentage points lower than predicted, on average, during the followup period.

The following regression model can be used to estimate the impacts in the exhibit:

where:

Y_t = the value of the outcome variable in year t,

T_t = one if year t is after Jobs-Plus began and zero otherwise,

t_t = the value for year t of the annual counter,

B ₀ = the impact of Jobs-Plus (the deviation from trend),

B ₁ = the slope of the baseline trend,

a = the intercept of the baseline trend.

If only years zero through six are included in the analysis, the coefficient, B ₀, equals the deviation from trend in year six (line D₁ in the exhibit) and the t statistic for this coefficient provides a test of its statistical significance. To include all three followup years in the analysis, and allow each to have a separate deviation from trend, one can replace T with a separate dummy variable for each year. The coefficient for each dummy variable equals the deviation from trend for the year that it represents (lines D₁, D₂, and D₃), and the t statistic for each coefficient provides a test of its statistical significance. If years zero through eight are included in the analysis, and one dummy variable is used to represent all followup years, then its coefficient is the mean deviation from trend for years six through 8 (9 percentage points in the example above).

For an interrupted time-series analysis to be effective there must be a stable baseline trend and a pronounced deviation from this trend during the followup period. The more stable the baseline trend is (the less the points vary around the trend-line), the more confidence one can place in the forecast of the counterfactual for the followup period. The larger and more abrupt the post-Jobs-Plus deviation from trend is, the easier it will be to identify this deviation statistically.

Thus, for public housing characteristics that do not change appreciably before Jobs-Plus, or that change gradually over time, there might be enough statistical power to produce meaningful estimates of Jobs-Plus impacts. For outcomes that change erratically before Jobs-Plus, the statistical power of impact estimates probably will be limited.

Because we anticipate the impacts of Jobs-Plus to be large, we are hopeful that an interrupted time-series analysis can identify them. However, because these impacts might appear gradually over several years, we are somewhat concerned that it will be difficult to identify them with confidence. On balance, then, we are cautiously optimistic.

A separate interrupted time-series analysis could be conducted for each outcome measure for each Jobs-Plus development. One then could pool findings for a specific outcome measure across developments by taking the mean of their impact estimates and computing the standard error of this mean.⁵⁹ Pooling these findings across sites will increase their statistical power.

One could account for non-linear trends, if the need arises, by replacing the year counter (t) with its logarithm, by replacing the outcome measure (Y) with its logarithm, or both. This will allow for curvature without reducing the number of degrees of freedom.

Impact Study Design #3: Interrupted Time-Series Analysis
for Jobs-Plus Developments and Comparison Developments

A logical extension of the preceding analysis is to construct a separate interrupted time-series for a comparison development where Jobs-Plus is not implemented. Exhibit 3.4 illustrates how this time-series can be used to improve estimates of the impacts of Jobs-Plus. The approach is applicable regardless of how the comparison development was chosen (whether random assignment was used or not, and whether matching was used or not).

The top panel of Exhibit 3.4 repeats the time-series for the hypothetical Jobs-Plus development in Exhibit 3.3. The bottom panel presents findings during the same period for a comparison development. The interrupted time-series analysis for the comparison development yields deviations from trend in years 6, 7 and 8 equal to E₁, E₂, and E₃, respectively. If the comparison development and the Jobs-Plus development were chosen from the same local environment, then the comparison development deviation from its trend provides an estimate of what the deviation would have been for the Jobs-Plus development without Jobs-Plus (the counterfactual).

Hence, D_t - E_t provides an estimate of the impact of Jobs-Plus in year t. The variance of this difference equals the sum of the variances of D_t and E_t, so one can readily test the statistical significance of the difference.⁶⁰

It is useful to proceed with this analysis in two steps. First, compute and test the deviation from trend, D_t, for the Jobs-Plus development. This step asks the question: "Was there a statistically significant improvement in the outcome for the Jobs-Plus development?"⁶¹ Next compute and test the difference between the two deviations from trend, D_t - E_t,. This step asks the question: "Did the outcome improve by an amount that was statistically significantly greater than its likely change without Jobs-Plus?"⁶²

There are many ways to pool these findings for a sample of Jobs-Plus developments and comparison developments. The simplest way to do so is to subtract the mean deviation for the comparison developments from the mean deviation for the Jobs-Plus developments and compute the standard error for this difference of means.⁶³

One important variation on the interrupted time-series theme occurs when only two or three baseline observations exist. This will apply to Jobs-Plus outcomes that have limited consistent past data because: (1) their collection began only recently, (2) the way they are collected changed recently, or (3) they are only kept for short periods. Two or three baseline observations are better than only one (for before-after analyses), and are less effective than more baseline observations (for interrupted time-series analyses). But how should this information be used?

A conservative approach is to assume that the underlying system is not changing rapidly over time. Under this assumption, one would use the mean value of the baseline outcome to forecast the future outcome without Jobs-Plus (the counterfactual). The observed difference between the actual future outcome and this forecast provides an estimate of the impact of Jobs-Plus. Comparing this estimated difference with the corresponding difference for a comparison development could further strengthen the finding. Pooling these findings across developments could strengthen the findings even further. In all cases, simple tests for statistical significance are possible.

A less conservative approach is to assume that the observed annual change in the outcome for the baseline period will continue into the future.⁶⁴ Hence, this annual change rate becomes the counterfactual. One then could compare the annualized change rate for the baseline period with the annualized change rate for the followup period.

Instead of annual data for an outcome, it might be possible to obtain information for shorter units of time — for example, monthly crime rates. Data for shorter units of time provide more observations, which in turn, make it possible to use more sophisticated time-series models to estimate program impacts (see McCain and McCleary, 1979, and McDowall, McCleary, Meidinger, and Hay, 1980).⁶⁵

Such models can enable researchers to accommodate the serial correlation of observations over time and account explicitly for the effects of seasonality. There are probably very few Jobs-Plus outcomes for which such data are available, however. In addition, the ability of these data to represent long-term trends is not necessarily greater than the ability of annual data to do so. Furthermore, data for shorter time periods have a larger relative variance and greater serial correlation.⁶⁶ Hence, procedures based on such data probably will have limited application to the Jobs-Plus impact study. Nevertheless, they will be considered.

IV. Jobs-Plus Impacts on the Community

Introduction

This section explores ways to measure the impacts of Jobs-Plus on the surrounding community of a public housing development. The focus here is on changes in an area over time caused by changes in people and firms who remain in the area plus differences in the people and firms who move in and out of the area.

I first present a simple model of how Jobs-Plus might affect a local community and identify some outcome measures that could be used to examine these effects. I then briefly discuss issues that must be addressed when trying to identify the relevant community for a public housing development. I next examine approaches for estimating Jobs-Plus impacts. Because of the elusive and indirect nature of these impacts, and because of the limited data available for estimating them, the approaches available for this purpose are less well-structured than those discussed in earlier sections of this paper.

A Model of Jobs-Plus Impacts and Measures of Community Outcomes

Exhibit 4.1 presents a simple model of how Jobs-Plus might affect a community that is located near a public housing development where the initiative is implemented. It begins with the three basic elements of the Jobs-Plus program, which are expected to produce positive changes in the lives of public housing residents and improvements in the living conditions at their developments. As the environment in the development improves, it can start to become a "positive anchor tenant" for the surrounding community. Much like a major department store draws other establishments to a shopping center, a newly-transformed public housing development could attract residential and commercial development to its surrounding community. Similarly, it might stimulate physical improvements, increased public services, healthier and more stable social conditions, increased civic engagement, and increased community empowerment. Exhibit 4.2 lists a number of possible outcome measures that could be used to test for community improvements.

It should be noted, however, that while it is possible for improved conditions at a public housing development to spill over into its surrounding community, and thereby leverage improvements to a wider area, these spillover effects are indirect and secondary, relative to the direct effects of Jobs-Plus on public housing residents and the developments in which they live. In addition, these effects might take a long time to materialize, given the many forces that must converge in order for them to occur. Hence, of all the impacts hypothesized for Jobs-Plus, community effects are the most speculative. This is especially true given the many outside forces — such as macroeconomic conditions, racial conflict, government funding decisions, and other public policies — that play a large role in determining the condition of an urban community.⁶⁷

Defining the Appropriate Community

A community usually refers to a social, rather than a geographic, unit.⁶⁸ However, to assess the impacts of Jobs-Plus will require defining local communities in spatially distinct terms. To do so, will present a major challenge, because there is little agreement about what constitutes a community.⁶⁹ Nevertheless, four factors should be considered when trying to delineate community boundaries:

• physical barriers, such as rivers, highways, parks, railroad tracks, and large-scale industrial, commercial or residential development;
• social interactions, such as shopping patterns, visiting patterns, church attendance, and recreational patterns;
• political units, such as voting wards or districts, and school districts;
• statistical units for which data are reported, such as census blocks or tracts, police precincts, neighborhood planning districts, and zip codes.

Local residents should be involved in defining community boundaries. Their input will help to ensure that the relationship between the public housing development and the residential, commercial, and industrial development adjacent to it is considered when the analytic boundaries of a development’s community are established. The key here is to designate an area to which the public housing development is, or could be, genuinely related when the barriers which tend to isolate public housing are removed. If the designated area is too small, however, estimates of the impacts of Jobs-Plus on it might omit some important spillover effects. If the area is too large, estimates also will understate the true impacts of Jobs-Plus, because they will be diffused too widely to be identifiable. In the end, pragmatic considerations of data availability will play a major role in determining community boundaries for analysis purposes. One important issue in this regard is the fact that boundaries for reporting areas shift over time; hence, the comparability of such data should be examined very carefully.

Issues to Consider When Measuring Community Impacts

Before discussing specific approaches for estimating the community impacts of Jobs-Plus, it is important to acknowledge some key issues that will arise when attempting to measure these impacts.

First, it probably will be necessary to use different methods to measure impacts on different outcomes. Some outcomes will be measurable through quantitative methods, and some will be measurable through qualitative methods.

Second, due to limited data and resources, it might be necessary to restrict certain impact analyses to a few Jobs-Plus sites, and limit their generalizability accordingly.

Third, because of the indirect, diffuse, and somewhat speculative nature of the potential community impacts of Jobs-Plus, resources probably should be limited to measuring only those impacts judged most likely to occur, and most likely to be observable. In particular, the primary emphasis of the impact analysis for Jobs-Plus should be on its direct effects on public housing residents and public housing development residents, discussed earlier.

Fourth, an explicit emphasis should be placed on "triangulating" impact estimates.