Box 1

Why Impacts Are Better than Outcomes for 
Assessing Program Effectiveness

Program operators generally have information only on program outcomes, for instance, the employment and welfare exit rates among people who enrolled or participated in the program. Though these statistics are valuable, they may lead to misleading conclusions about which programs are the most effective.

The first column in the chart below shows the employment rates and average earnings levels in the second and fifth years of the NEWWS follow-up period for welfare recipients who enrolled in two NEWWS programs, the Portland program and the Grand Rapids LFA program. Given only the program outcomes shown in the first column, it would be reasonable to conclude that the Grand Rapids LFA program was more successful in getting welfare recipients into jobs but that the Portland program was somewhat more successful in raising people’s earnings.

When the experiences of the control groups in both sites (shown in the second column) are taken into account, however, the conclusions change. The program groups’ experiences are compared with the control groups’ experiences in the “Difference” column. These differences are the programs’ impacts on employment or earnings. (The asterisks indicate whether the impacts are statistically significant, that is, very unlikely to have arisen by chance. The more asterisks appear next to an impact, the less likely the impact is to be due to chance.) The “Percentage Change” column expresses the impacts as percentage increases or decreases relative to the control group levels.

The impact analysis reveals that the Portland program by far outperformed the Grand Rapids LFA program: Looking only at year 2, during which both programs were successful, the Portland program produced a 21 percent increase in employment and a 40 percent increase in earnings, compared with an 11 percent increase in employment and an 18 percent increase in earnings for the Grand Rapids LFA program.