This is one in a series of 15 two-page, evidence-based framing memos on pressing education and social issues prepared by MDRC for the incoming Obama Administration and the new Congress.

Bottom Line

At a time when limited government resources demand that the nation make the most of investments in social and education programs, policymakers need credible information identifying effective strategies for addressing serious national problems, such as poor elementary school reading and math performance, low skill levels of displaced workers, and high recidivism rates among released prisoners. To understand which policies actually work, a scientific approach to evaluating programs has gained greater — but still too limited — currency. By comparing individuals who receive a program’s services to others who are similar in every way except that they do not receive the services, researchers can determine the real difference a program makes — over and above what otherwise would have happened.

Increasingly, to establish two identical groups of people, researchers have used a rigorous process similar to that used in medical trials: random assignment. In addition to their credibility, results from random assignment studies have a major advantage over other approaches: they are easy to understand and far less susceptible to statistical or ideological manipulation. Other rigorous methods that create credible comparison groups can also be used when the circumstances are right. In brief, while rigorous research is being used in some policy areas, the nation can do much better in building a reliable evidence base about what does and does not work.

What Do We Know?

There is an increasingly strong consensus in the research community, supported by the National Research Council of the National Academy of Sciences and the Council for Excellence in Government, that scientifically rigorous methods — especially random assignment — should be used to gather credible information about whether a program strategy works or not. This approach also:

• Quantifies the value of a program — for example, whether it increased high school graduation by 3 percent or 20 percent, or raised employment by 5 percent or 25 percent.
  
  
• Enables researchers to identify the population subgroups for whom the program works best.
  
  
• Provides the basis for accurate benefit-cost estimates.
  
  
• Partners with strong “implementation” research to help answer how and why programs work, information that is key to future replication.

• Type A “Demonstration” Studies: What program strategies succeed in addressing a social problem? Sometimes policymakers want to identify which responses to a social problem should be replicated because they are effective and which should be dropped because they are ineffective. For example, this approach can learn whether guaranteed transitional jobs for recently released prisoners help reintegrate them into society.
  
  
• Type B “National Average” Studies: Does a national social program work “overall”? At other times, policymakers may want to know whether an existing national program, on average, is effective in addressing a problem. Type B studies do not aim to identify effective strategies or approaches but rather to give an overall “grade” to the entire program. The national studies of Head Start and Jobs Corps are two good examples: they did not aim to identify the best ways to operate either program; they showed whether, on average, the programs made a difference.

In addition to random assignment, other methods can be use to create credible comparison groups. For example, when firm rules determine who is eligible for program services, those just above and just below the eligibility threshold should be similar except that one is eligible for program services (a method referred to as a regression discontinuity design). In natural experiments, a comparison group is created for reasons unrelated to research, such as when school systems use lotteries to assign children to different public schools.

Random assignment is not always appropriate and feasible, particularly when the program is an entitlement (such as unemployment insurance) or is meant to change entire communities or cities.

How Widely are Rigorous Research Methods Used at the Federal Level?

A rigorous body of evidence is being built in some agencies but not others. The Department of Health and Human Services has an impressive group of studies underway and a rich history of supporting rigorous research on welfare policy, but not on medical insurance programs; the Department of Education’s Institute of Educational Sciences has begun an ambitious effort to build knowledge in the K-12 reform area, albeit with some growing pains. Other agencies are also beginning to support or are exploring support for systematic knowledge-building. In addition, many pieces of federal legislation require or strongly recommend the use of random assignment to evaluate programs (e.g., federal welfare reform, Second Chance Act, Higher Education Act). Importantly, the Office of Management and Budget, has shown an increasingly strong commitment to random assignment studies, particularly Type A Demonstration studies. More is needed, however, especially among the nation’s highest-cost programs, where maximizing program effectiveness is most critical.

What’s Next?

• Those federal agencies that have a strong commitment to rigorous evaluations should be encouraged to continue in their commitment and provided the resources to do so.
  
  
• In other social program areas where rigorous methods have not been as widely used — particularly those that have serious long-term budget implications for the federal government, such as Medicaid, broader health care reform, and prison and prisoner reentry reform — rigorous studies could provide critical and credible information to enable policymakers to make informed decisions.
  
  
• Policymakers could review the research plans of federal agencies to ensure that they are addressing the questions of greatest importance and that they are likely to provide answers that are reliable and credible.

Key References

Bloom, Howard (Ed.). 2005. Learning More From Social Experiments: Evolving Analytic Approaches. New York: Russell Sage Foundation.

Committee to Evaluate the Research Plan of the U.S. Department of Housing and Urban Development. 2008. Rebuilding the Research Capacity at HUD. Washington, DC: National Research Council, National Academy of Sciences.

The White House Task Force for Disadvantaged Youth. October 2003. Final Report. Washington, DC.