Putting Evidence at the Heart of Making Policy
MDRC is dedicated to learning what works to improve the well-being of low-income people. Through our research, we seek to enhance the effectiveness of social and education policies and programs. As part of our “Looking Forward” series, we provide policymakers with memos that suggest ways to make progress on critical issues.
Bottom line: Over the last decade and a half, during a period defined in the public’s consciousness by political partisanship, the legislative and executive branches have quietly forged a bipartisan consensus: both parties now agree that to ensure a high rate of return on the nation’s social programs, it is necessary to build evidence that they work. To take only the most recent examples of this consensus, Congress established the Commission on Evidence-Based Policymaking in 2016, and included evidence provisions in the Workforce Innovation and Opportunity Act of 2014 and the Every Student Succeeds Act of 2015. But more should be done to improve the nation’s research capabilities, to embed evidence building in government programs, and to put evidence at the heart of making policy.
To create a more fiscally responsible federal government, promote the independent evaluation of programs and policies. To make informed decisions about how to spend government resources, policymakers and practitioners need evaluation findings that are credible, relevant, accurate, and timely. This information can help them decide what programs to improve, what programs to expand, and what programs to cut. Such information is even more urgent in a time of severe budget constraints and fiscal austerity. To make sure that the information is of high quality, federally supported evaluations should adhere to certain principles: (1) They should be relevant to policy issues. (2) They must be credible to the evaluations’ subjects and consumers. (3) They should be independent of political and other undue external influences. Evaluations that uphold these principles can provide the information that policymakers and the public require.
Create a culture of continuous improvement for programs that show promise. Just like any business committed to becoming a dynamic learning organization, the federal government should develop incentives for using research evidence to make programs more effective over time. A good example is the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) program, the largest evidence-based program of recent years. Several elements of MIECHV are worth emphasizing:
Existing evidence was used to influence how states could spend federal funds, making it more likely that the funds would make a difference for families.
The legislation that created MIECHV recognized that in some ways home visiting was not as effective as desired, so it offered states funds to test innovative approaches.
Funds were set aside for research to make sure that learning continued under MIECHV and could influence future versions of home visiting.
Integrate low-cost testing of process improvements into government programs. MDRC’s Center for Applied Behavioral Science has teamed up with 29 agencies in 15 states on program redesigns that use strategies drawn from behavioral science, developing successful interventions that have improved program efficiency for less than $10 per participant. For example, a messaging strategy that cost less than $3 per person boosted parents’ attendance at important appointments related to child care subsidies by more than 10 percentage points.
Build on the tiered evidence strategies embodied most recently in the bipartisan Every Student Succeeds Act of 2015. Such strategies set clear guidelines about standards of evidence and provide incentives for developing innovative new programs. Perhaps more importantly, they also provide incentives for testing models with evidence of effectiveness on a larger scale.
Use evidence to direct existing funding streams. As the MIECHV example illustrates, when there is evidence of what works, policymakers should build incentives into current funding streams to make sure that dollars follow the evidence.
Use existing funding streams to generate evidence. While dedicated funds for innovation have been an important source of new ideas, using resources within major program funding streams would allow federal agencies to continuously improve existing programs. A 1 percent set-aside within these funding streams would both be an adequate investment and be cost-neutral.
Expand access to administrative data to make research more cost-efficient. In the course of administering their programs, government agencies collect enormous amounts of data they could use to track progress and to improve program performance. Yet federal and state agencies (and their contractors) cannot regularly access and share data for evaluation purposes. For example, the federal government maintains the National Directory of New Hires (NDNH) database, a critical source of data collected by states from employers on employment and earnings. However, federal research contractors are forced to get the very same data directly from states, at great cost to the federal government and at considerable inconvenience to the states, as they must report the data twice. If the NDNH database were made more widely available to evaluators (with appropriate privacy safeguards), Congress and federal agencies could assess social programs’ effects on jobs and earnings at much less cost and burden to the federal government and the states. In general, agencies and their contractors need to be given clear authority to access administrative data for evaluation purposes, whether those data are maintained at the federal level or at the state level.
Some questions cannot be answered with administrative data alone. Administrative records do not always include the data required to answer important questions. For example, MDRC’s original 13 welfare-to-work studies were based entirely on four things that could be measured using administrative records: employment, earnings, the number of people receiving welfare, and the amount of welfare they received. But those measures could not tell us about welfare reform’s effects on children, or whether very young children would be harmed if parents were required to find work. We needed to conduct surveys to answer this question.
Privacy and confidentiality can be protected while still allowing access for research. Congress is considering amending the Family Educational Rights and Privacy Act (FERPA) because of concerns over threats to the privacy of student data. The goal is admirable; unfortunately, education researchers from academia and other nonprofit institutions have gotten swept up in the furor. Under current federal law, education agencies can share data with researchers only for research projects to benefit students and improve instruction — and only under extremely strict privacy conditions. Some are suggesting that Congress should significantly scale back even that authority. Without access to student data, researchers could conduct little education research at all. The bottom line is that while it is essential to continue to protect the security and privacy of student data, those student data were originally collected for the purpose of assessing and improving education. Congress must not unintentionally end the analysis that would make that goal possible.
For more information, e-mail Therese Leung at [email protected] or call her at 212-340-4453.
* * *
This policy memo was supported by the Annie E. Casey Foundation.