Addressing Early Warning Indicators
Interim Impact Findings from the Investing in Innovation (i3) Evaluation of Diplomas Now
Too many students in high-poverty urban communities drop out of high school, and too few graduate prepared for college and careers. Three national organizations — Talent Development Secondary, City Year, and Communities In Schools — partnered to form Diplomas Now in an effort to turn those numbers around. Supported by funds from a U.S. Department of Education 2010 Investing in Innovation (i3) validation grant and private sources, Diplomas Now teams have been implementing their data-driven, tiered intervention model in urban secondary schools across the nation. The model combines a comprehensive school reform strategy, intended to transform the academic experience of all students, with more targeted interventions for students who have “early warning indicators” related to attendance, behavior, and course performance. By identifying students at risk of dropping out and providing individual support, Diplomas Now attempts to get struggling students back on a stable trajectory toward their diplomas.
MDRC and ICF International are conducting an independent, experimental evaluation of the impact and implementation of Diplomas Now. Sixty-two secondary schools in 11 school districts agreed to participate in this study between 2011 and 2013. Thirty-two of these schools were randomly assigned to implement the Diplomas Now model while the other 30 schools were assigned to a control group, continuing their existing school programs or implementing other reform strategies of their choosing. Two prior evaluation reports focused on the first two years of Diplomas Now implementation. This third report shares interim impact findings for those years, paying particular attention to attendance, behavior, and course performance outcomes of students in sixth or ninth grade, their transition year into middle or high school — the first-year impacts of a multiyear program.
The Diplomas Now model produced a positive and statistically significant impact on the percentage of students with no early warning indicators — students with better than 85 percent attendance, fewer than three days suspended or expelled, and passing grades in both English/language arts and math. Helping students maintain or reach these thresholds is an explicit target of Diplomas Now school teams.
Diplomas Now did not have a statistically significant impact on the percentage of students meeting a more stringent threshold suggestive of a more stable educational trajectory: better than 90 percent attendance, no suspensions or expulsions, and passing all four core subject areas of English/language arts, math, social studies, and science.
Diplomas Now did not produce a significant impact on average attendance, discipline, and course passing rates in sixth and ninth grades compared with rates at schools that did not implement the model. These outcomes improved from baseline in both Diplomas Now and comparison schools.
There were more promising impacts for middle schools than for high schools. In middle schools, Diplomas Now had a positive, statistically significant impact on the percentage of sixth-graders with no early warning indicators. There were no significant impacts, positive or negative, on the attendance, behavior, and course performance outcomes of ninth-graders.
- Students at Diplomas Now schools reported participating in more academically focused after-school activities, and more reported having a positive relationship with an adult at school who is not a teacher, than their peers in the comparison schools. Students in both groups of schools reported similar perceptions of school safety and climate, and the Diplomas Now model did not have an effect on students’ self-perceptions or school behaviors.