The evaluation was conducted in 22 middle schools in five large, urban school districts. The evaluation included two impact studies:

1. A study of the effects of the WSWC model for all students based on a quasi-experimental comparative interrupted time series design. The analysis included two groups of schools: 22 middle schools that were implementing the WSWC model (program schools) and a group of similar, matched schools in the same districts that did not implement the model (comparison schools). The analysis looks at the outcomes of consecutive cohorts (entering classes) of sixth-grade students who were enrolled in the study schools in the years before (pre-intervention) and after (post-intervention) the WSWC model was implemented. Trends over time in the program and comparison schools were compared to determine (1) whether on average, students who were enrolled in the program schools while the WSWC model was being implemented had better outcomes than expected relative to the outcomes of pre-intervention cohorts of students, and (2) whether changes over time in student outcomes were larger and more positive in the program schools than in the comparison schools. The quasi-experimental study focuses on the impact of the WSWC model before the COVID-19 pandemic using student records data collected through the 2018-2019 school year. (The study was supposed to include data from the 2019-2020 school year as well, but student records were not available for this school year because of the COVID-19 pandemic.)
2. A study of the effects of Tier 2 services for eligible students based on a student-level randomized controlled trial. In each of the 22 program schools, sixth-grade students who were considered to be at increased risk of not graduating from high school were identified by City Year or the school. A lottery-like process was used to select randomly which of these students were offered Tier 2 services (the program group), and which students were not (the control group). This random assignment process was intended to create two groups of students (the program group and the control group) who were equivalent with respect to all their characteristics except for the offer of Tier 2 services. Random assignment was conducted in the fall of 2018 separately by school and by students’ area of need (ELA, math, behavior, attendance, or a combination). The effect of Tier 2 services for eligible students was evaluated by comparing the outcomes of students in the program group and the control group. The experimental study focused on the impact of Tier 2 services before the COVID-19 pandemic on social-emotional skills, academic outcomes, and suspensions during the 2018-2019 and 2019-2020 school years. Due to the COVID-19 pandemic, data from the 2019-2020 school year were limited to the first half of the year before schools closed, and did not include test scores. In addition, a second cohort of sixth-grade students who were randomly assigned in the fall of 2019 was subsequently dropped from the study because those students received less than a full year of services. All students in the Tier 2 study continued to receive the Tier 1 services, so the findings represent the effect of Tier 2 services over and above the effect of Tier 1 services.

When considered together, the findings from these two studies can provide a better understanding of the effects of the WSWC model than the studies could if conducted in isolation. Both studies examined the effects of the WSWC model and Tier 2 services in three domains: academic achievement (in ELA and math), school engagement (specifically, with respect to attendance and disciplinary outcomes), and social and emotional skills. These outcomes were measured using student records data acquired from school districts participating in the study. For the Tier 2 study, a student survey was also used to measure students’ social and emotional skills. Finally, information from school and district websites was used to illuminate the types of services provided to students in schools not working with City Year. (A principal survey was also administered in comparison schools in the spring of 2020 to collect data about the student support services they offered, but response rates were very low due to the pandemic.)

To put the impact findings in context, the evaluation also examined the implementation of the WSWC model in the 2018-2019 school year and the first half the 2019-2020 school year, before schools closed. The implementation study examined how the components and activities of the WSWC model were implemented in the 22 program schools in these school years, and the amounts and types of services that students received, drawing on City Year staff and teacher surveys and program-monitoring data. MDRC also conducted visits to summer training sessions for City Year Student Success Coaches in the summer of 2019 to observe the training activities and to conduct focus groups, and visited selected schools in the winter of 2020 to speak with staff members and to observe Student Success Coaches as they provided classroom and whole-school services.

AIR led the study of Tier 2 services. MDRC led the quasi-experimental study of the WSWC model and the implementation study.