City Year Evaluation

Project

Overview

Project Overview

Helping children and young adults develop their talents and strengths also helps them contribute to their communities in the future. Yet systemic inequities that disproportionately affect students of color and students from low-income households mean too many young people do not have such supportive learning environments. These inequities lead to disparities in educational, health-related, and economic outcomes — including high school graduation rates.

City Year partners with public schools with the goal of preparing students with the skills and mindsets they need to thrive and contribute to their communities. Diverse teams of City Year AmeriCorps members serve in schools full time to help students engage in their learning fully and stay on track to graduation. AmeriCorps members offer personalized support to students who need it most, and help to create a safe and supportive school climate where students can feel capable of and committed to their academic goals.

City Year has developed and widely implemented its Whole School Whole Child (WSWC) services. WSWC includes academic, social, and emotional services that are aimed at helping students develop the skills they need to navigate and complete high school. It has two tiers of services:

Universal (Tier 1) services are provided to all students in the school. They include after-school programs and school-wide events that recognize positive behavior and student success. They also include in-classroom support for selected English language arts and math teachers.
Targeted (Tier 2) services are provided to students who are at increased risk of not graduating based on early warning indicators related to attendance, behavior, and course performance in English language arts or math. Tier 2 services include tutoring in English language arts and math, social and emotional development and behavior support, and attendance coaching.

The American Institutes for Research (AIR) and MDRC are conducting an evaluation of WSWC services in 22 middle schools in five large urban school districts. The evaluation is focusing on middle school because it is a pivotal time when students form their identities and learn the foundational academic, social, and emotional skills that they need to succeed in high school and as adults. Therefore, building evidence on what works for this age group is an important pathway for improving graduation rates.

The evaluation includes two impact studies. The first study explores the effect of the WSWC model (Tier 1 and Tier 2 services) for all students, and the second study attempts to isolate the effect of Tier 2 services for students who are identified as being at heightened risk of dropping out of school. The evaluation also includes an implementation study and a cost study. The five-year evaluation began in fall 2017 and will be completed by the end of 2022. The research reported described here is supported by the Institute of Education Sciences in the U.S. Department of Education, through grant R305A170227 to AIR. The opinions expressed are those of the study team and do not represent views of the Institute or the U.S. Department of Education.

STATUS: Current

Basics

Issue: P-12 Education

Focus Area: Elementary and Middle School

Population: Children and Schools

Focus Population: Elementary and Middle School Students, Students Experiencing Academic Struggles

Locations: California, Florida, New York, Texas

Intervention: Student Services, Classroom Services, Case Management, Counseling/Advising

Agenda, Scope, <br> & Goals

Agenda, Scope, and Goals

The purpose of the evaluation is to provide rigorous evidence of the effects of the two-tiered WSWC model on the academic, social, and emotional outcomes of middle school students, and to provide information on the model’s implementation and costs. Specifically, there are four research goals:

To assess whether students enrolled in middle schools that provide WSWC services have better outcomes, on average, than students enrolled in similar schools that are not providing WSWC services.
To assess the value of Tier 2 services by measuring whether eligible students who receive these services have better outcomes than students in the same schools who do not receive them.
To determine how the different components of the WSWC model are implemented in schools, as well as the facilitators of and challenges to implementation.

To provide information on the costs of implementing the WSWC model.

STATUS: Current

Basics

Issue: P-12 Education

Focus Area: Elementary and Middle School

Population: Children and Schools

Focus Population: Elementary and Middle School Students, Students Experiencing Academic Struggles

Locations: California, Florida, New York, Texas

Intervention: Student Services, Classroom Services, Case Management, Counseling/Advising

Design, Sites, & <br> Data Sources

Design, Sites, and Data Sources

The evaluation is being conducted in five large, urban school districts. It focuses on 22 middle schools that have been implementing the WSWC model for at least five years. The evaluation includes two impact studies:

A study of the effects of the WSWC model for all students based on a quasi-experimental comparative interrupted time series design. The analysis includes two groups of schools: 22 middle schools that are implementing the WSWC model (program schools) and a group of similar, matched schools in the same districts that did not implement the model (comparison schools). The analysis looks at the outcomes of consecutive cohorts (entering classes) of sixth-grade students who were enrolled in the study schools in the years before (pre-intervention) and after (post-intervention) the WSWC model was implemented. Trends over time in the program and comparison schools are compared to determine (1) whether on average, students who were enrolled in the program schools while the WSWC model was being implemented had better outcomes than expected relative to the outcomes of pre-intervention cohorts of students, and (2) whether changes over time in student outcomes were larger and more positive in the program schools than in the comparison schools.
A study of the effects of Tier 2 services for eligible students based on a student-level randomized controlled trial. In each of the 22 program schools, sixth-grade students who were at increased risk of not graduating from high school were identified by City Year or the school. A lottery-like process was used to randomly select which of these students were offered Tier 2 services (the program group), and which students did not receive Tier 2 services (the control group). This random assignment process is intended to create two groups of students (the program group and the control group) who are equivalent with respect to all of their characteristics except for the offer of Tier 2 services. Random assignment was conducted separately by school and by students’ area of need (English language arts, math, behavior, or attendance). The effect of Tier 2 services for eligible students can be evaluated by comparing the outcomes of students in the program group and the control group. Two cohorts of eligible sixth-grade students are included in this study. All students in the Tier 2 study continue to receive the Tier 1 services, so the findings represent the effect of Tier 2 services over and above the effect of Tier 1 services.

When considered together, the findings from these two studies can provide a better understanding of the effects of the WSWC model than the studies could if conducted in isolation. Both studies examine the effects of the WSWC model and Tier 2 services in three domains: academic achievement (English language arts and math), school engagement (attendance and disciplinary outcomes), and social and emotional learning. These outcomes are measured using student records data acquired from school districts participating in the study. For the Tier 2 study, a student survey is also being used to measure students’ social and emotional skills. Finally, comparison school principals are being surveyed to record what types of services are provided to students in schools not working with City Year.

To put the impact findings in context, the evaluation examines the implementation and cost of the program:

The implementation study examines how the components and activities of the WSWC model are being implemented in the 22 program schools, and the amounts and types of services that students receive. To do so it draws on City Year staff surveys and program monitoring data. MDRC conducted field visits to summer training sessions for AmeriCorps members in summer 2019 to observe the training activities and to conduct focus groups. MDRC is also visiting selected schools in spring 2020 to speak with staff members and to observe AmeriCorps members as they provide classroom and whole-school services.
The cost study provides information on the cost per student of implementing the WSWC model. It draws on budget data, interviews with staff members, and publicly available cost databases and indices.

AIR is leading the study of Tier 2 services and the cost study. MDRC is leading the study of the effects of the WSWC model for all students and the implementation study.

STATUS: Current

Basics

Issue: P-12 Education

Focus Area: Elementary and Middle School

Population: Children and Schools

Focus Population: Elementary and Middle School Students, Students Experiencing Academic Struggles

Locations: California, Florida, New York, Texas

Intervention: Student Services, Classroom Services, Case Management, Counseling/Advising

Funders & <br />Partners

Funders

U.S. Department of Education

Partners

American Institutes for Research

City Year

MDRC Staff

Marie-Andrée Somers
Senior Associate, K-12 Education
Somers is a senior research associate who specializes in quantitative impact evaluation in the field of education. She has experience and expertise using randomized experiments (student-level and school-level) and comparative interrupted time-series designs to evaluate the effect of educational interventions on student and teacher outcomes.
More »
Zeest Haider
Research Associate, K-12 Education
Haider has worked as research analyst doing both quantitative and qualitative research on a variety of projects at MDRC. She serves as a data manager, leading the work of data processing and analysis with a team of programmers.
More »
Margaret Hennessy
Research Analyst, K-12 Education
Hennessy has worked as a project manager and qualitative researcher on several projects since joining MDRC in 2018.
More »