|
This is the first of two published reports for MDRC's evaluation of the Bay Area School Reform Collaborative (BASRC), a grant-making organization in San Francisco, California.1 BASRC is dedicated to improving student achievement in public schools and narrowing achievement gaps between different racial, ethnic, and socioeconomic groups. BASRC does not prescribe particular instructional practices, curricula, or school structures as "treatment." Rather, the organization promotes a vision of culture change at every level of the education system, in which teachers and administrators engage in a continuous improvement process driven by collective inquiry into student learning outcomes, school functioning, and teacher practice. This process-oriented reform strategy is designed to build professional knowledge of effective practice, mutual accountability, and collaboration. BASRC hypothesizes that such changes in culture will translate into improved teaching and learning. However, BASRC's reform strategy does not specify the particular changes in teaching practice that should evolve as a result of these processes.
This evaluation centers on a specific BASRC initiative, the "focal strategy," which was implemented in five Bay Area districts (the "focal districts") and two to four selected schools within those districts (the "focal schools") during the 2002-2003 school year. (During the 2003-2004 school year, BASRC added a sixth focal district that is not included in MDRC's study.) Building on earlier phases of BASRC reforms, which began in 1996, the focal strategy is meant to increase the intensity of the core BASRC reforms by creating more opportunities for district and school administrators to interact with BASRC staff. Thus, compared with the earlier reform strategies, the focal strategy serves as a stronger test of BASRC reform ideas in fewer places. Also, in selecting the focal districts, BASRC tended to focus on districts where there already was a strong working relationship developed during earlier BASRC efforts. Therefore, in many cases, focal reforms have essentially been implemented on top of existing BASRC reform work.
The BASRC focal strategy has three main features:
- coaching of superintendents, district and school leaders, and teachers
- evidence-based decision-making at all levels of the system
- networks and collaboration among administrators and teachers, within and across districts and schools
All three features were also part of BASRC's earlier reform efforts. Once the focal strategy began, school-level coaching by BASRC staff was redirected toward focal districts. In addition, "executive coaches" (former superintendents) were hired to advise and support the leaders in the focal districts. Along with regular meetings on comprehensive needs assessments and to review progress, these efforts were intended to reinforce all elements of the strategy in the focal districts and schools.
The theory of action underlying the BASRC focal reform strategy posits that the implementation of these three design features will yield changes in district, school, and classroom practices that will in turn improve student outcomes. Together, coaching, evidence-based decision-making, and networks and collaboration are thought to increase districts' and schools' engagement in a continuous improvement process, focused on improving the level and equity of student achievement levels. Because the strategy is process-oriented, it can result in different outcomes within each district, school, and classroom. The outcomes may or may not be policies or practices that are directly linked with specific strategies for improved teaching and learning. They may be incremental improvements in culture that eventually foster better teaching and learning.
The Evaluation of the BASRC Focal Strategy
The independent evaluation of BASRC is funded by the William and Flora Hewlett Foundation and includes studies by both MDRC and the Stanford University Center for Research on the Context of Teaching (CRC).2 The CRC studies focus on the process of reform as it relates to BASRC in general and the relationship between various BASRC reform efforts and changes at the district, school, and classroom levels. MDRC's study attempts to shed light on the relationship between the BASRC focal strategy and improvements in student achievement by investigating the following:
- The relationship between participation in the BASRC focal strategy and student achievement.
- The relationship between implementation of specific BASRC reform practices and changes in student achievement.
This report focuses on the first of these issues, that is, documenting the empirical relationship between BASRC participation and changes in student achievement. As such, it addresses three questions:
- What is the BASRC focal strategy reform model/theory of action?
- To what extent was this model implemented at the districts involved in this evaluation?
- What is the association between districts' participation in BASRC focal reforms and measured changes in average student achievement?
While MDRC's study will reflect on the relationship between the BASRC focal strategy and student achievement, it cannot identify a causal relationship. Due to the nature of the BASRC focal strategy, and the selection process for districts' participation in the strategy, it is not feasible to render a reliable, unbiased single "net impact" estimate summarizing BASRC's effect on student learning. The analysis presented in this report compares progress in the focal districts in the first two years of the focal reforms to progress in a set of carefully chosen comparison districts from throughout the San Francisco Bay Area over the same period of time. BASRC selected districts based on a variety of immeasurable factors, including the extent to which the superintendent was reform-minded. Therefore, any differences in progress in the focal districts versus that in other districts cannot necessarily be attributed to the BASRC focal strategy.
The BASRC focal reforms tended to be focused on literacy instruction and concentrated at the elementary level. As such, all analyses are focused at the elementary school level and emphasize district performance on measures of student literacy.
MDRC's next report, scheduled for later in 2006, will explore BASRC's theory of action by examining the relationship between changes in student achievement and schools' implementation of the practices the reform strategy was designed to encourage. In other words, regardless of why schools experience changes in reform practices, MDRC's next report will attempt to ascertain whether those reform practices are correlated with differences in students' learning.
Key Findings
Implementation Findings
- By Year 2, the school-level aspect of the focal strategy faded. Thus, the model evolved to be almost entirely a district strategy in which focal districts received district-level coaching from an executive coach, some additional coaching, and review meetings with BASRC staff.
In theory, coaching by BASRC staff was to be a primary feature of the school-level focal strategy. This "external" coaching effort was distinct from the other "internal" coaching efforts in place in non-focal schools (including school-level literacy coaches and coaches employed by the district to support reform work). However, in the first year of implementation, BASRC encountered several complications, including resistance to the BASRC coaches from school-level staff. By the end of the first year, BASRC coaches did not have a significant coaching role in most of the focal schools. At the district level, on the other hand, the key components of the focal strategy were in place by Year 2. BASRC executive coaches met with district leaders on a regular basis, and district staff attended networking and review meetings led by BASRC staff.
- The extent to which the BASRC focal strategy at the district level translated into specific reform activities in these districts is not yet clear.
The coaching delivered by BASRC executive coaches and other BASRC staff was not necessarily focused on implementation of the core BASRC concepts. Instead, in practice, coaching often revolved around a variety of needs of the superintendent or other district staff. This varied from improving the superintendent's communication skills to advising on the implementation of a new districtwide curriculum. The extent to which coaching or meetings with BASRC staff translated into specific reform activities in these districts is not yet clear. Gathering evidence on activities resulting from participation in the BASRC focal strategy is a priority for future MDRC field work.
- Although there was evidence of all three key features of the BASRC focal strategy in schools in the BASRC districts, these instances of the key features were likely vestiges of earlier BASRC reform phases. In addition, it was difficult to detect meaningful differences in the types of BASRC supports and reform activities in place in focal schools compared with non-focal schools.
MDRC found evidence of all three of the key features of the BASRC focal reform strategy — coaching by district or school staff, evidence-based decision-making, and networks and collaboration — in place at the schools in the focal districts. However, field research data indicate that these were typically implemented as a result of participation in earlier BASRC reform efforts rather than in the BASRC focal strategy itself. Moreover, it is also possible that these reform practices were in place before any participation in BASRC reform efforts. In general, in the schools visited by MDRC, it was difficult to detect meaningful differences in the types of BASRC supports and reform activities in place in focal schools compared with non-focal schools.
Student Achievement Outcomes
In order to explore the relationship between the BASRC focal strategy and student achievement, MDRC's analysis of student achievement compares progress in the BASRC focal districts in the first two years of the BASRC focal reforms to progress in a set of carefully chosen comparison districts from throughout the San Francisco Bay Area over the same period.
- In the years following implementation of the BASRC focal strategy, third-grade students in the BASRC focal districts and in similar districts throughout the Bay Area showed no progress in student achievement compared with the baseline period. In fact, average proficiency rates declined during this timeframe.
Neither BASRC focal districts nor their comparison counterparts experienced progress in third-grade proficiency rates on the California Standards Tests (CSTs). As shown in the top panel of Figure ES.1, in the two years preceding the BASRC focal strategy, an average of 43 percent of third-grade students in the BASRC focal districts scored proficient or above on the language arts portion of the CST. By the end of the second year of implementation, this dropped to 37 percent. Over the same period, average proficiency rates in similar districts throughout the Bay Area and across the state dropped slightly as well. Also, as shown in the lower panel of Figure ES.1, the percentage of third-grade students scoring below basic remained relatively constant in the comparison districts and increased slightly in the focal districts.
- On the other hand, fifth-grade students' performance in the focal districts improved over time, slightly outpacing improvements in the comparison districts, but the differences were not statistically significant.
As shown in the top panel of Figure ES.2, in the years immediately preceding implementation of the focal strategy, proficiency rates on the CST language arts test averaged 38 percent among fifth-graders in BASRC focal districts. This rate increased to 51 percent by the end of the second year of focal strategy implementation. At the same time, proficiency levels in the comparison districts increased from 39 percent to 50 percent. Although the increases were slightly larger in the BASRC focal districts, these differences were modest in size and not statistically significant. As shown in the lower panel of Figure ES.2, there was also a reduction in the percentage of fifth-grade students performing below basic. Again, although these reductions were slightly larger in the BASRC focal districts, the differences were not statistically significant. Rather than reflecting systematic differences between progress in the focal districts and progress in the non-focal districts, they may reflect chance or "random" fluctuations in student outcomes.
- Among blacks and Hispanics, English Language Learners (ELL), and economically disadvantaged students, reductions in the percent of fifth-grade students performing below basic levels in the BASRC focal districts in Year 2 outpaced the reductions in low-performing fifth-graders at the comparison districts. However, these differences were not statistically significant.
Across the BASRC focal districts, for each of these subgroups, there was a reduction in the percentage of fifth-grade students scoring below basic that outpaced the average reduction in the comparison districts. These differences were not statistically significant and were generally limited to Year 2 of the intervention. However, the fact that the pattern repeats itself across several subgroups suggests the possibility that the BASRC focal districts may have made progress in improving the performance of their lowest-performing fifth-graders. The improvements were largest among ELL students (not shown in the figures). For example, during the baseline period, 55 percent of fifth-grade ELL students scored below basic in the focal districts. By the second year of follow-up, only 41 percent scored below basic, a reduction of 14 percentage points. The comparison districts, in contrast, experienced less improvement, reducing the proportion of students scoring below basic by 5 percentage points.
Interpreting the Findings
The evidence presented in this report suggests that the BASRC focal strategy is not associated with improvements in achievement among third-graders. However, with respect to fifth-graders, the progress of the BASRC focal districts tended to outpace that of the comparison groups, particularly among lower-performing disadvantaged, minority, and ELL students.
It is important to note that these differences tended to be small, and were not statistically significant. As such, it may be that there are no true differences between the progress in the focal and non-focal districts. Moreover, the design of the study does not prove a causal relationship between participation in the BASRC focal strategy and improved student outcomes for lower-performing students. However, it is possible that focal reforms had a modest effect on student performance by lower-achieving fifth-graders. Since this relationship existed primarily in the second year of implementation, it will be important to examine follow-up data to ascertain whether these differences persist, grow, or fade over time.
What explains the evident lack of a substantial, pervasive association between the BASRC focal strategy and student achievement? The implementation and outcome findings suggest several possible interpretations:
- Hypothesis 1: The BASRC focal reforms are not intense enough to affect students' academic performance.
While the core components of BASRC focal reforms may be potential drivers of improved teaching and learning, the implementation of the focal reform strategy may simply not have been intense enough to change student achievement in a pervasive manner. For example, there may not have been a sufficient number of interactions between BASRC staff and district and school leaders, or these interactions may not have been sufficiently focused on implementation of the BASRC reform strategies. The fact that the school-level strategy faded away may have limited the intensity of the reform activities. A lack of intensity could also be the result of a lack of focus, which could lead to not spending very much time on any particular effort. To the extent that this is true, it might suggest that, for the focal reforms to have an effect, BASRC must increase the intensity of its district-reform efforts and imbue those efforts with a consistent focus.
- Hypothesis 2: BASRC focal reforms are not reaching the classroom.
It is also possible that the BASRC focal strategy, regardless of intensity (that is, regardless of frequent and focused coaching sessions with the superintendent), would not lead to improvements in teaching and learning because the intervention is not close enough to the classroom. This might suggest that, in order to affect teaching and learning, the focal strategy must intervene in ways that ensure reforms reach the classroom level. This could include sustaining school-level coaching efforts or providing other supports designed to increase the effectiveness of school- or classroom-level BASRC reforms. It is also possible that, by their nature, district reforms can be effective but simply require more time to take root at the school level.
- Hypothesis 3: Core BASRC reforms are not sufficiently powerful drivers of improved teaching and learning.
On the other hand, even if the focal strategy did increase the intensity of reform efforts at the school and classroom levels, it is possible that the reforms themselves are not strongly related to improved student achievement. In particular, it is possible that the BASRC reform activities supported by the focal strategy, even when effectively implemented, do not result in measurable improvements in teaching and learning. This would suggest that the BASRC focal districts and schools would make more progress by implementing reforms focused more directly on refining classroom practice or by implementing particular pedagogical approaches. MDRC's next report will explore this hypothesis by examining the relationship between school-level implementation of particular BASRC reforms and changes in student achievement.
Notes
1After this study was conducted and the report was written, BASRC changed its name to Springboard Schools. For ease of reference, the report uses the former name throughout.
2CRC also conducted an evaluation of BASRC's first phase of reform work (1995-2001).
|
