This is the second and final report for MDRC’s evaluation of the Bay Area School Reform Collaborative (BASRC), a grant-making and support organization in San Francisco, California.1 BASRC is dedicated to improving student achievement in public schools and narrowing achievement gaps among different racial, ethnic, and socioeconomic groups. BASRC reforms are designed as continuous improvement processes driven by collective inquiry into student learning outcomes, school functioning, and teacher practice. This process-oriented reform strategy is designed to build professional knowledge about effective practice, promote mutual accountability, and facilitate collaboration. BASRC hypothesizes that making changes along these lines will result in improved teaching and learning, but the organization does not prescribe particular instructional practices, curricula, or school structures that should evolve as a result of its reform processes.
This evaluation centers on a specific BASRC initiative, the “focal strategy,” as it was implemented during the 2002-2003, 2003-2004, and 2004-2005 school years in five Bay Area districts (the “focal districts”),2 and two to four selected schools within those districts (the “focal schools”).3 Building on earlier phases of BASRC reforms, which began in 1996, the focal strategy was meant to increase the intensity of the core BASRC reforms by creating more opportunities for district and school administrators to interact with BASRC staff. In selecting the focal districts, BASRC tended to focus on districts where a strong working relationship had developed during earlier BASRC efforts, and the focal reforms were implemented on top of existing BASRC reform work in these places. In short, compared with the earlier reform strategies, the focal strategy served as a stronger test of BASRC reform ideas in fewer places.
While the first report on the evaluation explored the relationship between the BASRC focal strategy and student reading achievement for third- and fifth-graders in the first two years of implementation in the focal districts, this report focuses only on fifth-graders and provides an additional year of follow-up data.
The key findings of MDRC’s evaluation of the BASRC focal strategy are as follows:
- In the districts that participated in the focal strategy, there were improvements in achievement during the years of the initiative, but these improvements were either similar to or only slightly greater than improvements in similar districts in the Bay Area that were not part of the focal strategy.
- The evidence suggests that the BASRC focal strategy is not associated with substantial districtwide improvements in average elementary student achievement on state standards tests of reading.
- Although BASRC originally intended to support both districts and schools in a “bottom-up and top-down” approach, in practice the focal strategy primarily served district leadership, was not sustained at the school level, and, thus, typically did not lead to specific changes in instruction or specific instructional supports at the school and classroom levels.
MDRC’s evaluation of the BASRC focal strategy expands knowledge about how districts approach systemic reform. BASRC’s focal strategy is an example of how external support organizations are increasingly partnering with districts to help initiate and sustain systemic reform. This report provides new knowledge about the difficulties external support organizations like BASRC face in their efforts to forge effective coaching relationships, influence norms and cultures, introduce new processes, create coherence, and build knowledge. Moreover, the report demonstrates the challenge of translating systemic changes into reforms that actually reach the classroom, improve instruction, and raise student achievement.
The BASRC Focal Strategy
The BASRC focal strategy centered on three core features:
- coaching of superintendents, district and school leaders, and teachers
- evidence-based decision-making at all levels of the system (for instance, using student achievement data to inform decisions about policy and practice)
- networks and collaboration among administrators and teachers and within and across districts and schools
All three features were also part of BASRC’s earlier reform efforts. Once the focal strategy began, however, school-level coaching by BASRC staff was redirected toward focal schools within the focal districts. In addition, “executive coaches” (former superintendents) were hired to advise and support the leaders in the focal districts. Along with regular meetings on comprehensive needs assessments and reviews of progress, these efforts were intended to reinforce all elements of the strategy in the focal districts and schools.
The theory of action underlying the BASRC focal reform strategy posits that the implementation of these three design features yields changes in district, school, and classroom practices that in turn improve student outcomes. When instituted together, coaching, evidence-based decision-making, and networks and collaboration are thought to increase districts’ and schools’ engagement in a continuous improvement process that is directed at improving the degree and equity of student achievement levels. Because the strategy is process-oriented, it can result in different policies or practices that may or may not be linked with specific approaches to teaching and learning.
The Evaluation of the BASRC Focal Strategy
The independent evaluation of BASRC is funded by the William and Flora Hewlett Foundation and includes studies by both MDRC and the Stanford University Center for Research on the Context of Teaching (CRC).4 The CRC studies focus on the process of reform as it relates to BASRC in general and the relationship between various BASRC reform efforts and changes at the district, school, and classroom levels. MDRC’s study attempts to shed light on the relationship between the BASRC focal strategy and improvements in student achievement.
It is important to note that MDRC’s study is limited in its capacity to establish a rigorous causal relationship between implementation of BASRC focal reforms and changes in student achievement that may have occurred after the focal strategy was introduced. BASRC selected districts where it already had a strong working relationship and where the superintendent was reform-minded and focused on achievement gaps. Therefore, the BASRC focal districts may have differed from comparison districts with similar demographic and achievement characteristics in unmeasured and meaningful ways. Consequently, subsequent differences between the progress made in the focal districts and progress made in comparison districts could be due to the fact that the BASRC focal districts had particularly entrepreneurial leaders rather than to additional contributions made by the BASRC focal strategy. Thus, the analyses undertaken for this evaluation are intended to be suggestive and descriptive of the potential relationship (or lack of relationship) between implementation of the BASRC focal strategy and trends in student achievement, rather than definitive estimates of the causal effect of the BASRC focal strategy on student achievement.
The BASRC focal reforms tended to center on literacy instruction and were concentrated at the elementary level. Therefore, all analyses of the focal strategy are confined to the elementary school level and emphasize district performance on measures of student literacy. The analyses of student achievement examine outcomes particularly for fifth-grade students, who have been in the schools longer than other students at the elementary school level and whose scores reflect a more cumulative effect of the quality of education across the elementary grades.
Key Findings
Implementation Findings
MDRC’s implementation findings are based primarily on interviews and focus groups conducted in four of the five focal districts during the 2003-2004 and 2004-2005 school years — the second and third years of the BASRC focal strategy.
- At the district level, the BASRC focal strategy resulted in coaching of superintendents and other district administrators, professional networking opportunities for district and school leaders, and needs assessment and review meetings with BASRC staff. However, the strategy did not typically translate into specific reforms aimed at improving teaching and learning.
Executive coaches were charged with helping superintendents identify, implement, and support system-wide efforts to raise student achievement and narrow achievement gaps. In practice, the work between the executive coaches and superintendents tended to center on building capacity for these efforts — such as strengthening leadership, becoming more focused on reform work, and improving communication — rather than on actually implementing reforms. The executive coaches, along with BASRC staff coaches, also began meeting with other district leaders who had a more direct role in affecting teaching and learning. This coaching often revolved around a variety of district needs related to readiness for education reforms, such as building support for district reform goals and mending communication difficulties within and across departments. In interviews, executive coaches said they made progress in these areas and putting instructional reforms in place was “the next step.” All together, the coaching, network meetings, and meetings with BASRC staff led to the establishment of goals for improving student achievement and reducing achievement gaps and to enhanced capacity for working toward the goals. However, they did not typically translate into specific reforms aimed at improving teaching and learning.
- By the 2004-2005 school year (the third year of the initiative), the focal district strategy had ended early, either formally or informally, in three of the five focal districts.
In three of the five original focal districts, BASRC staff encountered difficulties that led to an early conclusion of the BASRC focal reform efforts. According to BASRC staff, they never fully gained traction in one of the focal districts and officially ended the partnership before the start of the third year of the strategy. In another focal district, although the executive coach maintained a mentoring role with the superintendent, the coaching role as BASRC envisioned it ended in September 2004. And, in a third focal district, staff turnover, both at the district office and among the BASRC staff working with the particular district, led to a disruption in the work.
- By the 2003-2004 school years (the second year of the initiative), the school-level aspect of the focal strategy faded. Thus, the model evolved to be almost entirely a district strategy.
In theory, coaching by BASRC staff was to be a primary feature of the school-level focal strategy. This “external” coaching effort was distinct from the other “internal” coaching efforts in place in non-focal schools (including school-level literacy coaches and coaches employed by the district to support reform work). However, in the first year of implementation, BASRC encountered several complications, including resistance to the BASRC coaches from school-level staff. By the end of the first year, BASRC coaches did not have a significant coaching role in most of the focal schools. In the third year of the strategy, school-level coaches began to develop more prominent roles at elementary schools in two of the focal districts, but this coaching was not generally focused on delivery, support, or implementation of the BASRC focal reforms.
- At the schools within the focal districts, there was evidence of all three key features of the BASRC focal strategy; however, the instances of the key features were likely vestiges of earlier BASRC reform phases.
MDRC found evidence that all three of the key features of the BASRC focal reform strategy — coaching by district or school staff, evidence-based decision-making, and networks and collaboration — were in place at the schools in the focal districts. However, field research data indicate that these were typically implemented as a result of participation in earlier BASRC reform efforts rather than participation in the BASRC focal strategy itself. It is also possible that these reform practices were in place before any participation in BASRC reform efforts. Moreover, in the schools visited by MDRC, it was difficult to detect meaningful differences in the types of BASRC supports and reform activities in place in focal schools compared with non-focal schools.
- The extent to which teachers engaged in BASRC practices — coaching by district or school staff, evidence-based decision-making, and networks and collaboration — did not increase during the years of the focal strategy.
In investigating implementation of the focal strategy at the school level, the evaluation also included an annual teacher survey developed by MDRC and CRC. The survey was administered in BASRC elementary schools each spring in the same four focal districts in which field research was conducted. Among other things, the survey posed a variety of questions regarding the extent, nature, and frequency of teacher participation in school reform practices related to BASRC’s three key features. The survey does not capture all activities related to the focal strategy, but it does provide information on some key activities. Overall, the survey data revealed little change in the key reform activities during the years of the focal strategy.
Student Achievement Outcomes
In order to explore the relationship between participation in the BASRC focal strategy and changes in student achievement, MDRC’s analysis of student achievement compared progress in the BASRC focal districts in the three years of the BASRC focal reforms to progress in a set of carefully chosen comparison districts from throughout the San Francisco Bay Area over the same period.
- Fifth-grade students’ achievement in reading improved over time in the focal districts, in terms of both an increase in the percentage of students scoring proficient or above and a decrease in the percentage of students scoring below the basic level. There were similar improvements in the comparison districts.
Both the focal districts and the Bay Area comparison districts saw improvements in fifth-grade language arts achievement at both the top and bottom of the achievement scale. In terms of proficiency rates, the average percentage of focal district students scoring at or above proficient increased from 38 percent to 54 percent by the end of follow-up Year 3. A similar change, from 39 percent to 52 percent, occurred in the comparison districts. (These patterns are in line with upward trends across the Bay Area and across the state as well.) Both the BASRC focal districts and the comparison districts reduced the number of students performing below basic. In Years 2 and 3, the improvements in the focal districts may have outpaced those that occurred in the comparison districts. For example, the percentage of students scoring below basic dropped by about 5 percentage points in the focal districts compared with a drop of about 2 percentage points in the comparison districts, suggesting a net difference of 3.2 percentage points. However, the difference between progress in the focal and comparison districts is relatively small and not statistically significant. As such, the possibility that the difference is simply the result of chance cannot be ruled out. In short, it is not possible to conclude that there is actually a systematic association between implementation of the BASRC focal strategy and student achievement on California’s state standards. Moreover, any association that does exist appears to be very modest in size.
- Among blacks and Hispanics, English Language Learners (ELL), and economically disadvantaged fifth-grade students, the improvements in reading achievement in the BASRC focal districts outpaced the improvements in the comparison districts later in the follow-up period. However, the differences in the improvements in the focal districts and those in the comparison districts are small and generally not statistically significant.
When focusing on fifth-grade achievement among economically disadvantaged students, black and Hispanic students, and English Language Learners, performance in the focal districts appeared to surpass the improvements in the comparison districts in the second and third years of follow-up. The differences MDRC observed could suggest a positive association between the BASRC focal strategy and improvements in reading scores on the California Standards Tests (CSTs) among disadvantaged subgroups of students. However, the size of the differences is small and the differences are generally not statistically significant, meaning that the differences could have occurred by chance.
The findings are not surprising given that implementation of the strategy did not translate into instructional reforms or increased use of BASRC-related practices in the schools. Moreover, even if the small difference between the focal and non-focal districts points to an association, MDRC cannot make causal inferences. The selection of “reform-minded” districts as participants in the focal strategy could mean that the BASRC focal districts were more likely than districts with similar student populations and achievement track records to implement reforms and improve student performance.
- Schools with higher levels of BASRC-related practices did not have higher levels of achievement.
Using survey measures of reform practices related to BASRC’s three key features, MDRC examined the relationship between schools’ average levels of BASRC-related practices during Years 1, 2, and 3 of the focal strategy and proficiency levels in reading in Year 3, controlling for differences in the demographic characteristics of the students these schools served and for achievement levels prior to the start of the focal strategy. This analysis did not reveal relationships between the BASRC-related practices measured on the survey and fifth-grade reading achievement. The absence of such relationships may have several explanations. First, the measures may not have sufficiently captured enough variation across schools. For example, all schools had high average ratings for their tendency to examine achievement data, which may or may not be because they were all BASRC grantees. Therefore, the analysis assesses only whether improving from a high level to a higher level of examining achievement data is related to better achievement outcomes; it does not assess whether improving from a low level to a high level of examining achievement data is related to better achievement. The absence of a relationship could also reflect limitations of the survey measures to sufficiently capture teachers’ practices. However, it could also indicate that the BASRC-related practices are not associated with improved proficiency in reading.
Interpreting the Findings
This report builds on an earlier report, released in February 2006, which found no strong association between the BASRC focal strategy and changes in average student proficiency rates on state-mandated literacy tests. In this report, with an additional year of follow-up, the evaluation points to the same conclusion. In the districts that participated in the focal strategy, there were improvements in achievement during the years of the focal strategy, but these improvements were either similar to or only slightly greater than improvements in similar districts in the Bay Area.
Given the nature of the reforms and the implementation patterns, the findings may be understandable. By design, the model did not focus on implementing particular classroom instructional strategies; rather, it emphasized a process through which districts, schools, and teachers could come to learn how they can best support improved teaching and learning. In practice, the BASRC focal strategy primarily targeted district leadership and focused primarily on building capacity for reform efforts rather than on actually implementing reforms directed at teaching and learning. Also, at the school-level, coaching by BASRC staff fell by the wayside, and, although a goal of the focal strategy was to intensify the core BASRC reforms, it was unclear whether the BASRC-related practices at the school level resulted from the focal strategy. In addition, the teacher survey data indicated little change in BASRC-related practices during the years of the focal strategy.
Systemic reforms such as BASRC can take a long time to take root. If they do, and if they are translated into effective instructional improvements, the changes in teaching and learning could be substantial and more sustainable than those evoked by other reforms. This evaluation suggests that external support organizations like BASRC might be more successful in helping districts achieve successful systemic reforms if they can simultaneously support both district offices and schools, if they can move beyond building capacity, and if they can help district offices and schools identify and implement effective instructional changes.
Notes
1 During this study, BASRC changed its name to Springboard Schools and became a nonprofit, fee-for-service organization that offers its clients — urban, suburban, and rural school districts throughout California — professional development services, on-site district coaching, access to research, and a variety of tools and services for improving schools. For ease of reference, the report uses the former name throughout.
2 While the BASRC focal strategy continued during the 2004-2005 school year, it began to wind down during that period, as BASRC focused on developing new professional development services and strategies for a broader set of districts in California. See www.springboardschools.org.
3 During the 2003-2004 school year, BASRC added a sixth focal district that is not included in MDRC’s study.
4 CRC also conducted an evaluation of BASRC’s first phase of reform work (1995-2001).
|
