Nowhere is improving teaching and learning in America more critical than in those chronically low-performing schools that serve large proportions of economically disadvantaged and nonwhite students. This is the mission that the Institute for Learning (IFL), an arm of the Learning Research and Development Center (LRDC) at the University of Pittsburgh, has set for itself. Established in 1995, the IFL provides technical assistance to school districts, primarily in the form of strategic planning, coaching, and professional development for district and school administrators. In its work with districts, the IFL enunciates a set of “Principles of Learning” about the ideas and practices that lead to academic achievement for all students.

The IFL’s program of learning for district and school administrators is guided by a theory of action that is depicted in Figure ES.1. As the figure suggests, principals play a key role in the instructional improvement process by setting in motion a sequence of school-level behavior changes that make for improved teaching and learning. In the IFL’s leadership training for principals — which is designed to last at least two years, with principals typically receiving between 36 and 63 hours of training per year — principals learn about the Principles of Learning and about concrete actions that they can take to motivate and support their teachers. Principals are expected to organize professional learning for their teachers as well as to monitor teachers’ classroom practices and help them incorporate new behaviors that are in accordance with the Principles of Learning into their instructional repertory. With improved instruction, the theory holds, student achievement will improve.

This report was not intended to be an evaluation of the impact of the IFL’s work, which would have assessed whether changes in instructional quality and student achievement took place over and above what would have occurred without the IFL’s presence in the schools and districts. Instead, the analysis tests the IFL theory of action by examining the empirical relationships among measures of principal, teacher, and student behavior and performance that are associated with each step in that theory. It finds evidence of systematic relationships that is consistent with the conceptual framework: Statistically significant linkages connect variables at each step in the theory of action with variables at the next step. While these findings are suggestive and promising, however, limitations in the research design and the data collected mean that these linkages cannot be interpreted as causal; on the question of causality, the results are ultimately inconclusive.

Study Sites, Data, and Analytic Strategy

Research for this report was conducted at 49 elementary schools in three districts, or “sites” — Austin, Texas; Saint Paul, Minnesota; and Region 10 within New York City — that had worked with the IFL for between one and four years at the time that data collection began. Reflecting the shared priority placed by both the IFL and MDRC on improving educational outcomes for disadvantaged students, study schools were selected to include large numbers of students at high risk of adverse educational outcomes. Thus, between 82 percent and 87 percent of the students served by the study schools in each district were eligible for free or reduced-price lunch, and the schools also included more nonwhite students than did elementary schools in their districts more generally. Districts and schools participated voluntarily in the research.

At all sites, the scores of third-graders at each study school on the statewide high-stakes test were used to measure student achievement in the spring of 2006; the average of third-grade students’ test scores from the three previous years served as a control variable in the analyses of the relationships between student achievement and other variables. Because of budgetary limitations, all the other quantitative data used in the report were collected during the 2005-2006 school year. Surveys were collected from 44 principals and 274 third- and fourth-grade teachers in the study schools during the spring of 2006. Their primary purpose was to obtain information about the frequency and value of instruction-related professional development that both groups had received and, in the case of principals, had passed on to teachers at their schools.

Researchers conducted observations in third-grade classrooms to ascertain the extent to which practices that incorporated three key IFL Principles of Learning — Accountable Talk, Academic Rigor, and Clear Expectations — were evident in the study schools. Two observation instruments that were developed by LRDC researchers — the Reading Instructional Quality Assessment (IQA) and the Math IQA — were adapted for use in the study. Almost all third-grade teachers in the study were observed once during a reading lesson and once during a math lesson, for a total of 151 reading observations and 151 math observations.

Qualitative research, including interviews with high-level district officials and IFL liaisons for the study districts, enabled researchers to gain a better understanding of the IFL’s work. Further, case studies involving daylong visits to eight schools across the three sites helped illuminate the findings from close-ended surveys.

The analysis uses multiple regression analysis to ascertain the extent to which outcomes at each step of the theory of action are associated with (that is, are statistically linked to) outcomes at the one or two preceding steps in the theory. Data from teacher surveys and classroom observations at individual schools were aggregated so that, in all the regression analyses, the school is the unit of analysis. Since the goal of the study is to examine the nature of the relationships between the steps in the theory of action independent of other factors that may influence the outcomes, additional measures — principals’ length of experience, the average experience of the teachers at the school, and indicators for the three school districts in the study — are included in every analysis in order to control for the effects of these factors.

Key Findings

Findings About Relationships Between Steps in the Theory of Action

As noted above, outcomes at every step in the theory of action are linked to outcomes at earlier steps in statistically significant ways. With regard to principals’ involvement in the professional development environments at their schools:

• Greater receipt of instruction-related professional development on the part of principals and a greater value attached to that professional development are both significantly and positively associated with the principals’ involvement in professional development for their teachers.



• Greater principal involvement in professional development for teachers is significantly and positively associated with the frequency with which teachers reported receiving professional development.

Further, relationships were observed between the professional development received by teachers and instructional quality:

• In schools where teachers reported that they received more instruction-related professional development, researchers were more likely to observe higher implementation of the Principles of Learning in reading lessons.



• Schools where teachers placed greater value on the professional development that they had received related to the Principles of Learning were observed to have higher implementation of these principles in math lessons.



• A direct relationship was observed between the role that principals played in professional development for teachers related to Academic Rigor and Clear Expectations and higher implementation of these principles in reading lessons.

Finally, relationships were observed between instructional quality and student achievement on standardized exams in math and reading:

• Higher school-level scores on the Reading and Math IQAs are associated with greater percentages of students meeting the standard on the reading and the math state assessments, respectively.



• In particular, higher implementation of the principles of Accountable Talk and Academic Rigor in reading lessons are associated with higher student achievement in reading, and higher implementation of the principle of Accountable Talk in math lessons is associated with higher student achievement in math.

Other Findings

• Both principals and teachers especially valued opportunities to learn from their peers.



• Instructional quality in third-grade reading and math classes, as measured by overall IQA scores and by scores on the three Principles of Learning, was generally low: Students were rarely pushed to analyze their work, to provide evidence for their claims, or to hold each other accountable during class discussions.



• Teachers’ reports of what they do during reading and math lessons generally did not match what researchers observed teachers doing.

Implications and Limitations of the Study

The analyses in the report suggest that in-service professional development for principals can serve as a catalyst for positive changes in teaching and learning in the principals’ schools. While statistically significant linkages were discovered between all the steps in the theory of action, these connections must be regarded as exploratory and provisional rather than definitive, for two main reasons.

First, the small number of schools and of teachers sampled within each school makes the quantitative estimates about the size of relationships unstable — that is, highly sensitive to the exact sample used in the statistical analyses. The small sample size also limits the generalizability of the findings. It is impossible to know whether the same patterns would be observed in other districts where the IFL is working, in other schools in the same districts as the study schools, or even in grades other than third grade within the study schools.

Second, the theory of action informing the report assumes that behaviors will unfold in sequences that have an implicit temporal order. But because budgetary pressures precluded a second round of data collection, the data used in the study were all collected within the same academic year, so that is not possible to determine which actions come first and which follow. For this reason, the analysis can provide no empirical evidence that one action caused another.

In fact, for most of the linkages set forth above, an alternative explanation to the one suggested by the IFL theory of action can be found. For example, more professional development in reading may be associated with higher instructional quality, but it cannot be assumed that the professional development led to better teaching. Rather, teachers who are already skilled may be especially motivated to seek professional development that will make them even better at what they do.

Because the data supporting the theory of action are promising but inconclusive, a more definitive research project to study the impacts of professional development aligned with this theory is warranted. The key requirement for such a study is the presence of a counterfactual — an estimate of what would happen in similar schools if their principals do not receive such professional development — preferably a counterfactual that is established through a lottery-like process known as “random assignment.” If selection for the treatment is truly random and the sample size is reasonably large, preexisting differences between the treatment and control group principals and schools should be effectively eliminated, so that any differences between the two sets of schools that emerge over time can safely be attributed to their principals’ having received the professional development or not having done so. The present study, it is hoped, will lead to a rigorous evaluation of the theory of action guiding the IFL’s promising efforts to improve instructional leadership, teaching, and learning.