Study of the Impact of Instructional Coaches on Middle School Teachers and Student Achievement
The research design for evaluating the impact of the El Paso MSP instructional coaches on teaching and learning involved multiple measures and analyses. Data from staff developers, teachers, principals, and students was linked in order to evaluate the degree to which the work of the coaches and related professional development under the El Paso MSP is associated with changes in teachers' content and pedagogical knowledge, their instructional practices, their collaboration with peers, and ultimately, student performance. The following is a listing of the primary measures that were used in the study:
- Mathematics/Science Teacher Surveys
- Staff Developer Interviews
- Teacher Interviews
- Principal Interviews
- Measure of Pedagogical Content Knowledge for Teaching Mathematics
- Measures of Student Performance
Two teacher surveys were developed: a mathematics teacher survey and a science teacher survey. The teacher survey measured constructs including teachers' attitudes about student learning and effective teaching, their understanding and use of the curriculum frameworks, their use of specific teaching methods, their perceptions of principal leadership, their collaboration with other teachers, and their expectations for the achievement their students.
In developing these surveys, CPRE reviewed measures from previously published studies and current research studies that are directly relevant to our theory of action. The primary purpose of this review of other instruments was to identify existing sets of items that could be used on the El Paso surveys. The instruments and surveys reviewed included measures relevant to implementation of high-quality coaching (Camburn, Rowan, & Taylor, 2003), high-quality professional development (Garet, et al., 2001), and the use of formative assessments in differentiated instruction. CPRE also reviewed items that have been used in previous research to capture elements of schools as organizations, including items measuring instructional leadership of principals (Camburn, Rowan, & Taylor, 2003), instructional program coherence (Newmann, et al., 2001), and the strength of a school's professional community (Louis, Marks, & Kruse, 1996). CPRE examined items that had been used in previous research to measure various aspects of instruction and teacher quality in schools, including measures of teaching and curriculum enactment (Porter & Smithson, 2001), measures of teacher learning (Bryk, Camburn, & Louis, 1999). Sets of items selected in this review were adapted for use on the El Paso surveys where necessary, through rewording of items or elimination of items having less direct relevance. Although the numbers of items and scales selected from other instruments were significant, more than half of the items included on the El Paso surveys were developed specifically for this research.
Each staff developer was interviewed twice annually. These were extensive semi-structured interviews. Staff developers were asked open-ended questions, and then probed to provide detail and examples regarding their accomplishments, their strategies for working with teachers, their use of tools and evidence, their professional development, their workloads, and the challenges they faced during the school year. The baseline interview took place in August 2005 and focused on the staff developers' goals and expectations for their upcoming work in middle schools. Three follow-up interviews were conducted in May 2006, November 2006, and May 2007. A primary focus was placed on understanding the challenges to teaching at a high level of cognitive demand and the degree to which staff developers felt they had been successful in helping teachers do this.
Interviews of teachers focused on three main topics: an observed lesson, work with the staff developer, and school context. In terms of the lesson, interviews focused on teachers' selection of the content, their lesson design and the cognitive demand of the lesson, their choice of instructional techniques, the student learning challenges they encountered and anticipated, and their assessment of student mastery. Teacher interviews also probed their views on the role of the staff developer, their interactions to date, and their assessments of the quality of the staff developers' support. A final set of questions focused on teachers' perspectives on the math and science instructional priorities of campus administrators, opportunities for professional development and external initiatives that shape their classroom practice. Each structured interview took about 30-40 minutes. A total of 184 interviews were conducted; in spite of repeated attempts, we were not able to interview two teachers.
In Fall 2005 and Spring 2007, interviews were conducted with principals to collect information about school context. Questions focused on major initiatives in math and science, perceptions of the value of the MSP partnership, and current working relationship with staff developers assigned to the school. It also elicited principal perceptions of the difficulties their students were encountering in math and science, hoped for changes in instruction, and their early assessment of interactions to date between teachers and staff developers. Interviews were conducted at the beginning and end of project, Fall 2005 and Spring 2007. A total of 28 interviews were conducted.
Mathematics teachers' content knowledge was measured through the Content Knowledge for Teaching Mathematics Measures, or pedagogical content knowledge (PCK), developed by Hill, Schilling and Ball, (2004). The PCK was designed to measure teachers' knowledge in three domains: (1) number concepts and operations, (2) patterns, functions, and algebra, and (3) geometry. All math teachers in MSP middle schools were invited to participate in a professional development session on February 11, 2006. A total of 120 teachers and seven staff developers attended.
Student achievement data in mathematics and science come from the Texas Assessment of Knowledge and Skills (TAKS). Mathematics test scores were obtained for individual students in grades 3 through 8 during 2003 through 2007. Science test scores were obtained for individual students in grade 5 during 2003 through 2007 and in grade 8 during 2006 and 2007.
Data Analysis
Data analysis involved linkage and triangulation of multiple sources of baseline, year-one, and year-two data in an effort to describe changes in instruction and student performance associated with the MSP program in middle schools. Most of the analyses used multilevel modeling techniques, such as Hierarchical Linear Modeling or HLM, and separate analyses to explore hypotheses posited by El Paso MSP.
The four main impact analyses were:
- First, an analysis conducted to determine the impact of the El Paso MSP instructional coaches on middle school math and science teacher practices. This included a quantitative analysis of the relationship between participation in professional development and changes in teacher practices and attitudes as measured by the teacher survey and as captured by classroom observations.
- Second, the impact of the EL Paso MSP coaches on middle school math teacher content knowledge was examined through the analysis of the results from the content knowledge measure. The statistical model for this analysis was a two-level HLM model with teachers nested within schools. Pedagogical content knowledge scores from 2005 served as a pretest covariate for the 2007 scores, which served as the dependent variable.
- Third, two analyses were conducted to determine the impact of the El Paso MSP on middle school math and science achievement. The first linked student achievement data to teacher survey data to explore the links between annual student learning gains and the degree of participation by individual teachers in professional development provided by the coaches. The second analysis was longitudinal, employing a growth curve model to evaluate changes in student' rates of learning that were hypothesized to occur after the implementation of the MSP professional development program.
- Finally, an analysis of the factors which may mediate the impacts of the El Paso MSP instructional coaches was conducted. Data from principal and MSP director surveys were analyzed to determine the degree to which the relationships revealed in the aforementioned analyses are influenced by mediating factors such as the effectiveness of individual coaches as rated by MSP directors, concurrent reform activities in schools and districts, principal support and activities as instructional leaders, and general school conditions and context.
