In practice, teacher turnover appears to have negative effects on school quality as measured by student performance. However, some simulations suggest that turnover can instead have large positive effects under a policy regime in which low-performing teachers can be accurately identified and replaced with more effective teachers. This study examines this question by evaluating the effects of teacher turnover on student achievement under IMPACT, the unique performance-assessment and incentive system in the District of Columbia Public Schools (DCPS). Employing a quasi-experimental design based on data from the first years of IMPACT, we find that, on average, DCPS replaced teachers who left with teachers who increased student achievement by 0.08 standard deviation (SD) in math. When we isolate the effects of lower-performing teachers who were induced to leave DCPS for poor performance, we find that student achievement improves by larger and statistically significant amounts (i.e., 0.14 SD in reading and 0.21 SD in math). In contrast, the effect of exits by teachers not sanctioned under IMPACT is typically negative but not statistically significant.
teacher quality; teacher turnover; teacher evaluation
Having an effective teacher can dramatically alter students’ educational and economic outcomes. Yet, we know that there are substantial differences in the quality of public school teachers, and there is increasing evidence that in some urban areas less effective teachers are often concentrated in lower-performing schools serving disadvantaged students. Policymakers and researchers recognize these issues and have sought policies to provide all children with effective teachers. The selective retention of effective teachers has been one of the most-discussed strategies that may contribute to this goal. In theory, districts could dismiss ineffective teachers, hire more effective teachers, and redouble efforts to retain effective teachers in these schools. However, we know relatively little about how such policies would work in practice. In particular, the capacity of districts to identify effective teachers at the hiring stage is limited ([
Some school districts have begun to implement rigorous teacher-evaluation policies that could systematically dismiss meaningful numbers of ineffective teachers ([
Average IMPACT scores of all general education teachers (Groups 1 and 2) by year. Note. Results for 2011 indicate the average score of teachers who exited at the end of 2009–2010 compared with those entering in 2010–2011. Exits include teachers who retired, resigned, or were terminated. Teachers leaving schools that closed are excluded.
We employ a quasi-experimental event study to examine teacher turnover and its effect on student achievement in DCPS in 2011 through 2013. Specifically, we rely on school-grade-year cells as our fundamental unit of observation and examine, in “difference-in-differences” specifications, how the patterns of teacher mobility influence student test performance in math and English-language arts (ELA). We find that teacher turnover in DCPS had an overall positive effect on student achievement in math (i.e., 0.08 SD), and that the effect of turnover in reading is also positive (i.e., 0.046 SD) but is only significant at the 10% level. However, the overall effect of teacher turnover masks considerable heterogeneity. We find that, when low-performing teachers (i.e., those with “Ineffective” or “Minimally Effective” ratings under IMPACT) leave the classroom, student achievement grows by 21% of a standard deviation in math and 14% of a standard deviation in reading. We also find that the attrition of high-performing teachers (i.e., those rated “Effective” or “Highly Effective”) has a negative but statistically insignificant effect on student performance.
To be clear, this article should not be viewed as an evaluation of IMPACT, or even as an assessment of IMPACT’s differential effect on teacher composition in DCPS.1 [
Improving teacher quality in schools with poor, low-performing, and largely non-White students has become an imperative of education policy. A recent body of research has made it clear that the variance in teacher effectiveness is qualitatively large and that more effective teachers can dramatically improve students’ short- and long-run life outcomes ([
The composition of a district’s teachers improves when their policies retain the most effective teachers, exit poorly performing teachers, and select the most able entering teachers. High-performing teachers leave their schools and districts for a variety of reasons, some personal, but most related to attributes of their jobs ([
Increasing the retention of effective teachers would appear to be an obvious strategy to improve teaching effectiveness, yet over a third of high-performing teachers report that they received little encouragement from their principals to remain at their current school ([
In the absence of real-world evidence on the effects of policies that improve teacher composition, researchers have simulated the effects of such policies employing data-driven assumptions. It is estimated that annually replacing teachers who fall in the bottom 5% to 10% of the value-added distribution would improve student achievement by 50% of a standard deviation ([
In sum, there is at best limited empirical evidence of the effects of differential retention policies on teacher quality and student achievement. What evidence does directly bear on this issue are simulations dependent on a series of simplifying assumptions about the policies and the behavioral responses of existing teachers and the available labor market. Our study leverages the implementation of IMPACT, the high-stakes teacher-evaluation and compensation system in DCPS, to examine this issue directly in an at-scale setting.
In just the last few years, the design and implementation of teacher evaluation has evolved quickly as many districts look to improve teacher performance, partly under the encouragement of federal policies such as waivers from the No Child Left Behind (NCLB) Act and the Teacher Incentive Fund (TIF). While these policy innovations are still a work in progress, best-practice principles of effective evaluation are beginning to emerge ([
Each year, teachers receive a final IMPACT score that determines their IMPACT rating and their associated consequences. IMPACT scores range from 100 to 400 points and include several components, which depend on a teacher’s grade and subject of instruction. The teachers in this analysis all taught in tested grades and subjects and thus have a value-added component, which during the period of our analysis comprised 50% of their IMPACT score. Thirty-five percent of their IMPACT score was determined by rigorously scored classroom observations tied to the district’s Teaching and Learning Framework (TLF). The TLF specifies the criteria by which DCPS defines effective instruction and structures a scoring rubric. The TLF includes multiple domains such as leading well-organized, objective-driven lessons, checking for student understanding, explaining content clearly, and maximizing instructional time.3 [
All teachers are also assessed by their administrators on a rubric that measures their support of school initiatives, efforts to promote high expectations, and partnerships with students’ families and school colleagues: the Commitment to the School Community (CSC) measure. CSC is weighted to represent 10% of the overall IMPACT scores. Teachers also received a score based on their school’s estimated value-added (SVA), which contributes 5%. Finally, principals assess each teacher on their “Core Professionalism” (CP). The rubric for CP rates teachers on the basis of attendance, punctuality, policies and procedures and respect. Teachers are assumed to be professionals, and, therefore, CP scores can only reduce a teacher’s overall IMPACT score.
During the period of our study, IMPACT scores were translated into one of four IMPACT ratings, which dictated consequences as shown in [
IMPACT score IMPACT rating Consequence 100–174 Ineffective (I) Dismissal 175–249 ME Salary not advanced on salary schedule after first ME rating; dismissal after second consecutive ME rating 250–349 Effective (E) None 350–400 HE Bonus; eligible for permanent base-pay increase after second consecutive HE rating
-1 Note. ME = Minimally Effective; HE = Highly Effective.
During the period of our analysis, the average DCPS teacher attrition was 18% (Figure 2).7 [
Proportion of teachers exiting DCPS, by teacher performance and school poverty. Note. Teacher attrition indicates the average percentage of teachers leaving DCPS at the end of 2009–2010 through the end of 2011–2012. Exits combine voluntary and involuntary exits, where voluntary exits include resignations and retirements, and involuntary exits refer to teachers who were terminated due to performance. High-performers include teachers rated Effective or Highly Effective. Low-performers include teachers rated Ineffective or Minimally Effective. DCPS = District of Columbia Public Schools.
Proportion of exiting teachers who are high- or low-performing, by school poverty status. Note. Teacher attrition indicates the average percentage of teachers leaving DCPS at the end of 2009–2010 through the end of 2011–2012. Exits combine voluntary and involuntary exits, where voluntary exits include resignations and retirements, and involuntary exits refer to teachers who were terminated due to performance. High-performers include teachers rated Effective or Highly Effective. Low-performers include teachers rated Ineffective or Minimally Effective. DCPS = District of Columbia Public Schools.
This descriptive summary of retention highlights the challenges confronting DCPS to improve student achievement by improving the composition of its teacher workforce. Losing 13% of the best teachers each year places strong demands on teacher recruitment to prevent a reduction in achievement in those classrooms. However, exiting 46% of low-performing teachers creates substantial opportunity to improve achievement in the classrooms of low-performing teachers. In the remainder of this article, we explore how teacher turnover in DCPS under IMPACT affects student achievement. We examine this question in the aggregate and separately for low- and high-performing teachers. We also consider whether the relationship between teacher turnover and student achievement varies across schools and over time.
To examine the effects of teacher turnover on student achievement, we employ a panel-based research design that effectively compares how outcomes in school-grade cells changed following the exit of a teacher to the contemporaneous change in school-grade cells where no turnover occurred. We particularly want to understand whether teacher effectiveness and student achievement are higher or lower as a result of exiting teachers. Changes in overall teacher effectiveness depend upon the magnitude of mean difference in effectiveness between entering and exiting teachers and the proportion of teachers who turn over. Changes in student achievement depend on these differences and on the relationship between measured teacher effectiveness and student achievement. Our empirical model attends to these relationships.
To illustrate how our research design utilizes student- and teacher-level data, we begin with the commonly used specification of student-level achievement shown in [-1] . The achievement of student i in school s, grade g, assigned to teacher j and class c during year t ( A i s g j c t ) is a function of that student’s observables, including prior achievement (Xisgjct) and the attributes of classroom peers, X ¯ s g j c t , the teacher’s value-added (µ
A i s g j c t = β 0 + X i s g j c t β 1 + X ¯ s g j c t β 2 + μ j s g t + π s + τ t + ε i s g j c t .
To control for our student-level covariates, while facilitating further aggregation of this specification, we replace the dependent variable in [-1] with the student-level residuals ( A i s g j c t * ) obtained by regressing A i s g j c t on X i s g j c t . Aggregating the resulting equation to the teacher level, we have
A ¯ s g c j t * = β 0 + X ¯ s g c j t β 2 + μ s g c j t + π s + τ t + ε ¯ s g c j t * .
Consider the case of teacher j in a particular school and grade who was hired in year t to replace teacher j′ that left the school and grade at the end of the prior school year. [-3] shows the difference in average achievement of students taught by the entering teacher in the next year compared with that of the exiting teacher in the prior year:
Δ A ¯ s g j t * def _ _ A ¯ s g j t * − A ¯ s g j ′ t − 1 * = ( X ¯ s g j t − X ¯ s g j ′ t − 1 ) β 2 + μ s g j t − μ s g j ′ t − 1 + π s − π s + τ t − τ t − 1 + ε ¯ s g j t * − ε ¯ s g j ′ t − 1 * = Δ X ¯ s g t β 2 + Δ μ ¯ s g j t + Δ τ t + Δ ε ¯ s g j t * .
That is, [-3] models the change in student achievement (i.e., conditional on student traits) as a teacher-level function of the change in classroom peers, the change in teacher quality, and other unobserved time-varying changes. However, conducting the analysis at the teacher level would have several prohibitive limitations. For example, if the attrition of a teacher has negative consequences on the productivity of his or her grade-level colleagues ([
Δ A ¯ s g t * = Δ X ¯ s g t β 2 + Δ μ ¯ s g t + ω t + Δ ε ¯ s g t .
Our analysis aims to understand how student achievement changes as a function of teacher turnover, rather than as a function of changes in teacher value-added in [-4] . That is, teacher turnover may change teacher quality (e.g., Δµ
Δ A ¯ s g t * = Δ X ¯ s g t β 2 + γ 1 E s g t − 1 + ω t + Δ ε s g t * .
The identification strategy implied by this research design has a straightforward “difference-in-differences” logic.9 [
Our quasi-experimental specification unrestrictively controls for several unobserved determinants of student achievement. More specifically, this specification identifies the effect of teacher turnover controlling for time-invariant traits specific to each school-grade cell, time-varying traits shared across all schools and grades, and various student-level traits including prior achievement. However, the internal validity of the inferences based on this basic model still rests on several critical assumptions that we engage directly. First, our design implicitly assumes that students do not sort to (or from) turnover classes by switching schools in a way that biases the results. Second, as currently specified, our approach implicitly assumes that, when filling vacancies due to turnovers, schools do not manipulate teacher transfers within DCPS in a manner that biases turnover results. For example, when an exit occurs within a school, principals do not systematically move the most or least effective teachers from other grades to fill that vacancy. Although there are slight variations across years and subjects, on average 55% of replacement teachers come from outside the DCPS system, 34% transfer within DCPS schools, and 11% transfer across DCPS schools. Our specification also assumes that these teacher transfers have no achievement implications for the “sending” school-grade cell (e.g., due to disruption of teacher peers). Third, our design assumes that there are no important unobserved factors changing at the school- or grade level that influence student achievement and that are also correlated with turnover (e.g., increasingly effective principals).
To address the robustness of our results in the presence of these potential confounds, we modify our basic estimation approach and conduct several robustness tests. First, we add several additional controls to our empirical models to address potential challenges to internal validity. To address concerns that within-school or across-school transfers may influence our results, we also introduce direct controls for these transfers. To assess the relevance of unobserved school trends that are correlated with turnover, we also employ specifications that include school fixed effects. Time-invariant school effects have already been eliminated from our design as a result of first-differencing school-grade observations. Adding a school fixed effect to our first-difference specification implies that we are also controlling flexibly for school-specific changes over time (e.g., school trends in culture and leadership).10 [
Third, we estimate the effects of teacher turnover on teacher quality directly. If, as our conceptual model suggests, teacher quality is the mechanism through which turnover influences student achievement, we should observe consistent results for the effects of turnover on both teacher quality and student achievement. To provide increased assurance that any student achievement changes associated with teacher turnover reflect its effects on teacher quality, we estimate some specifications where we employ IMPACT scores as the dependent variable.11 [
We create three treatment variables to examine different types of teacher turnover. As before Esgt−
Δ A ¯ s g t * = γ 1 E s g t − 1 L + γ 2 E s g t − 1 H + δ S s g t − 1 + θ D s g t − 1 + Δ X ¯ s g t β 2 + ω t + φ s + ε s g t * .
Δ TQ ¯ s g t = γ 1 ′ E s g t − 1 L + γ 2 ′ E s g t − 1 H + δ ′ S s g t − 1 + θ ′ D s g t − 1 + ω t ′ + φ s ′ + ε s g t ′ .
Finally, we examine whether the effect of teacher turnover varies by year or school characteristics by interacting each treatment variable with the appropriate year or school-characteristic indicator variable (not shown). For instance, we assess whether the effect of teacher turnover differs between high- and low-poverty schools.
Our analysis draws on several sources of student, teacher, and school administrative data from DCPS. Students’ test scores, demographic variables, and teacher assignments come from DC’s Comprehensive Assessment System (DC CAS). These data span the 2009–2010 through 2012–2013 school years and include 56,564 student-years for tested students in Grades 4 through 8 with prior test scores.12 [
To construct our final analytical sample, we edited the data on the students and teachers in several conventional ways. First, we restricted our sample to general education classrooms, which resulted in dropping 12 special education campuses leaving 103 schools serving students tested in Grades 4 through 8. We then excluded students when they were tested in a grade other than their assigned grade (0.22% of student-year observations) or when they lacked a prior-year score (1.97% of student-year observations). To limit measurement error, we linked teachers to school-grade-year cells if the teacher is assigned to at least 10 tested students in that cell. This restriction eliminated 0.62% of teacher-school-grade-year observations. We also excluded teacher-year observations when those teachers taught in a school that closed at the end of that school year. This restriction also eliminated 0.62% of teacher-school-grade-year observations.
The primary outcome of interest is the year-to-year change in average residualized and standardized student achievement at the school-grade-year level.13 [
Math samples Reading samples Unrestricted Base Unbalanced Balanced Unrestricted Base Unbalanced Balanced Average student characteristics (N = 56,564 student-year observations) Students per s-g-y cell 50.6 51.2 51.0 51.1 50.6 51.4 51.7 52.8 (44.6) (45.1) (44.7) (44.9) (44.6) (45.0) (45.5) (46.7) Proportion male .51 .51 .51 .51 .51 .51 .51 .51 (.09) (.09) (.09) (.09) (.09) (.09) (.09) (.09) Proportion Black .77 .77 .77 .76 .77 .77 .77 .76 (.28) (.29) (.29) (.29) (.28) (.28) (.29) (.29) Proportion Hispanic .15 .15 .15 .15 .15 .15 .15 .15 (.21) (.21) (.21) (.22) (.21) (.21) (.21) (.21) Proportion LEP .08 .08 .08 .08 .08 .08 .08 .08 (.12) (.12) (.12) (.12) (.12) (.12) (.12) (.12) Proportion SpEd .18 .18 .18 .17 .18 .18 .18 .17 (.10) (.09) (.09) (.09) (.10) (.09) (.09) (.09) Proportion FRPL .69 .70 .70 .69 .69 .70 .70 .70 (.23) (.23) (.23) (.23) (.23) (.23) (.23) (.23) Residualized achievement −.02 −.02 −.02 −.01 −.03 −.03 −.03 −.03 (.23) (.23) (.23) (.23) (.18) (.18) (.18) (.18) Average teacher characteristics (N = 1,873 teacher-year observations) Teachers per s-g-y cell 1.68 1.68 1.68 1.70 1.86 1.86 1.86 1.89 (.86) (.87) (.87) (.88) (.93) (.93) (.93) (.95) Any exit .21 .21 .20 .19 .19 .20 .19 .19 (.36) (.36) (.35) (.34) (.34) (.35) (.34) (.34) High-performer exit .10 .10 .10 .10 .10 .10 .09 .09 (.26) (.27) (.26) (.25) (.26) (.26) (.25) (.25) Low-performer exit .11 .10 .10 .10 .10 .10 .10 .10 (.27) (.27) (.26) (.26) (.26) (.26) (.26) (.26) IMPACT score 283.5 283.7 283.7 286.3 284.8 285.2 285.2 286.6 (51.0) (51.2) (51.2) (50.7) (48.7) (48.7) (48.7) (48.4) Teaching experience 9.55 9.63 9.63 9.91 9.30 9.37 9.37 9.56 (6.92) (6.89) (6.89) (6.92) (6.77) (6.76) (6.76) (6.72) Average school characteristics Number of unique schools 100 97 97 88 100 97 97 90 % High-poverty 80 80.41 80.41 79.55 80 80.41 80.41 81.11 % Elementary 64 64.95 64.95 67.05 64 64.95 64.95 65.56 % Middle 14 13.40 13.40 12.50 14 13.40 13.40 14.44 % Senior high school 1 1.03 1.03 1.14 1 1.03 1.03 1.11 % Education campus 20 20.62 20.62 19.32 20 20.62 20.62 18.89 School-grade-year obs. 838 751 734 663 838 753 733 666
-2 Note. Unrestricted sample includes school-grade-year cells which contain nonmissing data for all variables in our model. The base sample restricts the sample to school-grade-year cells which contain nonmissing outcome data in 2 consecutive years (to form the first differences). The unbalanced sample further restricts to school-grade-year cells which contain both IMPACT scores and student achievement. The balanced sample is limited to school-grade cells which contain all 3 years of first differences. LEP = limited English proficiency; SpEd = special education; FRPL = free/reduced price lunch.
A final set of sample restrictions reflects concerns regarding missing data. First, differenced outcomes can only be created when the school-grade cell contains the outcome of interest in 2 consecutive years. This results in missing observations when schools open or close during the years of our analysis. This restriction produces school-grade-year cells with missing outcome data, which results in a loss of 87 school-grade cells in math (838 observations in the unrestricted sample to 751 in the base sample) and 85 in reading (838–753). Second, some school-grade-year cells are missing IMPACT scores, which results in different estimation samples for changes in IMPACT scores ([-7] ) versus changes in student achievement ([-6] ). Because we want to observe the effect of teacher turnover on teacher quality and student achievement in the same school-grade-year cells, we drop cells that are missing differenced IMPACT scores. This results in the loss of 17 school-grade-year cells in the math sample and 20 school-grade-year cells in the reading sample. The remaining sample is unbalanced, in that each school-grade cell is not observed in each year.
Third, we eliminate school-grade cells with fewer than 3 years of differenced outcomes. We are concerned that unbalanced observations introduce structural changes that influence estimates in ways that do not reflect responses to typical teacher exits. For example, school-grade cells may exist in some years but not others because schools close during the time frame of our analysis. In such situations, within-school, time-varying factors which we do not observe may influence student achievement and be correlated with teacher turnover, biasing our estimates. This restriction results in the loss of an additional 71 school-grade-year cells from the math sample and 67 school-grade-year cells from the reading sample, and creates the balanced sample.
[
The “treatment” variable in our setting is defined by the proportion of students in a school-grade-year cell experiencing different types of teacher turnovers.15 [
Our conceptual model suggests that the induced turnover of low-performing teachers (i.e., teachers rated by IMPACT as “Ineffective” or “Minimally Effective”) should result in improvements in teaching quality and student achievement, whereas the turnover of high-performing (“Highly Effective” and “Effective”) teachers may well result in a reduction in teacher quality and student achievement depending on the quality of entering teachers. The overall effect, which balances these two types of turnovers, is conceptually ambiguous and depends on the composition of exiting teachers and the quality of entering teachers.
Before turning to our estimates, it may be instructive to examine simple averages of the IMPACT scores of exiting and entering teachers. If our estimates, which control for a variety of potential confounds, are wildly different from these simple means, we would want to understand how our adjustments influence the outcomes. Figure 4 shows the unconditional means of IMPACT scores of all exiting and entering general education teachers (i.e., teachers of all subjects in tested and untested grades) in DCPS.16 [
Average IMPACT scores of all general teachers (IMPACT Group 1 and Group 2) by status of exiting teacher and year. Note. Results for 2011 indicate the average score for teachers who exited at the end of 2009–2010 compared with those entering in 2010–2011. Exiting scores are based on most recent IMPACT score. Scores of entering teachers are for all entering teachers as entering teachers cannot be linked to classroom of exiting teachers. Exits include teachers who retired, resigned, or were terminated. Teachers leaving schools that closed are excluded.
Average IMPACT scores of teachers who are matched to students with math achievement scores (IMPACT Group 1) by year. Note. Results for 2011 indicate the average score for teachers who exited at the end of 2009–2010 compared with those entering in 2010–2011. Exiting scores are based on most recent IMPACT score. Scores of entering teachers are for all entering teachers as entering teachers cannot be linked to classroom of exiting teachers. Exits include teachers who retired, resigned, or were terminated. Teachers leaving schools that closed are excluded.
Average individual value-added scores of teachers who are matched to students with math achievement scores (IMPACT Group 1) by status of exiting teacher and year. Note. Results for 2011 indicate the average score for teachers who exited at the end of 2009–2010 compared with those entering in 2010–2011. Exiting scores are based on most recent IMPACT score. Scores of entering teachers are for all entering teachers as entering teachers cannot be linked to classroom of exiting teachers. Exits include teachers who retired, resigned, or were terminated. Teachers leaving schools that closed are excluded.
Comparing the IMPACT scores of entering and exiting teachers suggests that teacher quality is improving as a result of teacher turnover. This is true whether teacher effectiveness is measured by overall IMPACT scores or by value-added. However, when teachers who are judged to be high-performing voluntarily exit, they are replaced on average by somewhat less effective teachers. Contrast that with the exit of teachers who are either forced to leave as a result of IMPACT or whose performance, if not improved, would lead to a forced exit. Turnover in this instance appears to result in a substantial improvement in measured effectiveness. As discussed above, there are a variety of reasons why these simple comparisons may misrepresent the effects of teacher turnover in DCPS. For example, the composition of students may have changed from one year to the next in a way that either favors or disadvantages teachers entering a school-grade cell which experienced teacher turnover. We now turn to the estimation of [-6] and [-7] , which control for a number of potentially confounding factors.
We report our main results (i.e., estimates based on [-6] and [-7] ) in [
Math Reading (1) (2) (3) (4) (5) (6) (7) (8) IMPACT score DC CAS IMPACT score DC CAS IMPACT score DC CAS IMPACT score DC CAS All exits 17.359* 0.079** 15.066* 0.046† (6.973) (0.03) (6.244) (0.024) High-performers −29.720** −0.055 −17.798* −0.047 (8.486) (0.039) (7.697) (0.034) Low-performers 63.838** 0.210** 46.129** 0.136** (8.071) (0.041) (7.987) (0.03) Student controls X X X X Observations 663 663 663 663 666 666 666 666 R2 .035 .015 .138 .045 .035 .017 .087 .04
-3 Note. Robust standard errors reported in parentheses. All models include year fixed effects and controls for teacher movement within and across schools. Student controls account for the year-to-year, across-cohort change in the percentage of students in a school-grade-year cell who are Black, Hispanic, other non-White race/ethnicity, limited English proficient, special education, or FRPL eligible. DC CAS = District of Columbia Comprehensive Assessment System; FRPL = free/reduced price lunch.
-4 p < .10. *p < .05. **p < .01.
In the remaining rows of [
In contrast, the exit of low-performing teachers substantially increases both teaching quality and student achievement. In math, the exit of low-performing teachers is estimated to improve teaching quality by 64 IMPACT points (1.3 SD) and student achievement by 0.21 SD. The effects on reading are somewhat smaller but still large, 46 IMPACT points and 0.14 SD of student achievement. Over the first 3 years of IMPACT, replacing teachers identified by IMPACT as low-performers leads to substantial improvement in student achievement as, on average, their replacements are meaningfully more effective teachers.
These estimates reflect the effect on student achievement if all teachers in a school-grade cell were of the identified type, for example, low-performing, and exited, and thus would overstate the effect on all the students in that school-grade cell if a low-performing teacher left a school-grade cell and the other teacher(s) in that cell remained. Alternatively, assuming no spillovers from one classroom within a grade to another, these estimates capture the average effect on the students in the exiting teacher’s classroom. A strength of our approach is to capture such spillovers.
The consistency of the effects of turnover on teacher quality and student achievement and their robustness to introducing student controls increases our confidence in the internal validity of our estimates. Nonetheless, legitimate concerns may remain that parents or principals may systematically respond to teacher turnover by altering the assignment of students to teachers in ways that threaten internal validity. For example, if turnover predicts changes in student attributes, it may signal strategic behavior by parents or principals that may bias our results. Fortunately, we find nothing of concern when we regress a variety of student characteristics on teacher turnover (Online Appendix Table 4, available in the online version of the journal). Of the 18 estimated coefficients (six student attributes by three types of teachers [all, high-performing, and low-performing]), only one is significant at conventional levels. The exit of all high-performing teachers from a school-grade cell is associated with a 2.4% decrease in limited English proficiency (LEP) students. These results suggest that there is not systematic sorting of students to teachers in response to turnover, and when there is some evidence, the magnitudes are modest. Nonetheless, we include controls for all student variables we can observe.
Another potential threat to the validity of our estimates may be that underlying trends in schools may cause student achievement to increase over time in school-grade cells with turnover but not in school-grade cells without turnover. To address this issue, we estimate first-difference models that introduce school fixed effects and models which include school-by-year fixed effects. The identifying variation for estimates with school fixed effects comes from within-school comparisons of school-grade cells with and without turnover. Adding a school-by-year fixed effect effectively limits our comparisons to grades in the same school and year with and without turnover. Estimates for our base models and those with school and school-by-year fixed effects are shown in Online Appendix Tables 5 (math) and 6 (reading; available in the online version of the journal). Adding school fixed effects to our base model changes the estimates only slightly. The one substantive change is the effect of a typical teacher exit on math student achievement. The coefficient is somewhat smaller (0.058 SD rather than 0.079 SD), the standard error larger (0.038 rather than 0.030), the combination of which results in a statistically insignificant estimate. Adding school-by-year fixed effects has a larger effect on some of the estimates. In math, while still significant and educationally meaningful, the effect of turnover of low-performing teachers on achievement is about half as large as in either of the other two models. In reading, the change is not nearly so dramatic. Adding school-by-year fixed effects substantially reduces the identifying variation in ways that have important implications for the identification of effects and for external validity. For example, 663 school-grade cells contribute to identifying the effects of our preferred specification in math ([
We include additional robustness checks in which we estimate the effects of two “placebo” models. In the first, turnover at the end of 2012–2013 is used to predict changes in student achievement from 2009–2010 to 2010–2011. If turnover is the mechanism that drives our results and not some other attribute of the school-grade cells that experience turnover, then the effects of the placebo estimates should not be similar to the estimates presented in [
In the second “placebo” test, we predict student achievement as before—a function of turnover in the same grade cell in the prior year—but also include turnover in the next higher school-grade cell. For example, in observations considering changes in achievement in fourth-grade cells we also include the value of turnover for fifth-grade cells in the same school. If turnover in the next-grade cell predicts achievement in the current grade, we might be concerned that turnover signaled something about the school rather than turnover per se. It is possible that, because teachers may work together across grades, turnover in fifth grade could influence achievement in fourth grade in the following year. Because our analysis is premised on school-grade cells as units of observation, about 40% of our original sample is unavailable when we include turnover in the next school-grade cell as a control.17 [
As shown in Online Appendix Table 8 (available in the online version of the journal), next-grade turnover has no effect on the change in current-grade achievement, and the coefficients of current-grade turnover are robust to the inclusion of prior-grade turnover. Column 1 shows the effect of turnover in the current grade on math achievement in the succeeding year. This is the main result from the article for this smaller sample of school-grade cells. Column 2 shows the estimates for both current-grade turnover and next-grade turnover. The estimates of the effects of current-grade turnover remain largely unchanged, and the estimate of next-grade turnover is not significantly different from 0. The results for reading are qualitatively similar although for this reduced sample the main effect is insignificant.
There are several other ways in which the effects of teacher turnover may be heterogeneous. For example, the contexts across low- and high-poverty schools are likely to shape both the prevalence of teacher turnover and its effects on students. Overall, we find that high-poverty schools appear to improve as a result of teacher turnover. We estimate that the overall effect of turnover on student achievement in high-poverty schools is 0.084 in math and 0.052 in reading. Both estimates are statistically distinguishable from 0 ([
Math Reading (1) (2) (3) (4) (5) (6) (7) (8) IMPACT score DC CAS IMPACT score DC CAS IMPACT score DC CAS IMPACT score DC CAS All exits Low-poverty 21.714 0.006 1.738 −0.038 (19.301) (0.082) (8.727) (0.043) High-poverty 16.793* 0.084** 16.032* 0.052* (7.259) (0.03) (6.548) (0.025) High-performer exits Low-poverty 23.648 −0.004 1.922 −0.041 (21.962) (0.097) (9.892) (0.05) High-poverty −39.234** −0.064 −20.437† −0.048 (−8.596) (0.042) (8.575) (0.038) Low-performer exits Low-poverty NA NA NA NA High-poverty 64.075** 0.209** 46.761** 0.138** (8.171) (0.041) (8.104) (0.03) Student controls X X X X Observations 663 663 663 663 666 666 666 666
-5 Note. Robust standard errors reported in parentheses. All models include year fixed effects and controls for teacher movement within and across schools. Student controls account for the year-to-year, across-cohort change in the percentage of students in a school-grade-year cell who are Black, Hispanic, other non-White race/ethnicity, limited English proficient, special education, or FRPL eligible. We do not include estimates for low-performer exits in low-poverty schools as these are found in only three school-grade cells. FRPL = free/reduced price lunch; NA = not applicable.
-6 p < .10. *p < .05. **p < .01.
DCPS appears to be quite capable of replacing exiting high-performing teachers in low-poverty schools with comparable teachers ([
Forty percent of teacher turnover in high-poverty schools is among low-performing teachers (Figure 3). Our estimates indicate that there are consistently large gains from the exit of low-performing teachers in high-poverty schools. In math, teacher quality improves by 1.3 SD and student achievement by 20% of a standard deviation; in reading, these figures are 1 SD of teacher quality and 14% of standard deviation of student achievement. In DCPS, virtually all low-performing teacher turnover is concentrated in high-poverty schools: on average, 1% of students in low-poverty schools experience low-performing teacher turnover.18 [
When we examine the effects of DCPS turnovers over time, we observe substantial consistency as well as a few interesting differences. Overall, the effects of DCPS turnover appear to become increasingly positive year to year. However, student achievement is estimated to be unaffected until 2013 when for math ([
Math Reading (1) (2) (3) (4) (5) (6) (7) (8) IMPACT score DC CAS IMPACT score DC CAS IMPACT score DC CAS IMPACT score DC CAS All exits 2011 13.053 0.092 2.113 −0.039 (11.876) (0.061) (9.113) (0.043) 2012 20.706 0.025 23.396** 0.046 (14.355) (0.059) (10.354) (0.039) 2013 20.017† 0.112** 20.258† 0.105* (11.429) (0.041) (10.895) (0.041) High-performer exits 2011 −15.404 −0.022 −38.553** −0.161** (13.793) (0.071) (9.513) (0.054) 2012 −53.682** −0.277** −9.302 −0.042 (15.312) (0.059) (12.900) (0.049) 2013 −21.426 0.057 −9.969 0.008 (13.399) (0.057) (11.766) (0.059) Low-performer exits 2011 43.824** 0.215* 31.473** 0.050 (14.939) (0.100) (10.267) (0.049) 2012 74.931** 0.243** 54.623** 0.133** (16.914) (0.067) (12.376) (0.050) 2013 70.166** 0.179** 53.750** 0.208** (12.678) (0.051) (15.822) (0.047) Student controls X X X X Observations 663 663 663 663 666 666 666 666
-7 Note. Robust standard errors in parentheses. All models include controls for teacher movement within and across schools. Student controls account for the across-cohort change in the percentage of students in a school-grade-year cell who are Black, Hispanic, other non-White race/ethnicity, LEP, special education, or FRPL eligible. LEP = limited English proficiency; FRPL = free/reduced price lunch.
-8 p < .10. *p < .05. **p < .01.
For most years, the exit of high-performing teachers does not influence teacher quality or student achievement. However, in one year for math (2012) and reading (2011), the exit of high-performing teachers has a substantial negative effect on teaching quality and student achievement. These estimates are similar across alternative analytic samples that employ the base and unbalanced data. When we examine the individual exiting and entering teachers in the school-grade cells with teacher turnover, we observe the exit of several very effective teachers who are replaced by teachers whose subsequent performance places them among the low-performers.
In contrast, the exit of low-performing teachers yields consistently large improvements in teaching quality and student achievement in math (0.18–0.24 SD of student achievement) and increasing effects over time in reading (0.05 [not significant] to 0.21 SD of student achievement). In almost every year, DCPS has been able to replace low-performing teachers with high-performing teachers who have been able to improve student achievement.
Finally, we examined differences in the effects of turnover between elementary and middle school grades. For math, we find no statistically significant differences in the effects of turnover in elementary and middle school grades for either teacher effectiveness or student achievement. This is true for the overall model and for models that estimate effects for low- and high-performing teachers. For reading, the results are similar with the exception that when a low-performing teacher exits an elementary grade, teacher effectiveness increases substantially more than for a similar exit from a middle school grade (there is not corresponding increase in student achievement). When we divide our sample this way, our sample sizes are reduced, which may limit our ability to discern differences. These results are available from the authors.
In general, higher rates of teacher turnover are legitimately thought to negatively influence student outcomes (e.g., [
The high stakes associated with IMPACT have been controversial, both within the District of Columbia as well as in broader discussions of education policy. There are elements of both sides of this debate in our estimates. While we are unable to identify high-performing teachers who leave DCPS because of IMPACT, our estimates indicate that replacing high-performing teachers who exit with teachers who perform similarly is difficult. In general, such turnover does not lead to statistically significant reductions in student performance, except in one notable instance (i.e., math teachers in 2011–2012).
Alternatively, IMPACT targets the exit of low-performing teachers. Our estimates show that doing so substantially improves teaching quality and student achievement in high-poverty schools. An improvement of 20% of a standard deviation of student achievement in math is roughly equivalent to 35% to 65% of a year of student learning, depending on grade level ([
We should note that our analysis does not have the causal warrant of an experimental design. Nonetheless, under certain identifying assumptions that we articulate and examine, our quasi-experimental design does identify the change in student achievement caused by teacher turnover. However, we do not claim that IMPACT caused all of the teacher turnover we observe. Although IMPACT certainly caused some teachers to leave DCPS through dismissals, voluntary teacher attrition is likely driven by myriad teacher preferences.19 [
Our empirical results were not inevitable, even for the turnover of low-performing teachers. As [
The challenge of improving the composition of teachers in DCPS is increasing. First, as the least effective teachers exit, there are fewer such teachers to exit over time, and we would expect the average effectiveness of exiting teachers to continue to increase. Second, in 2012–2013 DCPS adjusted its evaluation system, so that to be rated as “Effective” or better (and thus avoid sanctions) teachers needed IMPACT scores of at least 300 rather than 250 as had been true in 2011–2012. Increasing the threshold for high-performing status will likely lead to the exit of some previously “Effective” teachers who are now classified as “Developing” and may cause some “Effective” and “Highly Effective” teachers to leave as they perceive the system as more stressful. However, DCPS made several other changes to IMPACT in 2012–2013 that may cause the system to be more hospitable, such as reducing the number of teacher observations, increasing access to bonus and base-pay increases, and reducing the weight of value-added for Group 1 teachers.
We expect that both the declining numbers of very low-performing teachers and changes in the IMPACT rating thresholds place strong demands on the system to continue recruiting effective teachers to replace the exit of higher-performing teachers. Figure 1 presents some early evidence of these trends. The teachers exiting at the end of our study window were noticeably more effective than those exiting after IMPACT’s first year (i.e., by about 40% of a teacher-level standard deviation). However, over this same period, the performance of entering teachers also grew appreciably (i.e., 28% of a standard deviation). These trends appear unrelated to the average experience of entering and exiting teachers, which, throughout this period, remained relatively constant at 3.5 and 7 years, respectively. As long as DCPS continues to recruit more able teachers than it loses, compositional change will likely lead to increased student achievement. Whether DCPS can reap further performance benefits from compositional change in its workforce as it increases performance standards appears plausible but remains to be seen. Regardless, our results indicate that, under a robust system of performance assessment, the turnover of teachers can generate meaningful gains in student outcomes, particularly for the most disadvantaged students.
We are grateful to the District of Columbia Public Schools (DCPS) for the data employed in this article and to Scott Thompson, Kim Levengood, Alden Wells, and Luke Hostetter of DCPS for addressing our questions regarding the data and IMPACT. We appreciate comments from John Friedman, Steve Glazerman, Hamp Lankford, Luke Miller, and seminar participants at Stanford University, the University of Virginia, Michigan State University, and Vanderbilt, as well as conference attendees at APPAM and AEFP. We also received helpful suggestions from two anonymous reviewers.
AuthorsMELINDA ADNOT, PhD, is a visiting assistant professor of educational policy at Davidson College, North Carolina. Her research interests include educational policy analysis, teacher evaluation, and teacher quality.THOMAS DEE, PhD, is a professor of education at Stanford University and a research associate in the Programs on Economics of Education, Health Economics and Children at the National Bureau of Economic Research (NBER). His research focuses largely on the use of quantitative methods (e.g., panel data techniques, instrumental variables, and random assignment) to inform contemporary policy debates. Recent examples include econometric evaluations of incentive- and accountability-based reforms and an analysis of recent, stimulus-funded, school-turnaround initiatives.VERONICA KATZ is a doctoral student in education policy at the University of Virginia. Her research focuses on teacher quality and retention, especially in low-performing schools.JAMES WYCKOFF is the Curry Memorial Professor of Education and Policy, and director of the EdPolicyWorks, a research center at the University of Virginia. His research focuses on providing evidence for improving teacher quality through a variety of mechanisms including teacher preparation, recruitment, assessment, and retention. Currently, he is working with colleagues to examine the effects of teacher evaluation in the District of Columbia and the ways in which principals improve teacher quality in New York City.
By Melinda Adnot; Thomas Dee; Veronica Katz and James Wyckoff