Principals Can't Tell Who the Good Teachers Are

As you probably know, Memphis is one of the sites where the Gates Foundation's Measures of Effective Teaching Project (MET) is researching some central questions about how we can figure out who effective teachers are.  This puts Memphis in what should be the awkward position of making policy decisions based on as-yet-incomplete research.

Well, it's probably a good thing, then, that the Gates Foundation has finally finished the last stage of its research.  Here's the executive summary.  You know, since districts and states across the country have made huge, mainly controversial changes on the basis of what the Gates Foundation was pretty sure the research was going to say.

The Huffington Post has a story about the research results.  (You can compare the HuffPo report to  the glowing review of the research in the Commercial Appeal.)  HuffPo summarizes the results as "teacher observation less reliable than test scores" in measuring effective teaching.  The Gates Foundation research has now found that effective teaching can be measured - through some combination of teacher observation, test scores, and student surveys.  The basic methodology is that teachers had a baseline year to establish their effectiveness (2009-10) - the Foundation calls this "produc[ing] estimates of teaching effectivness for each teacher."  The report states:  "we adjusted those [original] measures [of teaching effectiveness] for the backgrounds and prior achievement of the students in each class.  But, without random assignment, we had no way to know if the adjustments we made were sufficient to discern the markers of effective teaching from the unmeasured aspects of students' backgrounds."  So the next year, academic year 2010-11, "we then randomly assigned a classroom of students to each participating teacher."

By "background", the Foundation is obliquely referring to students' socioeconomic backgrounds.  The researchers at RAND (the ostensibly independent corporation that Gates Foundation hired to conduct the research) hoped to answer two questions:  "First, did students actually learn more when randomly assigned to the teachers who seemed more effective when we evaluated them the prior year?  And second, did the magnitude of the difference in student outcomes following random assignment correspond with expectations?"  The short answer, according to the RAND researchers, is yes and yes.

Interesting.  Should we be surprised that the Gates Foundation research finds what the Gates Foundation thought it would find?  Or that it cost $50 million to find it out?

A couple of things jump out at me.  The first has to do with the role of teacher observations and the principals who conduct them.  Traditionally, teachers were judged only on the basis of their principal's once-every-several-years announced-in-advance observation.  Teachers generally performed very well in these observed lessons.  Where students did not make "adequate yearly progress" but teachers were uniformly rated as effective and highly effective, critics found a disconnect.  The solution has been to require that student standardized test results be included in teachers' overall assessment.  So - no surprise that the researchers have found that principals actually do not adequately assess their own teachers in the classroom observations.  Districts have tried a number of methods to "norm" their evaluators - they have to take a class before they can be evaluators, they have to grade a video of a teacher within an accepted range of how the researchers have "objectively" graded the video, etc.  But despite all of that "norming", principals' observations are viewed as unreliable.

Now the summary doesn't just come out and say that - instead, the summary states that "adding a second observer increases reliability significantly more than having the same observer score an additional lesson."  The researchers go on to qualify this statement in three ways.  First, they acknowledge that it may be too expensive to have enough observers if they have to observe full-length lesson, and so portions of lessons could be observed. 

The second and third qualifying statements both have to do with principals, and embedded in the second is what underlies the Foundation's concern:  "although school administrators rate their own teachers somewhat higher than do outside observers, how they rank their teachers' practice is very similar and teachers' own administrators actually discern bigger differences in teaching practice, which increases reliability."  Essentially, principals do not objectively recognize effective teaching, but they can recognize how teachers within a school stack up against each other.  Talk about a back-handed compliment.  So even if principals don't actually know if they have a good teacher, if they observe all of the teachers at their school, then they can tell you who is better than who.  So the principals' observations still have some value.

But then there's the third qualifying statement - the crux of the Foundation's concern:  "adding observations by observers from outside a teacher's school to those carried out by a teacher's own administrator can provide an ongoing check against in-school bias."  There's the there.  Principals can't be trusted to adequately judge their own teachers.  If you don't believe the Gates Foundation, just ask Tennessee's Commissioner of Education Kevin Huffman, who says that Tennessee's teachers observation scores are consistently inflated. 

I have mixed feelings about principal evaluations to start with - I'm not really all that opposed to administrators from outside the school doing the observations.  I've known enough bad principals - who unfairly reward their favorites and unfairly criticize their less favorites that I've viewed that as a bigger problem than generalized inflation.  I guess I assumed that principals could objectively observe teachers, but chose not to.  May be time to re-think that.  But I do think that principals do understand teachers' fears about the new evaluation systems that so heavily weight student achievement, and the principals do, after all, walk a fine line between supervision and maintaining working relationships with their subordinate teachers.  Therefore, some inflation - just for good employee relations - should not be all that surprising.  Principals don't always get a lot of training in how to handle walking that fine line, and they are human.  And we know that principals are not selected because they are the best teachers, though they are now required to be instructional leaders whatever their teaching success.  Still thinking about this one - but suspicious of a "measurement" that encourages the devaluation of the observations of the one person that most likely understands the teacher, the students, and the realities of teaching and learning in a particular school - and in some neighborhoods, the dangers of just getting to school.  Teachers - how do you look at your principals in this particular role?

The second thing that jumps out at me is the methodology.  So the idea is that a teacher has her principal-assigned class the first year, and the RAND randomly-assigned class the second year to see if the teacher can meet or exceed the expectations based on their baseline year.  But these aren't just any students who are assigned - these are students that attend the same school.  Teachers are not assigned new schools, just new kids.  Why is this important?  Because - at least in Memphis - we use "neighborhood schools".  Generally, schools serve kids that live near the school.  So the students are generally going to be more similar to each other than dissimilar.  At least in Memphis, our neighborhoods are still fairly racially segregated, and definitely defined by socioeconomic status. 

So let's tease that out.  If you're a good 4th grade teacher at Richland Elementary, you'd probably be a good 3rd grade teacher there, too.  And you would likely do well with a different set of 4th graders at the same school.  But let's acknowledge that a 4th grade teacher at Richland Elementary could be hard-pressed to be an equally as great 4th grade teacher at Coro Lake Elementary.  The Gates Foundation acknowledges that "as a practical matter," they could not "randomly assign students or teachers to a different school site."  I think what students the teacher teaches matters as much as how well the teacher teaches - and while value added measures get closer to this issue than traditional did-the-student-meet-this-benchmark testing - none of the testing addresses how what is happening in students' lives outside of the classroom impacts what is possible inside of the classroom.

Underlying this all is my fundamental disagreement that a student's achievement should be measured based on that student's performance on any particular standardized test.  We know, from the dozens of studies on the SAT and the ACT, that the single best predictor of a student's performance is that student's socioeconomic background.  I'm just not convinced that standardized tests measure how well a teacher taught something, or even how well a student learned something.  And they definitely do not measure a number of other things that are hugely important, including creative and critical thinking.  Don't get me wrong - student achievement is hugely important on its own - I'm just not sold that the testing products being sold to our school districts are the correct tool to measure it.

Still more to read in this executive summary - working to get the full report.  I'm very interested in what the Gates Foundation calls its ambiguous results on "peer effects" (the demographics and average prior achievement of each student's classmates), as well as the "trade-offs" of the different models for how we can weight the various measures of teacher effectiveness - for example, test scores as 50% versus 33% versus less.  Let me know what you think about it.