deconstructing merit pay

Ryan at I Thought a Think explains why merit pay is never a simple matter.

So, who is your Most Valuable Teacher?

Is it Teacher A, who added the most value to her class over the course of the year?
Is it Teacher B, who had more of her kids meet the year-end goal?
Is it Teacher C, whose class scored the highest in the spring?
Is it Teacher D, who turned around more failing kids than any of the others?

"Value" is a homophone; there's the value signified by the numbers, but there's also the values of the school, the district, and the state which have to be superimposed atop any effort to link the data to the teacher. If the incentive pay/merit pay/whatever pay in this case goes to only one of the four teachers, you're making a statement about the value of the work the other three did, and it's a pretty lousy thing to say to the other three who also made progress that their success didn't matter as much.

And, of course, there are even more fundamental assumptions at work:
1. That the differences across teachers are statistically significant. (In small sample sizes, chance is magnified.)
2. That there are no mitigating factors that better explain students' growth within and across classes. (How much is due to good ol' maturation? Are all relevant factors controlled for?)
3. That the test measures something important.

We run into more trouble when we deal with mobile populations, or when we consider high school teachers who see their students for only fifty-five minutes a day in a single subject.

This is not to say only nay to the prospect of performance pay for teachers. I'm sure with today's data collecting and crunching powers, some magic formula can be worked out--something akin to the Netflix prize for education--but as Ryan shows, first we have to agree on what we actually value.