Image result for moneyball
It seemed to work till everyone was doing it. Then, not so much.

Moneyball was the book that described the transformation that grabbed professional baseball by the throat and said the old way of determining a player’s worth, the batting average, wasn’t enough. Baseball is the ultimate sport of statistical measures and the argument was that better and more measures, put through a meat grinder of a formula, could produce a magic number that would tell managers, front office suits, owners, and fans the value any given ballplayer added to the team.

Sounds familiar, doesn’t it?

But with a difference. The statisticians attempted to bring more measures into the formula because they realized that batting average, the number of hits divided by the number of times at-bat ignoring walks, did not capture the full measure of the production a player brought to the team.

So they expanded the measure. Instead of batting average, they would calculate a measure that included on-base percentage, slugging average (unlike the batting average, the slugging average gives credit for extra-base hits), and other stats deemed relevant.

Ben Orlin, in his book Math with Bad Drawings (Black Dog & Leventhal Publishers, a division of Hachette Book Group, 2018) mentions a precursor to the Moneyball theory, published by Life Magazine August 2, 1954, a statistical measure developed by Branch Rickey, general manager of the Pittsburgh Pirates at the time:

Lots of inputs, for sure. You might guess most of them, but what’s up with that F at the end?

In his endnotes to a subsequent chapter, Orlin also gives us the formula for rating NFL quarterbacks:

Multiple inputs, but what’s up with those fractions at the end?

Just for laughs (not really, GOT is going somewhere with this post), let’s look at Florida’s VAM formula for teachers:

Much better: a formula I can really understand. Florida only needed 24 pages to explain it.

What do these formulas have in common?

  1. Each one tries to reduce a complex performance to a single number for the purpose of determining who is better than whom.
  2. Each one is selective in what is measured. Each one ignores other important aspects of performance. For baseball, the formula ignores the defensive contribution of a player to a team. For example, a shortstop may be low in terms of offensive output but is outstanding in making outs and turning double plays. A low batting catcher may have a strong arm that no one attempts to steal a base.
  3. None of them captures the intangibles that contribute real value to performance. How do you measure leadership in the locker room? An ability to resolve player disputes to keep the team cohesive? The non-department chair that other teachers go to for help and advice? Statistics don’t capture these contributions.
  4. Nor do these formulas capture the importance of public relations. How much value does the ballplayer willing to sign balls for kids at the fence add to the team versus the player who will only sign at a convention where he earns hundreds of dollars to ink a ball? How much value for the teacher who knows how to work with parents and how to represent a school in public to the credit of all? These formulas can’t capture these vital but non-numerical contributions.
  5. Each one seems to feature some arbitrary feature that makes it hard to interpret. The maximum score a quarterback can get is 158 1/3. One-third? Seriously? As for teachers, the score is equally meaningless. It means how far above the prediction of student performance the teacher reached. But that depends upon the prediction being valid.

How do these formulas differ?

  1. The sports formulas try to include as many measures as possible. In contrast, the teacher VAM formula has only one input: test scores. Don’t be fooled by the many variables. They are an attempt to control for the many factors that affect student test performance much as the quarterback formula throws in a multiplier and an addend that make as much sense as picking numbers out of the sky. 158 1/3? Seriously?
  2. Replication. Anyone having the patience to look up sports stats can calculate the formulas for themselves and verify reported numbers. Anyone trying to do so with teacher VAM will run into a wall. While state departments of education are happy to provide the numbers to media intent on teacher shaming, they will not provide their calculations so that people can double-check their results.
  3. Legitimacy of the data. Want a better Moneyball rating? Get on base more often. Want a better NFL rating? Stop with the interceptions, make better passes for more yards. Want a better VAM rating? Game the test.
  4. In the end, sports is a competitive endeavor. Only one team can be the champion. Fortunes are won and lost as teams seek the players who will take them to the summit. Education is a cooperative endeavor in which we seek to make everyone the champion. It is not a zero-sum game and resists a single measure of quality.

Don’t expect the uber-wealthy and their politicians to understand this. As the current Secretary of Education once said, “Money is how we keep score.”

Moneyball. It worked for the Oakland As until other teams caught on. It has never worked in education and never will.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s