Via Slashdot, we find this excellent deconstruction of the bad math behind many so-called “hard” tests.

As one who took several medical licensure and specialist exams, and the Virginia bar exam, passing all, I might be inclined to pat myself on the back, but my former background as a mathematician won’t let me do that. I do remember, however, some remarks from a noted orthopedic surgeon about his own specialty exam: “It was a hellishly hard test, and went on for hours,” he said, “but I’m really glad I passed the first time I took it. Only about 35 percent who took it passed the exam.”

He was describing, with only the slightest tinge of boastfulness, the qualifying exam for specialists in orthopedic surgery. Passing the exam entitled one to join the “college” of orthopedic surgeons, and list oneself as specialist.

“Was it all multiple choice?” I asked. “And how did they grade it?” I was thinking of my own exams. “Did they count only the right answers.?”

When he said Yes to all the questions questions, I did not have the heart to tell him what I knew as a mathematical certainty–that the exam was, like most graduate medical exams, and large parts of legal licensing bar exams in most states , virtually a complete fraud.

Ouch. What the author is driving at is simple: unless there’s some penalty for guessing, “very hard” tests aren’t good measures of anything. This, as you may recall, was a key difference in scoring between the SAT and the ACT at one time (and may still be).