I recently stumbled across Blackboard Grade Centre Item Analysis Discrimination. No, I hadn't seen this before either.
So is this the answer to construction of good (valid) online MCQs?
Blackboard's (sketchy) description of how the tool works is that "Discrimination correlates question performance with overall performance on the test". Hang on - if the categories are not independent, is this analysis valid? The Discrimination score tells you whether good students (those who score highly on the test) do better than poor students, or whether poor students (who get low scores on the test) answer the question correctly. Is this a valid basis for judging the value of a question? In my hands, the tool also fails to calculate scores for some questions for reasons not given. So is it better than nothing? Well, possibly. Is it a valid analysis of question utility? You decide.
Get ready for lots more MCQs in the coming years. And keep lying awake at night worry about using MCQs. In the future, we will all be Sebastian Thrun.