Pages

Monday, December 09, 2013

Can you trust MCQs?

Blackboard logo With rising student numbers and increasing pressure on feedback turnaround times it is almost inevitable that we will see the increased use of online MCQs, at least for formative if not for summative assessment. But can you trust MCQs?

I recently stumbled across Blackboard Grade Centre Item Analysis Discrimination. No, I hadn't seen this before either.


So is this the answer to construction of good (valid) online MCQs?

Blackboard's (sketchy) description of how the tool works is that "Discrimination correlates question performance with overall performance on the test". Hang on - if the categories are not independent, is this analysis valid? The Discrimination score tells you whether good students (those who score highly on the test) do better than poor students, or whether poor students (who get low scores on the test) answer the question correctly. Is this a valid basis for judging the value of a question? In my hands, the tool also fails to calculate scores for some questions for reasons not given. So is it better than nothing? Well, possibly. Is it a valid analysis of question utility? You decide.

Get ready for lots more MCQs in the coming years. And keep lying awake at night worry about using MCQs. In the future, we will all be Sebastian Thrun.




3 comments:

  1. We use MCQs extensively for formative and summative assessment and find discrimination scores (or quintile bar charts) valuable for quality control.

    Anomalous discrimination scores often indicate problems with a question, so are a useful indicator that the question and responses should be double checked.

    The wrong answer may have been indicated as being correct, or the question may include ambiguous wording, or exceptions to a general statement may be apparent to students who have read outside the material covered in lectures.

    ReplyDelete
    Replies
    1. Hi Moira.
      Do you run your question stats in Blackboard, and if not, how do you collect and process the data?

      Delete
    2. The version of Blackboard that we are currently running does not provide exactly the analysis we want, so we process in Excel. For offline tests, our OMR software provides bar charts which serve a similar purpose.

      Delete