Multiple-choice question

From Learning
Jump to: navigation, search

Definition

A multiple-choice question (MCQ) is a question where multiple choices (also called options) are provided, and the respondent is expected to select one or more of the options based on the question.

Multiple-choice questions could be:

  • questions where respondents are expected to pick a single response
  • questions where respondents are expected to pick one or more options in response

Further, they could involve:

  • a pre-specified correct answer option (or option set) against which the respondent's option is compared
  • a survey multiple-choice question, that is intended to get information about the respondent's preference or individual-specific information. In this case, there is no right answer.

This page is not about survey MCQs. Rather, it is about questions where there is a single specific correct answer. For clarity, we will restrict attention to the case where respondents are expected to pick a single response. Cases where respondents need to pick sets of options can be reduced to this case through a mathematical equivalence.

Methods of administration

  • MCQs may be administered as part of homeworks, class quizzes, or tests, to be done individually (without discussion), individually (with discussion), or in groups.
  • MCQs may be brought up in class discussions and students may be polled for their responses using clickers or more primitive methods.

Advantages

Rapid and unbiased scoring

Multiple-choice questions can be scored rapidly by hand or by a scantron device, and in bulk. For instance, a 10-question MCQ administered by hand can be scored by an individual who memorizes the string of answer options in less than a minute per student attempt. It can also be scored by entering the data into a spreadsheet that can automatically compute scores, again with less than a minute per student attempt. Further, even in the case of manual scoring, the scorer need not have subject matter knowledge, so that the task may be outsourced to a less skilled (and cheaper) worker.

Easy computerized recording and summary analysis of responses

MCQs can easily be administered by computer, but even those administered by hand can be quickly transcribed to a computer using a suitably set up spreadsheet or database. Simple spreadsheet commands can be used to calculate the frequency of specific options chosen by students on specific questions. A similar summary analysis for free response or essay questions is difficult, because: (i) entering the data in a standardized format is hard, (ii) the analysis requires identifying common features between answers.

Question types

Questions with a clear objective answer that can be computed without looking at the options

These are questions where the question statement itself allows a person with subject matter knowledge to arrive at the correct answer, that can then be compared (without subject matter knowledge) to the options presented to select the correct option. An example is the question:

What is
(A) 401
(B) 471
(C) 501
(D) 541
(E) 611

For this question, a person solving the question could first compute the product of 47 and 13. Once the person arrives at the answer (which happens to be 611), the person can then compare it with the existing options, and choose Option (E). Note that the step of comparing with the existing options requires no subject matter knowledge.

This question type is to a large extent substitutable with a non-MCQ asking the same question. The use of a MCQ plays the important role here of making the question somewhat easier:

  • The form factor of the answer can be deduced by looking at the options.
  • Careless errors that do not result in arrival at one of the other answer options can be detected and potentially corrected.
  • It may be possible to use approximate estimation techniques that are insufficient to obtain a precise answer but sufficient to narrow down within the range of options. For instance, one might reason that should be close to , and therefore pick 611, without doing any calculation. The extent to which approximate estimation works depends on the nature of the other options. (For instance, in the above question, the technique of looking at the last digit does not work because all options have the same last digit of "1").

Questions that require evaluating existing options until one reaches an option that satisfies the question criterion

These are questions where the question statement itself does not permit one to determine the correct answer. Rather, the options need to be read and evaluated. For instance:

Which of the following statements is false about mammals?
(A) Mammals are vertebrates
(B) Mammals are warm-blooded
(C) Mammals give live birth, rather than laying eggs
(D) Mammals are two-legged creatures, i.e., they can walk on two legs

In this case, (D) is the correct answer (i.e., false choice). Note that (C), while true for most mammals, is false for monotremes; however, in the context, (D) is clearly a far more outrageously false statement. In order to get to (D), one needs to read and evaluate each option. Even though one can in principle stop as soon as one finds a statement that is false, it makes sense to read all the options because of the potential for presence of options such as "All of the above" or "None of the above" or due to differing degrees of truth or falsehood of the options (for instance, if Option (D) were not present, (C) would be the most suitable option to select).

Questions where the options are a partition of the possibility space

These questions occupy an intermediate niche between the other two types of questions. The question typically asks about the nature of a value, and subdivides the possibility space into mutually exclusive (and possibly collectively exhaustive) options. For instance, consider the question:

The number 31/47 - 13/18 is
(A) Less than 0
(B) Equal to 0
(C) Greater than 0 and less than 1
(D) Equal to 1
(E) Greater than 1

Here, one does need to look at the answer options before selecting the correct one. However, to a large extent, the problem revolves around solving the original question in a manner that makes it easy to identify the relevant part of the possibility space that the answer falls within. Note again that there may be estimation methods that allow one to arrive at the correct option without doing the computation.

Here is another question.

What is the relationship in time between the American Civil War and World War II?
(A) The American Civil War was over before World War II started
(B) The American Civil War started before World War II but finished while World War II was ongoing
(C) World War II started before the American Civil war and ended after the American Civil War was over
(D) World War II started before the American Civil War and ended during the American Civil War
(E) World War II started was over before the American Civil War started.

The options are not quite a full partition of the possibility space -- there is in fact a total of nine logical possibilities and only five have been listed. To some extent, therefore, the choice of options itself narrows the possibilities.

On reading the question, the reader can jot down what he/she knows of the start and end dates of the two wars. Once that is done, the process of comparison with the options comes close to requiring no subject matter knowledge (though it does require verbal knowledge and mathematical comparison skills).

Existence of "All of the above" or "None of the above" options

Some MCQs have "All of the above" or "None of the above" options, and some have both. These options are typically found for questions such as "Which of the following statements is true?" or "Which of the following statements is false?" For a well-designed question:

  • The "All of the above" and "None of the above" options should be at the end.
  • If both the "All of the above" and "None of the above" option appear, the "None of the above" option should appear below the "All of the above" option.
  • There should be an indication in the question statement, before the options appear, that these options are present. This alerts the reader to look for these options. This is particularly important for the "All of the above" option, and particularly so if that is the correct option, because a particular reader may simply stop at the first correct option and select that without reading the "All of the above" option.
  • In case the question is of the form which of these statements is false? it is preferable that the "All of the above" and "None of the above" are clear in terms of their meaning. For instance, in case of questions with five options (A)-(E) where (D) and (E) are the "All of the above" and "None of the above" options, "All of the above" may be rewritten as "All of the above, i.e., the statements of Options (A)-(C) are all false" and "None of the above" may be rewritten as "None of the above, i.e., the statements of Options (A)-(C) are all true."

Existence of "Insufficient information" options

Some MCQs have options of the form "The information in the question is insufficient to decide." These options typically appear in the context of the "partition the possibility space" type questions. For a well-designed question:

  • The "insufficient information" option should be at the end.
  • There should be an indication in the question statement, before the options appear, that such an option is present.
  • It is often good practice to specify how the existing information may be insufficient. For instance, "we need to know whether is positive or negative" or "we need to know the time period in which the event occurred" can help provide context in understanding this option.

Design

Deciding the number of options

The number of options is an important variable in MCQ design. In general, the following need to be kept in mind:

  • The greater the number of options, the longer the question takes to read and process. This may mean that fewer questions can be used for a given time limit.
  • A greater number of options makes random guessing less likely to work as a strategy, and thereby makes measurement more accurate.

The following are typically seen:

  • Class quizzes and tests may feature yes/no or true/false questions, which can be viewed as MCQs with two options. These may, however, be accompanied with requests for explanation.
  • Class quizzes, tests, and standardized tests, when using MCQs, typically use MCQs with four or five options.
  • Raven's IQ tests, particularly the harder ones, typically use larger numbers of options, ranging from 5 to 10. Harder tests typically use larger numbers of options in order to include more distractors and more clearly separate people who can get even a few questions correct from random guessers.

The use of free responses to generate options

Some versions of test design, particularly for concept inventories, use free responses to generate the options for the multiple choice questions. Explicitly, students are asked to write down answers to the question part of the MCQ in free response form. Some form of text analysis is done on the answers to extract the most frequent lines of student thought, and these are used to formulate the incorrect answer options. Note that such a process is highly likely to generate good distractors (see below) as the incorrect answer options.

Distractors

A distractor is defined as an option for a multiple-choice question that is incorrect, but that can be arrived at as the correct answer by a respondent who has a particular misconception or makes a particular kind of error in reading or working out the question, where the misconception or error are relatively likely. The way that distractors are used is a major part of the design of the MCQ, and considerably affects its perceived difficulty.

In general, a distractor should be chosen such that the distinction between the correct answer and the distractor is part of the goal of the assessment.

For instance, suppose solving a problem requires a conjunction of skills A and B. Some people make careless errors with the part that requires skill B, leading them to converge on a particular wrong answer. If testing the error-free application of skill B is part of our goal, it makes sense to include the incorrect answer as a distractor. If, however, the primary goal is to test skill A, then the distractor does not make sense.

Distractors can also be used for diagnostic purposes. For instance, if answering a question requires a conjunction of skills, and erroneous application at each level leads to a particular distractor, and these distractors are separate, choosing the different distractors as options makes it possible to perform an analysis of responses and determine what misconceptions dominated. The use of multiple distractors per question, however, may depress the total score of individuals too much and be bad for assessments based on total scores.

Note that the concept of "distractors" as independently chosen options makes the most sense for the questions where the answer can be arrived at independent of looking at the choices or questions where one needs to read through answers to find one that satisfies the question criterion. For questions where the answers form a natural partition of the possibility space, the choice of options is intrinsic to the choice of the question itself, and therefore, baked into thinking up the original question.

Use for diagnosis and assessment

Controlling for random guesses: the basic model

Here, we assume no scoring penalty for random guesses, and we assume that respondents attempt every question.

The simplest model for controlling for random guesses is that for every question, every respondent falls into one of two categories:

  • The respondent knows the correct answer
  • The respondent makes a guess that has a probability of being correct where is the number of options

If we follow this model, then:

Expected number of answers the respondent does not know = * (Apparent number of wrong answers)

Or equivalently:

Expected number of answers the respondent knows = Number of answers the respondent gets right - * (Number of answers the respondent get wrong)

Similar formulas apply to the study of a single question across multiple respondents.

The basic model is unrealistic for a number of reasons, namely that people often have specific misconceptions or make specific careless errors that lead them to select wrong answers. On the other hand, some might guess the right answer with a higher probability than mere chance through a process of elimination, without actually knowing the answer. The design of the options is extremely relevant to inferring the respondent's knowledge from the responses.