Multiple-choice question - Revision history

Vipul at 23:52, 3 December 2023

2023-12-03T23:52:14Z

← Older revision		Revision as of 23:52, 3 December 2023
Line 1:		Line 1:
			{{question format}}

	==Definition==		==Definition==

Issa Rice at 22:40, 8 October 2018

2018-10-08T22:40:11Z

← Older revision		Revision as of 22:40, 8 October 2018
Line 140:		Line 140:

	The basic model is unrealistic for a number of reasons, namely that people often have specific misconceptions or make specific careless errors that lead them to select wrong answers. On the other hand, some might guess the right answer with a higher probability than mere chance through a process of elimination, without actually knowing the answer. The design of the options is extremely relevant to inferring the respondent's knowledge from the responses.		The basic model is unrealistic for a number of reasons, namely that people often have specific misconceptions or make specific careless errors that lead them to select wrong answers. On the other hand, some might guess the right answer with a higher probability than mere chance through a process of elimination, without actually knowing the answer. The design of the options is extremely relevant to inferring the respondent's knowledge from the responses.

			==External links==

			* [https://terrytao.wordpress.com/2008/12/14/on-multiple-choice-questions-in-mathematics/ "On multiple choice questions in mathematics"] by [[Terence Tao]]

Vipul: /* Questions where the options are a partition of the possibility space */

2013-12-28T18:58:10Z

Questions where the options are a partition of the possibility space

← Older revision		Revision as of 18:58, 28 December 2013
Line 60:		Line 60:
	{{quotation\|The number 31/47 - 13/18 is<br>(A) Less than 0<br>(B) Equal to 0<br>(C) Greater than 0 and less than 1<br>(D) Equal to 1<br>(E) Greater than 1}}		{{quotation\|The number 31/47 - 13/18 is<br>(A) Less than 0<br>(B) Equal to 0<br>(C) Greater than 0 and less than 1<br>(D) Equal to 1<br>(E) Greater than 1}}

	Here, one does need to look at the answer options before selecting the correct one. However, to a large extent, the problem revolves around solving the original question in a manner that makes it easy to identify the relevant part of the possibility space that the answer falls within. Note again that there may be estimation methods that allow one to arrive at the correct ~~part~~ without doing the computation.		Here, one does need to look at the answer options before selecting the correct one. However, to a large extent, the problem revolves around solving the original question in a manner that makes it easy to identify the relevant part of the possibility space that the answer falls within. Note again that there may be estimation methods that allow one to arrive at the correct option without doing the computation.

	Here is another question.		Here is another question.

Vipul at 17:45, 26 September 2013

2013-09-26T17:45:04Z

← Older revision		Revision as of 17:45, 26 September 2013
Line 100:		Line 100:
	* [[Class quiz]]zes and tests may feature yes/no or true/false questions, which can be viewed as MCQs with two options. These may, however, be accompanied with requests for explanation.		* [[Class quiz]]zes and tests may feature yes/no or true/false questions, which can be viewed as MCQs with two options. These may, however, be accompanied with requests for explanation.
	* Class quizzes, tests, and [[standardized test]]s, when using MCQs, typically use MCQs with four or five options.		* Class quizzes, tests, and [[standardized test]]s, when using MCQs, typically use MCQs with four or five options.
	* Raven's IQ tests, particularly the harder ones, typically use larger numbers of options, ranging from 5 to 10. Harder tests typically use larger numbers of ~~items~~ in order to include more distractors and more clearly separate people who can get even a few questions correct from random guessers.		* Raven's IQ tests, particularly the harder ones, typically use larger numbers of options, ranging from 5 to 10. Harder tests typically use larger numbers of options in order to include more distractors and more clearly separate people who can get even a few questions correct from random guessers.

	===The use of free responses to generate options===		===The use of free responses to generate options===
Line 114:		Line 114:
	For instance, suppose solving a problem requires a conjunction of skills A and B. Some people make careless errors with the part that requires skill B, leading them to converge on a particular wrong answer. If testing the error-free application of skill B is part of our goal, it makes sense to include the incorrect answer as a distractor. If, however, the primary goal is to test skill A, then the distractor does not make sense.		For instance, suppose solving a problem requires a conjunction of skills A and B. Some people make careless errors with the part that requires skill B, leading them to converge on a particular wrong answer. If testing the error-free application of skill B is part of our goal, it makes sense to include the incorrect answer as a distractor. If, however, the primary goal is to test skill A, then the distractor does not make sense.

	Distractors can also be used for ''diagnostic'' purposes. For instance, if answering a question requires a ~~conjunctoin~~ of skills, and erroneous application at each level leads to a particular distractor, and these distractors are separate, choosing the different distractors as options makes it possible to perform an analysis of responses and determine what misconceptions dominated. The use of multiple distractors per question, however, may depress the total score of individuals too much and be bad for assessments based on total scores.		Distractors can also be used for ''diagnostic'' purposes. For instance, if answering a question requires a conjunction of skills, and erroneous application at each level leads to a particular distractor, and these distractors are separate, choosing the different distractors as options makes it possible to perform an analysis of responses and determine what misconceptions dominated. The use of multiple distractors per question, however, may depress the total score of individuals too much and be bad for assessments based on total scores.

	Note that the concept of "distractors" as ''independently'' chosen options makes the most sense for the questions where the answer can be arrived at independent of looking at the choices or questions where one needs to read through answers to find one that satisfies the question criterion. For questions where the answers form a natural partition of the possibility space, the choice of options is intrinsic to the choice of the question itself, and therefore, baked into thinking up the original question.		Note that the concept of "distractors" as ''independently'' chosen options makes the most sense for the questions where the answer can be arrived at independent of looking at the choices or questions where one needs to read through answers to find one that satisfies the question criterion. For questions where the answers form a natural partition of the possibility space, the choice of options is intrinsic to the choice of the question itself, and therefore, baked into thinking up the original question.

Vipul: /* Definition */

2013-09-18T14:24:25Z

Definition

← Older revision		Revision as of 14:24, 18 September 2013
Line 14:		Line 14:

	This page is ''not'' about survey MCQs. Rather, it is about questions where there is a single specific correct answer. For clarity, we will restrict attention to the case where respondents are expected to pick a single response. Cases where respondents need to pick sets of options can be reduced to this case through a mathematical equivalence.		This page is ''not'' about survey MCQs. Rather, it is about questions where there is a single specific correct answer. For clarity, we will restrict attention to the case where respondents are expected to pick a single response. Cases where respondents need to pick sets of options can be reduced to this case through a mathematical equivalence.

			==Methods of administration==

			* MCQs may be administered as part of homeworks, [[class quiz]]zes, or tests, to be done individually (without discussion), individually (with discussion), or in groups.
			* MCQs may be brought up in class discussions and students may be [[polling\|polled]] for their responses using [[clicker]]s or more primitive methods.

	==Advantages==		==Advantages==

Vipul: /* Questions with a clear objective answer that can be computed without looking at the options */

2013-09-18T00:47:00Z

Questions with a clear objective answer that can be computed without looking at the options

← Older revision		Revision as of 00:47, 18 September 2013
Line 39:		Line 39:
	* The form factor of the answer can be deduced by looking at the options.		* The form factor of the answer can be deduced by looking at the options.
	* Careless errors that do not result in arrival at one of the other answer options can be detected and potentially corrected.		* Careless errors that do not result in arrival at one of the other answer options can be detected and potentially corrected.
	* It may be possible to use approximate estimation techniques that are insufficient to obtain a precise answer but sufficient to narrow down within the range of options. For instance, one might reason that <math>13 \times 47</math> should be close to <math>12 \times 50 = 600</math>, and therefore pick 611, without doing any calculation. The extent to which approximate estimation works depends on the nature of the other options. (For instance, in the above question, the technique of looking at the ''last digit'' does not work because all options have the same last digit of "1").		* It may be possible to use approximate estimation techniques that are insufficient to obtain a precise answer but sufficient to narrow down within the range of options. For instance, one might reason that <math>47 \times 13</math> should be close to <math>50 \times 12 = 600</math>, and therefore pick 611, without doing any calculation. The extent to which approximate estimation works depends on the nature of the other options. (For instance, in the above question, the technique of looking at the ''last digit'' does not work because all options have the same last digit of "1").

	===Questions that require evaluating existing options until one reaches an option that satisfies the question criterion===		===Questions that require evaluating existing options until one reaches an option that satisfies the question criterion===

Vipul: /* Design */

2013-09-17T18:20:23Z

Design

← Older revision		Revision as of 18:20, 17 September 2013
Line 97:		Line 97:
	* Raven's IQ tests, particularly the harder ones, typically use larger numbers of options, ranging from 5 to 10. Harder tests typically use larger numbers of items in order to include more distractors and more clearly separate people who can get even a few questions correct from random guessers.		* Raven's IQ tests, particularly the harder ones, typically use larger numbers of options, ranging from 5 to 10. Harder tests typically use larger numbers of items in order to include more distractors and more clearly separate people who can get even a few questions correct from random guessers.

			===The use of free responses to generate options===

			Some versions of test design, particularly for [[concept inventory\|concept inventories]], use free responses to generate the options for the multiple choice questions. Explicitly, students are asked to write down answers to the question part of the MCQ in free response form. Some form of text analysis is done on the answers to extract the most frequent lines of student thought, and these are used to formulate the incorrect answer options. Note that such a process is highly likely to generate good distractors (see below) as the incorrect answer options.

			===Distractors===

			A distractor is defined as an option for a multiple-choice question that is incorrect, but that can be arrived at as the correct answer by a respondent who has a particular misconception or makes a particular kind of error in reading or working out the question, where the misconception or error are relatively likely. The way that distractors are used is a major part of the design of the MCQ, and considerably affects its perceived difficulty.

			In general, a distractor should be chosen such that the distinction between the correct answer and the distractor is part of the goal of the assessment.

			For instance, suppose solving a problem requires a conjunction of skills A and B. Some people make careless errors with the part that requires skill B, leading them to converge on a particular wrong answer. If testing the error-free application of skill B is part of our goal, it makes sense to include the incorrect answer as a distractor. If, however, the primary goal is to test skill A, then the distractor does not make sense.

			Distractors can also be used for ''diagnostic'' purposes. For instance, if answering a question requires a conjunctoin of skills, and erroneous application at each level leads to a particular distractor, and these distractors are separate, choosing the different distractors as options makes it possible to perform an analysis of responses and determine what misconceptions dominated. The use of multiple distractors per question, however, may depress the total score of individuals too much and be bad for assessments based on total scores.

			Note that the concept of "distractors" as ''independently'' chosen options makes the most sense for the questions where the answer can be arrived at independent of looking at the choices or questions where one needs to read through answers to find one that satisfies the question criterion. For questions where the answers form a natural partition of the possibility space, the choice of options is intrinsic to the choice of the question itself, and therefore, baked into thinking up the original question.

	==Use for diagnosis and assessment==		==Use for diagnosis and assessment==

Vipul: /* Use for diagnosis and assessment */

2013-09-17T18:15:25Z

Use for diagnosis and assessment

← Older revision		Revision as of 18:15, 17 September 2013
Line 120:		Line 120:

	The basic model is unrealistic for a number of reasons, namely that people often have specific misconceptions or make specific careless errors that lead them to select wrong answers. On the other hand, some might guess the right answer with a higher probability than mere chance through a process of elimination, without actually knowing the answer. The design of the options is extremely relevant to inferring the respondent's knowledge from the responses.		The basic model is unrealistic for a number of reasons, namely that people often have specific misconceptions or make specific careless errors that lead them to select wrong answers. On the other hand, some might guess the right answer with a higher probability than mere chance through a process of elimination, without actually knowing the answer. The design of the options is extremely relevant to inferring the respondent's knowledge from the responses.

	~~===MCQs that have clear answers independent of seeing the options versus MCQs whose answer depends on the options===~~

	~~===The use of free responses to generate options===~~

	Some versions of test design, particularly for [[concept inventory\|concept inventories]], use free responses to generate the options for the multiple choice questions. Explicitly, students are asked to write down answers to the question part of the MCQ in free response form. Some form of text analysis is done on the answers to extract the most frequent lines of student thought, and these are used to formulate the incorrect answer options. Note that such a process is highly likely to generate good distractors (see below) as the incorrect answer options.

	~~===Distractors===~~

	A distractor is defined as an option for a multiple-choice question that is incorrect, but that can be arrived at as the correct answer by a respondent who has a particular misconception or makes a particular kind of error in reading or working out the question, where the misconception or error are relatively likely. The way that distractors are used is a major part of the design of the MCQ, and considerably affects its perceived difficulty.

	~~In general, a distractor should be chosen such that the distinction between the correct answer and the distractor is part of the goal of the assessment.~~

	For instance, suppose solving a problem requires a conjunction of skills A and B. Some people make careless errors with the part that requires skill B, leading them to converge on a particular wrong answer. If testing the error-free application of skill B is part of our goal, it makes sense to include the incorrect answer as a distractor. If, however, the primary goal is to test skill A, then the distractor does not make sense.

	Distractors can also be used for ''diagnostic'' purposes. For instance, if answering a question requires a conjunctoin of skills, and erroneous application at each level leads to a particular distractor, and these distractors are separate, choosing the different distractors as options makes it possible to perform an analysis of responses and determine what misconceptions dominated. The use of multiple distractors per question, however, may depress the total score of individuals too much and be bad for assessments based on total scores.

Vipul: /* Existence of "All of the above" or "None of the above" options */

2013-09-17T18:14:56Z

Existence of "All of the above" or "None of the above" options

← Older revision		Revision as of 18:14, 17 September 2013
Line 67:		Line 67:
	===Existence of "All of the above" or "None of the above" options===		===Existence of "All of the above" or "None of the above" options===

	Some MCQs have "All of the above" or "None of the above" options, and some have both. For a well-designed question:		Some MCQs have "All of the above" or "None of the above" options, and some have both. These options are typically found for questions such as "Which of the following statements is true?" or "Which of the following statements is false?" For a well-designed question:

	* The "All of the above" and "None of the above" options should be at the end.		* The "All of the above" and "None of the above" options should be at the end.
Line 73:		Line 73:
	* There should be an indication in the question statement, before the options appear, that these options are present. This alerts the reader to look for these options. This is particularly important for the "All of the above" option, and particularly so if that is the correct option, because a particular reader may simply stop at the first correct option and select that without reading the "All of the above" option.		* There should be an indication in the question statement, before the options appear, that these options are present. This alerts the reader to look for these options. This is particularly important for the "All of the above" option, and particularly so if that is the correct option, because a particular reader may simply stop at the first correct option and select that without reading the "All of the above" option.
	* In case the question is of the form ''which of these statements is '''false'''?'' it is preferable that the "All of the above" and "None of the above" are clear in terms of their meaning. For instance, in case of questions with five options (A)-(E) where (D) and (E) are the "All of the above" and "None of the above" options, "All of the above" may be rewritten as "All of the above, i.e., the statements of Options (A)-(C) are all false" and "None of the above" may be rewritten as "None of the above, i.e., the statements of Options (A)-(C) are all true."		* In case the question is of the form ''which of these statements is '''false'''?'' it is preferable that the "All of the above" and "None of the above" are clear in terms of their meaning. For instance, in case of questions with five options (A)-(E) where (D) and (E) are the "All of the above" and "None of the above" options, "All of the above" may be rewritten as "All of the above, i.e., the statements of Options (A)-(C) are all false" and "None of the above" may be rewritten as "None of the above, i.e., the statements of Options (A)-(C) are all true."

			===Existence of "Insufficient information" options===

			Some MCQs have options of the form "The information in the question is insufficient to decide." These options typically appear in the context of the "partition the possibility space" type questions. For a well-designed question:

			* The "insufficient information" option should be at the end.
			* There should be an indication in the question statement, before the options appear, that such an option is present.
			* It is often good practice to specify ''how'' the existing information may be insufficient. For instance, "we need to know whether <math>x</math> is positive or negative" or "we need to know the time period in which the event occurred" can help provide context in understanding this option.

	==Design==		==Design==

Vipul: /* Questions where the options are a partition of the possibility space */

2013-09-17T18:11:12Z

Questions where the options are a partition of the possibility space

← Older revision		Revision as of 18:11, 17 September 2013
Line 53:		Line 53:
	These questions occupy an intermediate niche between the other two types of questions. The question typically asks about the nature of a value, and subdivides the possibility space into mutually exclusive (and possibly collectively exhaustive) options. For instance, consider the question:		These questions occupy an intermediate niche between the other two types of questions. The question typically asks about the nature of a value, and subdivides the possibility space into mutually exclusive (and possibly collectively exhaustive) options. For instance, consider the question:

	{{quotation\|The number 31/47 - 13/18 is<br>(A) Less than 0<br>(B) Equal to 0<br>(C)Greater than 0 and less than 1<br>(D) Equal to 1<br>(E) Greater than 1}}		{{quotation\|The number 31/47 - 13/18 is<br>(A) Less than 0<br>(B) Equal to 0<br>(C) Greater than 0 and less than 1<br>(D) Equal to 1<br>(E) Greater than 1}}

	Here, one does need to look at the answer options before selecting the correct one. However, to a large extent, the problem revolves around solving the original question in a manner that makes it easy to identify the relevant part of the possibility space that the answer falls within. Note again that there may be estimation methods that allow one to arrive at the correct part without doing the computation.		Here, one does need to look at the answer options before selecting the correct one. However, to a large extent, the problem revolves around solving the original question in a manner that makes it easy to identify the relevant part of the possibility space that the answer falls within. Note again that there may be estimation methods that allow one to arrive at the correct part without doing the computation.