The Influence of Distractor Strength and Response Order on MCQ Responding

Multiple-choice questions (MCQs) play a key role in standardised testing and in-class assessment. Research into the influence of within-item response order on MCQ characteristics has been mixed. While some researchers have shown preferential selection of response options presented earlier in the answer list, others have failed to replicate these results. This paper investigates a possible explanation for these mixed findings by assessing the influence of distractor strength on MCQ response order effects. A real-world assessment was administered to 232 undergraduates in which the response order within items was systematically varied. A generalised multilevel model was then used to show a significant interaction between distractor strength and response position with regard to response behaviour. Furthermore, the effect was found to be independent of student ability. These findings have implications for MCQ test construction, minimising the negative consequences of MCQ test exposure and promoting improved test-taking strategies.

Examination; metacognition; assessment theory; test

Educators have long utilised multiple-choice questions (MCQs) in standardised testing and classroom assessment. The objectivity (Douglas, Wilson, & Ennis, [14] ; Zeidner, [48] ), convenience, grading accuracy (Holder & Mills, [21] ), scalability to large classroom testing and the ease of providing customised feedback (Epstein & Brosvic, [15] ; Epstein, Epstein, & Brosvic, [16] ; Guo, Palmer-Brown, Lee, & Cai, [18] ) make MCQs an invaluable assessment tool at virtually all educational levels across different countries. While MCQs are largely used to assess low level content knowledge, they can also target higher order skills, particularly through the use of cognitive models (Pugh, De Champlain, Gierl, Lai, & Touchie, [36] ; Tractenberg, Gushta, Mulroney, & Weissinger, [46] ). The ease of integration with modern technologies has also led to a rise in the use of MCQs as pedagogical tools which encourage in-class participation and real-time learning assessment (Hoppenbrock, [22] ). The strengths of MCQs are particularly relevant in today’s academic environment, where demand for targeted teaching approaches continues to grow as classroom sizes continue to expand (Powers, [35] ; Wittich, [47] ).

Given the ubiquitous role of MCQs in modern standardised testing and classroom assessment, one of the most common goals in MCQ test construction is the creation of equivalent items and parallel test forms. These parallel test forms are used to reliably assess differences within or between cohorts while maximising test security, minimising repeat item exposures among repeat test-takers and facilitating the updating of test items to reflect changes in the state of knowledge within specific fields. Given the importance of creating equivalent parallel test forms, it is therefore important to consider that even minor changes, such as changing the order of MCQ items or their responses, may have unanticipated influences on MCQ test item characteristics.

The impact of psychological factors on MCQ response behaviour has received particular attention with regard to question and response order effects. Research on question order effects has been largely consistent with most investigations showing question presentation order (e.g. ordered by topic vs. randomised) to have a slight impact on perceptions of test difficulty (Perlini, Lind, & Zumbo, [32] ) but no significant effect on student performance (though see also Balch, [5] ; Neely, Springston, & McCann, [30] ; Perlini et al., [32] ; Peters & Messier, [33] ; Pettijohn & Sacco, [34] ). Research on the effect of MCQ response order, on the other hand, has been significantly less consistent.

Prior research in this area has examined frequency of correct response selection (Marcus, [26] ; Schroeder, Murphy, & Holme, [42] ; Tellinghuisen & Sulikowski, [45] ), or response selection in general (Clark, [12] ; Gustav, [19] ) with mixed results. The first investigation of MCQ response order was conducted by Clark ([12] ), who assessed response behaviour in students on MCQs in which correct responses and distractors had been randomly distributed. In his results, Clark ([12] ) noted a selection preference for response options presented earlier in the five-choice response list. This early finding was supported by several other researchers over the past 50 years (Clark, [12] ; Gustav, [19] ; Krosnick & Fabrigar, [24] ; Schroeder et al., [42] ; Tellinghuisen & Sulikowski, [45] ). However, other researchers have found MCQ response order to either have no significant impact on response behaviour (Aiken, [1] ; Marcus, [26] ) or to have unpredictable item-dependent consequences (Cizek, [11] ; McNamara & Weitzman, [29] ). As an example of this, Cizek ([11] )’s investigation of medical exam performance found that changing the position of correct responses had little impact on the majority of items but had unpredictable effects on others, increasing correct response selection rates on some and reducing them on others. One possible explanation for these inconsistent findings is the strength and position of distractor responses, defined in this study as incorrect response options that require effortful consideration (McMahan, Pinckard, Prihoda, Hendricson, & Jones, [28] ), relative to correct response options. None of the prior investigations into the influence of MCQ response order on response selection behaviour have investigated the possibility of distractor options having a stronger or weaker effect based on their presentation order. An interaction between distractor strength and response order could account for prior mixed findings in this area (Clark, [12] ; Gustav, [19] ; Krosnick & Fabrigar, [24] ; Schroeder et al., [42] ; Tellinghuisen & Sulikowski, [45] vs. Aiken, [1] ; Cizek, [11] ; Marcus, [26] ; McNamara & Weitzman, [29] ), shed light on the cognitive processes underlying MCQ response reasoning and highlight parameters relevant to efficient parallel test construction.

A deeper understanding of the interplay between distractor strength and response order could also help minimise the negative consequences of MCQ testing. A rich body of literature has repeatedly highlighted the potentially positive impact of testing on long-term memory, often referred to as the ‘testing-effect’ (see Roediger & Karpicke, [38] for a review). While the positive consequences of MCQ testing on long-term retention of tested material clearly outweigh any negative outcomes (Roediger, Putnam, & Smith, [41] ), research has also shown that the processing of strong MCQ distractor responses can lead to the creation of false knowledge (Roediger & Marsh, [39] ) and negative shifts in the way students reason with their knowledge (Marsh, Roediger, Bjork, & Bjork, [27] ). Uncovering potential interactions between distractor strength and position could potentially help test constructors and educators improve pedagogical techniques to minimise these negative outcomes, identify items with an increased risk of promoting false knowledge formation and help students adopt test-taking techniques which minimise those risks.

Distractor strength has the potential to interact with MCQ response order through multiple routes. The first possible influence draws from satisficing theory (Simon, [44] ). Satisficing on response selection (i.e. selecting the first response that meets an acceptability threshold) was first proposed by Krosnick ([23] ). Satisficing has since been shown to influence response choices across a wide variety of domains including election voting (Chen, Simonovits, Krosnick, & Pasek, [10] ), consumer behaviour (Reutskaja, Nagel, Camerer, & Rangel, [37] ) and perceptual decision-making (Oh et al., [31] ). Of particular relevance to educational assessment, evidence for satisficing in MCQ response selection has been shown in the work of (Caplin, Dean, & Martin, [8] ). A satisficing model of MCQ response behaviour would suggest that a significant number of responders may adopt a strategy of selecting the first item response that appears ‘good enough’ or one they cannot find a reason to reject (Caplin et al., [8] ). As a result of this strategy, presenting strong distractors before correct responses would increase the odds of those distractors being selected whereas changes in the position of weak distractors, which are less likely to meet acceptability thresholds, would have a weaker effect.

A second, potentially complementary reason for how distractor strength may interact with response order comes from research work involving memory interference (Anderson, Green, & McCulloch, [4] ; Roediger & Neely, [40] ). Memory inference models argue that one of the key processes in the retrieval of memories, including semantic or factual content knowledge (Anderson & Bell, [2] ; Campbell & Phenix, [6] ; Campbell & Thompson, [7] ; Gómez-Ariza, Pelegrina, Lechuga, Suárez, & Bajo, [17] ), is the inhibition of related but non-retrieved memory traces. A side effect of this inhibitory process is the reduced accessibility of the inhibited traces should an attempt be made to retrieve them in the near future, a phenomenon known as retrieval-induced forgetting (Anderson, Bjork, & Bjork, [3] ; Carter, [9] ). The predictions of memory interference models with regard to the interactive influence of distractor position and strength are similar to that of satisficing. As models of memory interference indicate that greater similarity between memories increase levels of impairment (Anderson et al., [4] ), evaluating strong distractors before correct response option may increase the difficulty of retrieving information related to the correct response reducing the odds of correct response selection. In comparison, changes to the presentation order of weak distractors, which are less strongly associated with information related to correct responses, would have a weaker effect.

In summary, there are multiple reasons to suspect that distractor strength significantly interacts with distractor position to influence MCQ response behaviour. A better understanding of this influence could help resolve inconsistent findings regarding the influence of response order on MCQ response behaviour, improve the efficiency of parallel test form development, and minimise the potential negative influences of MCQ testing. Thus, this study investigates the interaction between distractor strength and distractor position in a real-world testing environment. The influence of the position of strong and weak distractors on MCQ response behaviour within participants was investigated by systematically manipulating the order of response options in a preselected set of MCQ items administered as part of a real-world classroom assessment.

Drawing on models of satisficing and retrieval interference, this study has two primary hypotheses. Firstly, presenting strong distractors before correct response options will reduce correct response selection rates relative to presenting the same strong distractors after. Secondly, presenting weak distractors before or after correct response options will have no significant impact on correct response selection rates. The role of individual differences in ability on these effects was also investigated.

A single cohort of 232 undergraduates enrolled in a second-year statistics course at a large, urban Malaysian university provided the response data as part of a pre-existing in-class assessment. Given the minimal risk, anonymity of data collection and the fact that implemented manipulations had no impact on any already scheduled assessment, the local Ethics Review Board (ERB) waivered the requirement of informed consent for this study. The local ERB also approved the matching of participant response data on the critical test with their response data on a second in-class test. This matching was conducted by one of the study’s authors who was blind to the link between student identities and their IDs. After the matching process had been conducted, the author replaced all student IDs with new randomised ID numbers before passing the data to the rest of the team for further analysis.

The measure used in this study consisted of 24 MCQs aimed at measuring students’ comprehension of various statistical concepts. This test was one of two in-class assessments given to students over the duration of the course. Performance on the second in-class test, administered at the second half of the semester, was used as the measure of student ability. The questions did not require numerical calculation and primarily focused on either factual knowledge involving statistical concepts or the identification of appropriate statistical analysis based on descriptions of research study designs. All questions had four response options. Prior to the administration of the test, three statistics instructors, who were not directly involved in the targeted course, independently rated the strength of all the distractors in the test as being either weak or strong. All three raters were given the same rating instructions which classified strong distractors as ‘response options that were likely to be confused with the correct response given the context of the question’ and weak distractors as ‘response options that were unlikely to be confused with the correct response given the context of the question’. Raters could rate more than one response option as strong distractors in each item or none at all. Eight items, each containing only one strong distractor and two weak distractors, all of which were rated with perfect agreement among all three raters, were selected for use in this study.

Counterbalancing of the position of the distractors for each item proceeded as follows. First, four strong distractor positions of interest were identified, in which the distractor was presented, (1) one option before correct response, (2) directly before the correct response, (3) directly after correct response and (4) one option after correct response. Using this taxonomy, four versions of the selected eight items were created in which the placement of distractor was varied through positions (1) through (4).

During this generation process, weak distractors were inserted in between strong distractors and correct response for the above-mentioned positions (1) and (4) with the remaining weak distractors being placed either before or after the target items to produce the combinations shown in Table 1. Finally, four separate test sets were created to counterbalance the orders of the distractor position based on the sequential presentation of test items, using a balanced two-block Latin square design, as shown in Table 2. This counterbalancing ensured that participants experienced each strong distractor position twice, with the order of response presentation being counterbalanced between participants. Within each test set, the eight target items were separated by one non-target item. Table 3 shows a sample of a multiple-choice response order for the test item ‘Most inferential test statistics can be seen to be a ratio of ______ by______’.

Positions of correct response relative to strong distractor for target items.

1 Notes: SD: Strong distractor; WD: Weak distractor; C: Correct response.

Counterbalancing of distractor positions for target items.

• 2 Strong distractor one option before correct response.
• 3 Strong distractor directly before correct response.
• 4 Strong distractor directly after correct response.
• 5 Strong distractor one option after correct response.

Sample of multiple-choice response order: correct response (underlined) and strong distractor (in bold) position across the four conditions.

Prior to the test, students were informed of the exam question format, scoring procedures as well as the specific lectures and topics from which the questions would be drawn from. The test sets and sheets were labelled by their specific set numbers and distributed in sequential order among the students to ensure an even distribution of item sets throughout the class and minimise the influence of factors related to student seating position preferences such as student ability or test-taking motivation. Students were also seated one seat apart to ensure privacy and minimise copying behaviour on the test, which was invigilated by the course instructor and several teaching assistants. Responses on the test were made using MCQ scantron sheets, which were then scored and recorded electronically to eliminate human error.

After coding correct response selections on all items, distractor response selection was coded based on their type (Strong, Weak) and position relative to the correct response (Before vs. After) for a total of four response categories (Strong Before, Strong After, Weak Before and Weak After). The probability of students selecting specific response options relative to others was estimated in PROC GLIMMIX in SAS 9.3, with a multilevel generalised multinomial logit model using a Laplace approximated maximum likelihood estimator and a random intercept for subjects. Student ability, as indexed by the second in-class test, was then added to the model after being mean centred. A Bonferroni correction was applied to yield a model-wise p-value corrected threshold of.005 for all comparisons.

The odds ratios of selecting each response category relative to all other responses after controlling for student ability are given in Table 4 with 95% confidence intervals provided in brackets. The reported significance tests in Table 4 assessed the significance level of the difference between each odds ratio and a 1:1 equal selection likelihood response pattern. In the critical comparison of this study, students were 1.67 times more likely to select Strong Distractors presented before the correct response relative to Strong Distractors presented after (F(1,224) = 8.01, p = .005). The position of Weak Distractors, on the other hand, did not have a significant influence on their probability of being selected (F(1,224) = 4.52, p = .035). The results also show that students were more likely to select Correct Response options relative to all distractor options (e.g. 15.40 times more likely relative to Strong Distr. Before).

Odds ratio of a response category selection vs. reference category.

• 6 p < .001
• 7 p < .005
• 8 All reported p-values surpassed a Bonferroni corrected threshold for multiple comparisons.

Higher levels of student ability significantly increased the odds of selecting Correct Response options relative to all other response categories (Strong Distr. Before, F(1,224) = 76.26, p < .001, Strong Distr. After, F(1,224) = 78.01, p < .001, Weak Distr. Before, F(1,224) = 52.49, p < .001, Weak Distr. After, F(1,224) = 151.74, p < .001).

While student ability was modelled as a continuous variable, selection odds ratio differences between students with ability scores one standard deviation above and below the mean were estimated to facilitate interpretation of these results. These estimated differences showed that high (one standard deviation above the mean) ability students were more likely than low (one standard deviation below the mean) ability students to select correct responses relative to all other responses (5.86 more likely compared to Strong Distr. Before; 8.12 more likely compared to Strong Distr. After; 5.48 more likely compared to Weak Distr. Before; and 10.09 times more likely compared to Weak Distr. After). Student ability did not, however, have a significant influence on the likelihood of selecting Strong Distractors presented before Correct responses relative to Strong Distractors presented after, F(1,224) = 1.33, p = .250. Student ability also did not have a significant influence on the likelihood of selecting Weak Distractors presented before Correct responses relative to Weak Distractors presented after, F(1,224) = 5.14, p = .024.

To bridge the results of this study with prior response order investigations (Clark, [12] ; Gustav, [19] ; Krosnick & Fabrigar, [24] ; Schroeder et al., [42] ; Tellinghuisen & Sulikowski, [45] ), a multilevel generalised binary logit model was used to assess the influence of correct answer position on correct response selection odds disregarding distractor position and strength. In this model, correct answers were coded by their position in the response option list with the outcome coded as correct response selected vs. not selected. As in the main analysis model, student ability was included as a fully interacting control variable in its original continuous form.

As expected, higher levels of ability were associated with increased rates of selecting correct responses (F(1,230) = 198.13, p < .001) overall. Students with ability scores one standard deviation above the mean were 6.35 times more likely to select the correct response relative to students who were one standard deviation below the mean, regardless of correct response position. Ability did not interact with the effect of correct response location (F(1,1613) =.09, p = .967). The main effect of correct response location was marginally significant (F(3,692) = 2.38, p = .070). As shown in Table 5, with the exception of correct responses placed second relative to third, there was a consistent trend towards earlier responses being more likely to be selected with one comparison, correct responses placed first relative to last (t(692) = 2.62, p = .009), being statistically significant.

Odds ratio of correct response selection by position of correct response.

9 p < .01.

The results of this study indicate that distractor strength has a significant moderating influence on the effect of response position on MCQ response behaviour. After controlling for student ability, strong distractors placed before correct answers were more likely to be selected relative to strong distractors placed after, whereas the position of weak distractors had no influence on their selection probability. As expected, student ability increased the odds of correct response selection but did not significantly interact with any of the distractor position effects. Our findings replicate prior work in the area, which found a selection advantage for correct responses presented earlier in the response list (Clark, [12] ; Gustav, [19] ; Krosnick & Fabrigar, [24] ; Schroeder et al., [42] ; Tellinghuisen & Sulikowski, [45] ) and extend those findings by showing the effect to be largely independent of student ability.

The findings of this study contribute to a rich literature of order effects in MCQ testing. While prior research on the influence of question order has consistently found null effects (Neely et al., [30] ; Perlini et al., [32] ; Peters & Messier, [33] ; Pettijohn & Sacco, [34] ; though see also Balch, [5] ), findings regarding the influence of response order has been mixed with some studies reporting significant effects (Clark, [12] ; Gustav, [19] ; Krosnick & Fabrigar, [24] ; Schroeder et al., [42] ; Tellinghuisen & Sulikowski, [45] ) and others reporting null results (Aiken, [1] ; Cizek, [11] ; Clark, [12] ; Gustav, [19] ; Krosnick & Fabrigar, [24] ; Marcus, [26] ; McNamara & Weitzman, [29] ; Schroeder et al., [42] ; Tellinghuisen & Sulikowski, [45] ). The results of this study suggest that distractor strength may play a role in determining the effect of response order, as presenting correct responses earlier in the response list would have a stronger effect in items containing multiple strong distractor items. In items with multiple weak distractors, on the other hand, changes to the position of correct responses would have less of an impact as weak distractors are less likely to meet satisficing thresholds or interfere with the retrieval of information which would lead to correct response endorsement.

The findings of this study highlight a previously undemonstrated influence of response order on MCQ response behaviour which has the potential to improve the efficiency of parallel test construction. Controlling for the interaction between distractor strength and response order could help streamline the construction process and reduce pilot testing cycles involved in assessing test form equivalency.

Given the impact of test-taking experiences on long-term learning (see Roediger & Karpicke, [38] for a review), the results of this study also highlight the importance of providing students with the knowledge base needed to reject plausible sounding but nonetheless incorrect response alternatives. This is particularly important given that research involving the testing effect has shown that repeated selection of strong distractors on MCQ tests has the potential to strengthen false knowledge pathways (Marsh et al., [27] ; Roediger & Marsh, [39] ). On the positive side, the findings of this study also suggest that systematic manipulation of response order may be a useful tool in strengthening the positive impact of testing effect-related influences. For instance, repeated evaluation of test items in which the correct response is repeatedly presented early in the response list may help strengthen true semantic memories traces and improve long-term retention.

One potential method to achieve this goal may be to highlight similarities and differences between easily confused concepts during teaching sessions, a technique which has been shown to reduce retrieval-induced forgetting effects (Anderson et al., [4] ). The findings of this study lend additional support to the value of implementing these instructional design changes with regard to enhancing student learning and testing performance. They also indicate that these changes stand to benefit students across the ability continuum as the effects observed in this study were found to be independent of student ability level. Finally, the results of this study also make useful contributions to the literature regarding the impact of cognitive and psychological factors in educational testing (Danili & Reid, [13] ; Liew, Lench, Kao, Yeh, & Kwok, [25] ; Sievertsen, Gino, & Piovesan, [43] ). The current study also demonstrates the value of utilising multilevel modelling to assess interactions between individual-level (i.e. ability) and item-level (i.e. response positions and distractor strength) variables in educational testing. The use of multilevel modelling to assess item-level effects allows for partial response data to be included in the modelling process and greater flexibility in addressing dependencies between observations, as well as potentially increased power in testing fixed effects (Hoffman & Rovine, [20] ).

One of the limitations of this study was the use of a 2:1:1, weak distractor: strong distractor: correct response, ratio as opposed to a 1:1:1 ratio. This restriction, which was due to the constraints of working within a pre-existing testing framework, precluded the use of a perfectly balanced response counterbalancing structure. Nonetheless, the counterbalancing system used ensured that the main finding of this study of strong distractors being more likely to be selected when presented before the correct response relative to after was unlikely to be affected by this limitation.

A second limitation is the limited content scope as the current study only tested response order effects with regard to statistical knowledge. While use of limited knowledge domains is standard in research on MCQ testing (Cizek, [11] ; Schroeder et al., [42] ; Tellinghuisen & Sulikowski, [45] ), it may be interesting to assess the generalisability of these effects across multiple subject areas, particularly those which may be less susceptible to satisficing or retrieval-induced forgetting related effects such as mathematics. Furthermore, given this study was only conducted on a single cohort of university students in Malaysia, future research could examine the cross-cultural generalisability of the findings across countries with different educational systems.

While this study does not discriminate between accounts of satisficing or memory interference, it does lays the groundwork for future investigations. Future studies could potentially examine the distinction between these satisficing or memory interference by assessing the influence of test-taking instructions or electronic testing presentation methods which require students to consider all responses before finalising their response. If satisficing is the predominant effect, this would likely weaken the strong distractor order effect demonstrated in this study. If the effect of strong distractor position is predominately driven by memory interference however, it would likely instead be strengthened or unaffected by these instructions.

This study makes an important contribution to research into order effects in MCQ testing by demonstrating an interaction between distractor strength and response order with regard to MCQ response selection behaviour. Given the ubiquity of MCQ items in educational assessment and standardised testing, these findings contribute towards improving the efficiency of parallel test development and highlight the importance of providing students with the knowledge base needed to reject plausible but nonetheless incorrect response alternatives. This is particularly crucial as repeated selection of strong distractors on MCQ tests may in the long run strengthen false knowledge memory pathways (Marsh et al., [27] ; Roediger & Marsh, [39] ). These findings also lay the foundation for future researchers to investigate the factors underlying response order effect, potentially differentiating between several possible theoretical accounts for them such as satisficing and retrieval interference.

No potential conflict of interest was reported by the authors.