Accurate self-regulation of performance is important for trainees. Trainees rely on cues to make monitoring judgments to self-regulate their performance. Ideally, cues and monitoring judgements accurately reflect performance, as measured by cue diagnosticity (the ability of a cue to predict performance) and monitoring accuracy (the ability of a monitoring judgement to predict performance). However, this process is far from perfect, emphasizing the need for more accurate cues and monitoring judgements. Perhaps the mental effort of a task could be a cue used to inform certainty judgements. The purpose of this study is to measure cue utilization and cue diagnosticity of mental effort and monitoring accuracy of certainty for self-regulation of performance. Focused on the task of ECG interpretation, 22 PGY 1-3 Internal Medicine residents at McMaster University provided a diagnosis for 10 ECGs, rating their level of certainty (0-100%) and mental effort (Paas scale, 1-9). 220 ECGs completed by 22 participants were analyzed using path analysis. There was a negative moderate path coefficient between certainty and mental effort (β = − 0.370, p < 0.001), reflecting cue utilization. Regarding cue diagnosticity of mental effort, this was reflected in a small negative path coefficient between mental effort and diagnostic accuracy (β = − 0.170, p = 0.013). Regarding monitoring accuracy, a moderate path coefficient was observed between certainty and diagnostic accuracy (β = 0.343, p < 0.001). Our results support mental effort as a cue and certainty as a monitoring judgement for self-regulated performance. Yet, reported correlations are not very high. Future research is needed to identify additional cues.
Keywords: Cognitive load; Diagnostic certainty; Self-regulation; Mental effort
Trainees need to be able to accurately self-regulate their clinical performance. Accurate self-regulation of performance would enable trainees to proceed independently when appropriate or seek help when needed. However, research has shown that self-regulation is not always accurate (Friedman et al. 2005; Cavalcanti and Sibbald 2014), highlighting the need to improve the self-regulation of performance in trainees.
How do learners self-regulate their performance? Learners are not able to make direct assessments of their performance (Koriat 1997) and consequently need to make judgements based on indirect information. Within the context of self-regulated learning theory (e.g., Nelson and Narens 1990) and the cue utilization framework (Koriat 1997), there are three elements involved in self-regulation: cues, monitoring judgements, and diagnostic accuracy. Cues are used to make monitoring judgements that regulate performance (Koriat 1997). The use of a cue to inform monitoring judgements is referred to as cue utilization (de Bruin et al. 2017). How well the use of a cue predicts actual performance is referred to as cue diagnosticity (Brunswik 1949). Diagnostic certainty and diagnostic confidence are examples of monitoring judgements. The accuracy of a monitoring judgement in predicting performance is termed monitoring accuracy (de Bruin et al. 2017).
Literature highlights the challenges in accurate self-regulation of performance. Imperfect monitoring accuracy between monitoring judgements and actual performance has been identified in multiple studies (Barnsley et al. 2004, Friedman et al. 2005; Cavalcanti and Sibbald 2014). This is problematic because inaccurate assessment of performance has negative consequences: a trainee falsely assuming to be correct when actually being wrong could make an incorrect diagnosis or an inappropriate management plan.
How can self-regulation of performance be improved? De Bruin et al. (2017) argue that improved selection of cues could enhance performance. Selected cues ideally accurately inform the monitoring judgements (cue utilization) and predict actual performance (cue diagnosticity) (de Bruin et al. 2017). However, a robust understanding of cues used by learners to monitor their performance is lacking. The literature predominantly focuses on response time as a cue. Longer response times have been associated with lower reported certainty (McConnell et al. 2012). Furthermore, learners took more time to provide an incorrect answer relative to a correct answer (Eva and Regehr 2007; Norman et al. 2014). While this may hold true in research settings, confounders may limit the use of time as an accurate cue in clinical training environments.
Are there other cues that could inform monitoring judgements in clinical training environments? Perceived mental effort could be a cue to inform monitoring judgements, such as certainty. According to Cognitive Load Theory, there is a finite amount of mental effort available to make decisions (Sweller 1988). More complex or challenging decisions may require more mental effort than simple decisions. Learners could perceive higher mental effort and lower certainty when they are approaching decisions they find challenging. While mental effort has not been explicitly studied as a cue in self-regulated performance, the literature supports its potential use as a cue. Koriat et al. (2014) described higher certainty when lower mental effort was reported by undergraduate students, supporting the diagnosticity of perceived mental effort as a cue to inform certainty judgements. In another study, learners provided with negative feedback reported higher mental effort than learners provided with positive feedback on their performance (Raaijmakers et al. 2017), highlighting the potential cue diagnosticity of mental effort. These studies support a potential framework for self-regulation of performance where the cue is mental effort, the monitoring judgement is certainty, and the performance is diagnostic accuracy (Fig. 1).Overview of conceptual framework.(Adapted from de Bruin et al. 2017)
PHOTO (COLOR)
ECG interpretation represents an appropriate model to study mental effort as a cue. Firstly, the ideal model would be sufficiently complex to allow for low certainty to occur. The multiple variables that must be identified and interpreted on each ECG should provide this complexity. The diagnostic accuracy suggests that we will observe variations in reported certainty; if the task was too simple then observed certainties may be restricted to high values and vice versa. Secondly, ECG interpretation is an authentic task. The clinical importance of this skill is emphasized in a spectrum of specialties and clinical settings (Salerno et al. 2003). Lastly, from a practical perspective, ECG interpretation is a time efficient and cost-effective model.
The purpose of this study is to study the utilization of mental effort as a cue, the cue diagnosticity of mental effort, and the monitoring accuracy of certainty as a monitoring judgement using ECG interpretation in internal medicine residents as a model.
- Is perceived mental effort a cue used to inform certainty judgements (cue utilization)?
- What is the cue diagnosticity of perceived mental effort (i.e., does it predict diagnostic accuracy)?
- What is the monitoring accuracy of certainty (i.e., to what extent does it predict diagnostic accuracy)?
All 97 post-graduate year (PGY) 1-3 Internal Medicine residents at McMaster University in Hamilton, Ontario, Canada were invited to participate in the study by email. Twenty-two residents participated. Participants provided informed consent. These residents were selected as the study population because we anticipated that they would utilize analytic skills to interpret ECGs.
Participants indicated their gender, residency program, prior Cardiology rotations, and self-perceived global performance estimate in ECG interpretation. Prior experience on Cardiology rotations and self-perceived performance estimate in ECG interpretation were elicited to ensure that the participants studied had a homogenous baseline.
The ECG diagnoses included normal sinus rhythm, anterior STEMI, acute pericarditis, supraventricular tachycardia, ventricular tachycardia, pre-excitation pattern, long QTc, complete heart block, atrial flutter, and hyperkalemia. The selection of ECGs diagnoses was informed by the diagnostic accuracies reported in two studies assessing ECG interpretation in Internal Medicine residents (Eslava et al. 2009; Sibbald et al. 2014). After the diagnoses were decided, we selected ECGs with these diagnoses from ECG Maven (Nathanson et al. 2001), specifically in levels 1-3 without secondary findings (e.g. axis deviations). Two cardiologists (SB and MS) reviewed the selected ECGs to ensure they had one unifying diagnosis without secondary diagnoses. All participants completed the same ten ECGs in the same order.
The prompt "How sure are you of your diagnosis?" was presented after each ECG. The participants indicated their certainty by sliding a marker across a line that ranged from 0 representing "I have no clue" to 100 representing "I am 100% certain". A similar scale has been used in previous work (Cavalcanti and Sibbald 2014).
While objective measures of mental effort exist, such as data obtained through eye-tracking (May et al. 1990), we favoured the use of a subjective assessment of mental effort as we hypothesized perceived mental effort to be a cue. The Paas scale, a 9-point scale that has been validated to assess mental effort subjectively, was used (Paas 1992). Experimental studies have demonstrated the high validity and reliability of the 9-point Paas scale (Ayres 2006; Paas 1992; Paas et al. 2003). The participants indicated their mental effort by sliding a marker across a line ranging from 1 (very very low mental effort) to 9 (very very high mental effort).
A score of 0 was assigned for incorrect responses. A score of 1 was assigned for correct responses.
A quantitative correlational design (Fig. 2) was used. Participants completed a demographic questionnaire before individually interpreting 10 ECGs of similar difficulty without any instructional tools on an online survey platform (SurveyGizmo, Boulder, CO). Participants rated certainty (0-100%) and mental effort (
PHOTO (COLOR)
Pilot testing of the instruments was conducted prior to the start of the study with two PGY 3 residents at another institution. The length of the survey was deemed appropriate and the instruments were used appropriately with wide ranges of responses. We observed a diagnostic accuracy of 70%, suggesting an appropriate level of difficulty in the ECGs.
Descriptive statistics were performed with the data from the demographic questionnaire, diagnostic accuracy, mental effort, and diagnostic certainty.
In assessing cue utilization, cue diagnosticity and monitoring accuracy, we performed a path analysis. Regression analyses were performed between (a) certainty and mental effort (cue utilization), (b) effort and diagnostic accuracy (cue diagnosticity), and (c) certainty and diagnostic accuracy (monitoring accuracy). The path coefficients are the standardized beta coefficients from the regression analysis.
All statistical analyses were performed with SPSS 23 (IBM, Redmond, WA).
Ethics approval was obtained from McMaster University Research Ethics Board.
Two hundred and twenty ECGs completed by 22 participants were analyzed.
Baseline demographics are presented in Table 1. The mean self-reported performance estimate for ECG interpretation was 70 ± 14%.
The average diagnostic accuracy was 65 ± 47%. The performance by ECG diagnosis is summarized in Table 2. The average mental effort was 4.4 ± 1.9. The mean diagnostic certainty was 66 ± 25%.
The results of the path analysis are summarized in Fig. 3. In exploring cue utilization, the path coefficient between mental effort and certainty was β = − 0.370, p < 0.001. In assessing cue diagnosticity, the path coefficient between mental effort and diagnostic accuracy was β = − 0.170, p = 0.011. In assessing monitoring accuracy, the path coefficient between certainty and diagnostic accuracy was β = 0.343, p < 0.001.Summary of path analysis results
PHOTO (COLOR)
To our knowledge, this is the first study to explore the utilization of mental effort as a cue, the cue diagnosticity of mental effort, and the monitoring accuracy of certainty as a monitoring judgement. Our observational design suggests that mental effort may indeed be used as a cue for certainty and is also diagnostic because higher effort negatively correlated with diagnostic accuracy. While we observed that certainty judgements were related to diagnostic accuracy, the moderate path coefficient between certainty and accuracy suggests that the monitoring accuracy is still far from perfect.
Our findings support the proposed use of mental effort to bridge Cognitive Load Theory and Self-Regulated Learning Theory (de Bruin and van Merriënboer 2017). Our study provides evidence for the hypothesis that mental effort could be a cue used to self-regulate performance: the analyses suggest that mental effort is a predictive cue (because it correlates with diagnostic accuracy) and mental effort was used by participants as a cue for certainty monitoring judgements.
There are two interesting aspects of the path analysis worth mentioning. Firstly, while the analysis supports our proposed framework, we cannot exclude alternative frameworks. The relative magnitude of path coefficients could suggest certainty may be more influential than mental effort within the set of variables studied. Future experimental studies are needed to further test the assumed relationships as well as alternative frameworks. Secondly, we acknowledge that the path coefficients are not very high. We hypothesize that there may be two alternative considerations. The first consideration is that, in addition to perceived effort, there may be other diagnostic cues used to inform certainty judgements. The second consideration is that there may be more appropriate performance outcomes than diagnostic accuracy. In clinical environments, we strive for accurate self-monitoring of performance to inform trainees of when they can move forward independently or when they need to ask for help. Perhaps asking the participants to indicate if they would have sought expertise from a colleague or supervisor to verify their answer could have been considered as the performance outcome.
There are limitations of this study worth discussing. The task of ECG interpretation through a hypothetico-deductive or scheme-based reasoning approach limits generalizing the results to tasks that could be solved through pattern recognition. Future work could explore the relationship between certainty and mental effort with different reasoning strategies. Furthermore, we did not counterbalance the order of mental effort and certainty ratings, which could have added further insight into the relationship between mental effort and certainty.
In conclusion, our results support the utilization of mental effort as a cue, the cue diagnosticity of mental effort, and the monitoring accuracy of certainty as a monitoring judgement. Given the reported correlations are small or moderate, further research is needed to further explore the cue diagnosticity of mental effort and to explore other potential cues to further optimize self-regulation of performance.
The authors are grateful to the support from the McMaster University Internal Medicine Residency program. The work was supported by the Western University Resident Research Career Development Program. The authors are appreciative of Dr. Geoff Norman's insightful direction to improve the conceptual framework and analysis of the study.
By Sarah Blissett; Matthew Sibbald; Ellen Kok and Jeroen van Merrienboer
