The pursuit of fairness in assessment: Looking beyond the objective

Health professions education has undergone significant changes over the last few decades, including the rise of competency-based medical education, a shift to authentic workplace-based assessments, and increased emphasis on programmes of assessment. Despite these changes, there is still a commonly held assumption that objectivity always leads to and is the only way to achieve fairness in assessment. However, there are well-documented limitations to using objectivity as the 'gold standard' to which assessments are judged. Fairness, on the other hand, is a fundamental quality of assessment and a principle that almost no one contests. Taking a step back and changing perspectives to focus on fairness in assessment may help re-set a traditional objective approach and identify an equal role for subjective human judgement in assessment alongside objective methods. This paper explores fairness as a fundamental quality of assessments. This approach legitimises human judgement and shared subjectivity in assessment decisions alongside objective methods. Widening the answer to the question: 'What is fair assessment' to include not only objectivity but also expert human judgement and shared subjectivity can add significant value in ensuring learners are better equipped to be the health professionals required of the 21st century.

Keywords: Clinical; postgraduate; undergraduate; general; assessment

Fourteen years ago, Schuwirth and van der Vleuten made a plea for new psychometric models in education assessment (Schuwirth and Van der Vleuten [57]).

Practice points

• Objective approaches remain a dominant discourse in assessment.
• There are limitations to using objectivity as the 'gold standard' to which assessments are judged.
• Within the literature, there is an increasing push to better utilise human judgement and subjectivity in assessment.
• Changing perspectives to focus on the fundamental underlying value of fairness in assessment may help re-set a traditional objective approach.

Their paper argued 'Assessment should be fair, honest and defensible...but the strict operationalisation of these values is–in our humble opinion–currently of limited value' (Schuwirth and Van der Vleuten [57]). They made an appeal for a major revision of statistical concepts, approaches to assessment, and the development of a new model that fits current assessment developments better (Schuwirth and Van der Vleuten [57]). Indeed, around the turn of the century, many changes were made to medical education assessment. Competency-based medical education became the dominant approach to medical education in many countries (Ten Cate [59]). With this, the role of the doctor was redefined to include features that had previously not been emphasised, and learners were certified based on outcome rather than the input (Ten Cate and Billett [60]). Assessment of clinical competence moved from written assessments back into the authentic context of the workplace, and individual assessments made way for programmes of assessment (Dauphinee [11]; Van der Vleuten and Schuwirth [67]; Valentine and Schuwirth [65]). More recently, competencies have been defined into professional tasks which a learner is entrusted to complete independently (Ten Cate and Scheele [62]).

Throughout these times of change, objective approaches have still remained a dominant discourse in assessment, with many seeing objectivity as the 'gold standard' to which assessments should be judged (Van der Vleuten et al. [66]; Govaerts and Van der Vleuten [30]; Ten Cate and Regehr [61]; Valentine and Schuwirth [65]).

More recently, there has been an increasing push in the literature to better utilise the role of human judgement and subjectivity in assessment (Jones [38]; Schuwirth and Van der Vleuten [57]; Govaerts and Van der Vleuten [30]; Hodges [37]; Gingerich et al. [25]; Bacon et al. [4]; Rotthoff [56]; Ten Cate and Regehr [61]) and in 2020, the Ottawa consensus statement report for performance in assessment specifically called for assessment programs to 're-instate expert judgement' (Boursicot [6]).

Perhaps with the benefit of a decade and a half of hindsight, we could say this 2006 paper (Schuwirth and Van der Vleuten [57]) didn't go quite far enough as it was still looking for new psychometric methods with an 'objective' mindset. Objectivity which De Groot defined as 'judgement without interference or even potential interference of personal opinions, preferences, modes of observation, views, interests or sentiments' (Ten Cate and Regehr [61]) has frequently become synonymous with fairness and is often used to determine the quality of the assessment. Workplace-based assessments, designed to assess authentic performance, are often judged using a quantitative psychometric framework and therefore criticised for not meeting validity and reliability criteria (Govaerts and Van der Vleuten [30]). However, an exclusive focus on traditional psychometric approaches can disregard key issues of competence, performance, and assessment in complex workplace settings (Govaerts and Van der Vleuten [30]). Taking a step back and changing perspectives to focus on the fundamental underlying value of fairness in assessment may help re-set the traditional objective approach.

Fairness is a fundamental quality of assessment and a general principle that no one contests (Green et al. [33]; Tierney [63]; General Medical Council [24]). However, fairness is not a simple construct that is easy to define or conclusively described in the literature. There is no simple, all-encompassing consensus definition of fairness. Fairness has been associated with a wide range of assessment-related qualities, such as equitable, consistent, balanced, useful, and ethically feasible. This breath demonstrates that fairness in assessment is a comprehensive term and not something that can be reduced to a number, determined dichotomously, or a straightforward process (Tierney [63], [64]). More recently, Shaw and Nisbet considered fair assessment in the light of COVID-19 and similarly identified multiple challenges to consider in defining fairness (Shaw and Nisbet [58]). Simply equating fairness to objectivity, can reduce a complex, diverse multi-dimensional, context-dependent construct to a single linear, likely non-representative rule. If the lens of improvement remains on optimising objectivity, then the focus is on better psychometric techniques, but if we return to the principle of fairness, then the focus becomes much wider.

In this paper, we seek to focus more broadly on fairness as a fundamental quality of assessment, not in a traditional systematic review but rather by synthesising and linking literature, identifying established knowledge and perspectives, highlighting gaps in understanding, and providing direction on what remains to be understood (Eva [20]). We look to challenge the often-held assumption that objectivity always leads to, and is the only way to achieve fairness in assessment, and suggest that subjective human judgement has a legitimate place alongside objectivity in fair assessment. Two arguments will be put forward to challenge this assumption. Firstly, we consider the contention that objectivity can comprehensively assess the complexity of clinical practice to be a fallacy, and secondly, that true objectivity in assessment is unobtainable. Finally, from the collation of the perspectives in the literature, we will suggest that focusing on fairness rather than objectivity ensures that expert judgement and shared subjectivity can be seen as at least equal to and used in combination with objectivity. We will also discuss opportunities for future research from a fairness lens.

Whilst there are many benefits to objectivity in assessment, it can lead to a naïve trust in linear causality, a reliance on reproducible, quantitative, and psychometric data, and reification. During the Vietnam War, Robert McNamara, the US Secretary of Defence, quantified the war effort into metrics, such as body counts and territory gained. As a past chairman of the Ford Motor Company, McNamara applied objective, quantified metrics to improve production lines with great success. However, war is a complex non-linear, and largely unpredictable process, and the approach of reducing complex human processes to body counts and territory gained failed as it ignored the actions of highly motivated people and the chaos and destruction of war (McNamara [44]). This led to the McNamara fallacy, which was coined by Yankelovich (Figure 1) (Yankelovich [72]).

Graph: Figure 1. The McNamara fallacy.

Medicine, like war, is also complex, non-linear, and to a certain extent unpredictable. Complex systems are characterised as having multiple, dynamic components interacting in non-linear and unpredictable ways, where the whole system is more than the sum of the parts (Reed et al. [55]). Health professionals work with complex problems presented in a variety of different ways and through multiple dimensions of human experience (biological, psychological, social, spiritual). Treatment decisions are often made in the face of uncertainty as every patient responds differently to the array of therapeutic options (Kaldjian [39]). Furthermore, society expects health professionals to have not only knowledge of established and evolving diseases, but also interpersonal and communication skills, be able to apply ethical principles, and so on (Norcini et al. [49]).

McNamara was blindsided by the data, convinced the USA was winning the war despite his commanders telling him the exact opposite (Carmody [7]). Similarly, within clinical practice, equating 'quality' with someone who strictly adheres to guidelines or protocols, is to overlook the evidence on the more sophisticated process of expertise (Greenhalgh et al. [34]). With regard to assessment, any effort to prioritise and quantify one aspect of trainees' qualities, such as knowledge, will inevitably reduce the emphasis on other aspects which might be deemed important (Eva [21]). It could actually be argued that objectivity can reduce fairness because it only measures what can be measured by a quantitative value. This is unfair to learners with broader skills and unfair to society who highly value unquantifiable competencies, such as compassion, kindness, and courage in their health care professionals (O'Mahony [52]; Wayne et al. [70]). These skills, as well as other not easily quantifiable skills, such as communication, collaboration, and professionalism are often the ones needed within our health care systems (Frank et al. [23]). Such reductionist approaches may also carry the risk of negatively impacting student learning behaviour. Cilliers et al. demonstrated that the effects of assessment on student learning are complex. Overreliance on 'objectivity' and quantitative results were perceived as punitive and unfair, and encouraged students' learning activities to be directed to passing assessments rather than learning to become a good clinician (Cilliers et al. [8]; Cilliers et al. [9]).

There are also further limitations to the use of objectivity in assessment. Assessment is always an evaluative process and therefore subjective. In the late 20th century, medical education assessments moved back into the authentic context of the workplace to help ease the tension between what is being measured and what should be measured (Rotthoff [56]; Boulet and Durning [5]). However, in an attempt to remain true to the paradigm of objectivity, assessments, such as the objective structured clinical examination (OSCE), were designed to minimise human judgement as much as possible. This was believed to improve fairness (Norcini et al. [51]; Gingerich et al. [25]; Ten Cate and Regehr [61]; Valentine and Schuwirth [65]). Objectivity, from a positivist perspective that has played a prominent role in health professions education, suggests that for each item being measured, a 'true' score exists and any deviation from this true score is a measurement error (Ten Cate and Regehr [61]). However, even an 'objective' multiple-choice examination involves a series of judgements by experts: what topics should be included, the choice of questions, specific wordings, decisions about pass scores, and so on (Norcini and Shea [50]; Ten Cate and Regehr [61]; Valentine and Schuwirth [65]). And, as other authors have noted, these judgements are rarely unanimous, often requiring complex negotiation between experts, which based on De Groot's definition, is far from objective (Ten Cate and Regehr [61]).

This can also be said of all quantitative measurement scales. Downie and colleagues stated 'if the underlying purpose of questionnaires and measurement scales is to avoid the need for judgement then it does not succeed' (Downie and Macnaughton [15]). Judgement is required in deciding what questions to ask, what numbers to assign, and how to interpret final scores. Judgements can be dangerous when the professionals are unaware they are making them and believe themselves to be 'objective' (Downie and Macnaughton [15]). Within the health professions education literature it has been assumed that expert practitioners have a shared understanding of what competency-based assessment is and the criteria for a competent performance (Apramian et al. [1]). But, assessors have been shown repeatedly to have different interpretations of individual performances (Apramian et al. [3]; Ten Cate and Regehr [61]), different perceptions of whether a performance upholds competence principles (Bacon et al. [4]; Apramian et al. [1]), and make different inferences about knowledge, skills, and attitudes which can't be directly observed (Kogan et al. [40]; Gingerich et al. [25]). Furthermore, it has been noted that assessors attach varying importance to different aspects of assessments or clinical procedures (Kogan et al. [40]; Apramian et al. [2], [3]) as well as attach importance to factors outside of competency frameworks (Oudkerk Pool et al. [53]). Assessors draw from multiple frames of reference (Kogan et al. [41]; Kogan et al. [40]; Yeates et al. [73]; Bacon et al. [4]; Apramian et al. [1]) and use various methods to synthesise judgements into numerical ratings (Kogan et al. [40]).

Moreover, the complexity of the task and the context of the work environment also influence assessment decisions (Gingerich et al. [26]). Returning to McNamara, The Economist observed 'he was haunted by the thought that amid all the objective-setting and evaluating, the careful counting and the cost-benefit analysis, stood ordinary human beings. They behaved unpredictably' (O'Mahony [52]). Workplace-based assessment occurs in environments where people are free to act in ways that are not predictable, and whose actions are interconnected so that one person's actions change the context for other people (Plsek and Greenhalgh [54]; Mennin [45]; Greenhalgh and Papoutsi [35]). For example, no one can predict what a patient will say in 3 min time so there can be no protocol to access the learner taking the history. In their 2006 paper, Schuwirth and van der Vleuten noted 'We dismiss variance between observers as an error because we start from the assumption that the universe is homogeneous, where in fact the more logical conclusion would have been that the universe is more variant' (Schuwirth and Van der Vleuten [57]).

This difficulty in obtaining agreement and a 'single truth' is not surprising because decision-making is idiosyncratic and individual (Durning et al. [18]; Bacon et al. [4]). The psychology and cognitive science literature note that decision-making processes are highly complex, subject to multiple influences and no single theory of learning or performance can fully represent the underlying mechanisms. Furthermore, human working memory is thought to only hold approximately seven information elements at a time, and actively process no more than two to four elements at a time (Young et al. [74]). To overcome a limited working memory, information is rearranged and connected to pre-existing knowledge frameworks of information (schema) activated from long-term memory (Moulton et al. [46]; Young et al. [74]). These pre-existing schemas are used to judge the 'new' information being observed and influence what judgements are reached and an assessor's recall of what occurs. In assessment, assessors are active information processors who recognise, select and interpret relevant information, and integrate this information using their past experiences as well as their understandings of their social, cultural, and contextual surroundings to form impressions and assign ratings (Gingerich et al. [26]; Govaerts et al. [31]; Govaerts and Van der Vleuten [30]). Previously, assessor variability has been framed primarily as a 'training issue' with the belief that the assessor is trainable. However, widespread assessor-agreement, improved test psychometrics and reduced measurement 'error' has remained elusive despite extensive efforts at faculty training (Newble et al. [48]; Downing [16]; Gingerich et al. [26]; Gingerich et al. [25]). Delandshere and Petrosky noted: 'Judges' values, experiences, and interests are what makes them capable of interpreting complex performances, but it will never be possible to eliminate those attributes that make them different, even with extensive training and 'calibration' (Delandshere and Petrosky [12]).

One common attempt to overcome this variability has been to simply exclude outliers to ensure a common perspective and perceived reliability among the remaining raters (Newble et al. [48]). However, as noted by other authors, this approach does not exclude subjectivity—at best it masks subjectivity behind a constructed consensus (Ten Cate and Regehr [61]) and perhaps eliminates the outliers who are unwilling to modify their assessment despite fear of unpleasant repercussions (Kogan et al. [40]). The development of the OSCE approach (Harden and Gleeson [36]) was an illustration of reducing this assumed assessor-related 'error' through the process rather than 'objectifying'. Although standardisation was initially prescribed through checklists, a sampling framework was also developed (having the candidate rotate from examiner to examiner) which accepted variability in assessors. The narrative at that time was psychometric but through the current lens, this can be seen as a procedural approach to ensuring fairness.

Pursuing fairness in assessment through objectivity has many benefits but also has limitations. Firstly, it overlooks the complexity of clinical and educational practice and the wide variety of skills demanded of health professionals. Secondly, despite multiple efforts over several decades at both internal (such as faculty training) and external solutions (such as structured forms), researchers have not satisfactorily managed to achieve 'bias-free objectivity'.

The call to move away from an over-reliance on an objectivity paradigm has been echoed throughout the literature for several decades now. Gingerich and colleagues have suggested that perhaps the time has come to acknowledge a 'single' truth does not exist and consider an alternative conception of rater 'error' (Gingerich et al. [25]) and in 1999 Jones wrote that it is time 'to acknowledge that the role of a professional vocational educator is to make educational judgements' (Jones [38]). These views have been echoed by other authors as they call for health professions education to have the courage to acknowledge the benefits of subjectivity in assessment (Govaerts and Van der Vleuten [30]; Hodges [37]; Bacon et al. [4]; Rotthoff [56]; Ten Cate and Regehr [61]).

Changing perspectives to look at fairness in assessment, rather than objectivity in assessment, allows for many different legitimate perspectives. Ten Cate and Regehr argue 'objectivity' might be better understood as negotiating a 'shared subjectivity': a convergence on an agreed-upon and socially constructed perspective (Ten Cate and Regehr [61]). They note that although this convergence might achieve consensus, it isn't bias-free objectivity (Ten Cate and Regehr [61]). However, at the core of subjectivity is human judgment. To embrace subjectivity requires embracing human judgement and accepting that multiple different perspectives are not measurement errors to be corrected psychometrically but rather legitimate complementary perspectives of performance, much like how different angles of an object are present in different but still 'correct' visions. And if we accept this premise, then we must develop new ways to determine the fairness of the judgements supervisors make.

The idea of human judgement and shared subjectivity in health professions education is widely used through competence committees and exam boards, especially since the introduction of programmatic longitudinal assessment (Van der Vleuten et al. [68]). Ten Cate and Regehr argue there are many positives in embracing multiple legitimate perspectives on a single performance, including helping the learner be alert to the fact that in real life, there are multiple ways in which behaviours can be interpreted and respond to accordingly (Ten Cate and Regehr [61]). Differences in opinions may be noise in a psychometric perspective, but these differences may be very beneficial for the learning of an individual. Through the use of narrative instead of numbers, human judgement offers meaningful learning affordances through the possibility of richer feedback. Furthermore, the use of descriptive narrative in assessment has been shown in several studies to be a sensitive way of identifying at risk learners earlier (Cohen et al. [10]; Durning et al. [19]), is a reliable way to distinguish between learners even after only a few assessment reports (Ginsburg et al. [27]; Ginsburg et al. [28]) and can predict future performance or need for remediation (Cohen et al. [10]).

Accepting and legitimising human judgement allows for context to be considered and this may be more defensible as assessors are not forced to document a context-free inference about the learner (Ten Cate and Regehr [61]). Furthermore, 'objectivity' in assessment can lead to many assessment forms aiming to be context-independent, where assessors are forced to make judgements on a wide range of competencies not observed or in the context of the clinical situation every time they complete an assessment form. Some authors have noted this can diminish the learners' trust in the assessor and process, and hides potentially credible decisions in a 'mountain of meaningless platitudes' (McCready [43]; Watling [69]). Subjectivity may help overcome this.

The human judgement also has its limitations. Multiple studies have demonstrated actuarial methods are superior to clinical prediction in many situations (Marchese [42]). The use of clinical guidelines has improved patient outcomes in many scenarios, such as heart failure, breast cancer, atrial fibrillation, ventilator-assisted pneumonia, and so on (Murad [47]). In both medical education and clinical medicine, objectively derived scales, guidelines, and matrixes are essential tools. However, these tools need to be used smartly. As Woolf and colleagues note it, 'clinical guidelines are useful when practitioners are unclear about appropriate practice and when scientific evidence can provide an answer. They are a poor remedy in other settings' (Woolf et al. [71]). Human judgement is essential in determining when and how to apply these tools. We have tried to emphasise in this paper, that a desire to better utilise and legitimise subjective judgements in assessment, is not to dismiss the work done on objectivity in assessment over the last century. Nor does acknowledging that quantitative multiple-choice tests are in fact based on a series of subjective decisions, mean they no longer have a place in modern assessment. Numerical ratings and standardised assessments are valuable elements in fair assessment. Instead, objectivity, subjectivity, and human judgement are tensions that should be reconciled (Govaerts et al. [32]). Recognising the role of human judgement in assessment acknowledges that alongside knowledge tests there needs to be the assessment of professional capabilities, and alongside debates about psychometrics, there need to be debates about the credibility and defensibility of human judgements. It isn't an either-or, but rather a careful balancing of approaches in assessment programmes (Govaerts and Van der Vleuten [30]; Govaerts et al. [32]). Changing focus from objectivity to fairness can assist with this. Widening the answer to the question: 'What is fair assessment' to include human judgement can add significant value in ensuring a broader range of capabilities are being assessed to ensure learners are better equipped to be the health professionals required of the 21st century.

In increasingly complex clinical and educational environments, the challenge is to continue to move beyond the assumption that objectivity always leads to, and is the only way to achieve fairness. Changing the focus from workplace-based assessment being judged in terms of an objective psychometric framework to the assessment being judged in terms of fairness will help avoid falling prey to McNamara's fallacy and ensure we fulfill our social contract with society to train health professionals who can thrive in this ever-changing environment, whilst remaining fair to the learners themselves. If we can move beyond the objective paradigm in our pursuit of fairness in assessment, we can start to explore shared subjectivity and human judgement in more depth. And a different ontological understanding of what makes human judgements and shared subjectivity fair in assessment is crucial.

There is no consensus roadmap to determine what is fair assessment conveniently published in the literature. Developing a deeper understanding of what fair human judgement looks like, how this can be defined, how it can be optimised for learning, and how this can be supported is needed. What are the essential foundations of fairness and how can these be applied to judgements in the complex environment of workplace-based assessment? It has been suggested fairness of human judgement can be enhanced through the use of a palette of assessor perspectives, the combination of multiple assessments, the use of narrative and paper trails in judgement decisions (Dijkstra et al. [14]; Dijkstra et al. [13]; Van der Vleuten et al. [68]). Some authors have also suggested looking to qualitative research strategies as an alternative to build rigour in assessment (Driessen et al. [17]; Frambach et al. [22]), however further research is still needed. Gipps and Stobart noted 'We will never achieve the fair assessment, but we can make it fairer' (Gipps and Stobart [29]). And in 21st century clinical practice, perhaps we can make the health profession's assessment fairer by looking beyond the objective paradigm.

The authors declare that they have no conflict of interest. The views expressed herein are those of the authors and not necessarily those of the Department of Defense, Uniformed Services University of the Health Sciences or other Federal agencies.