In a recent editorial opinion in Orthodontic Update it was argued that removing treated cases in the RCSEd MOrth examination will be detrimental to the overall quality of Orthodontic training. RCSEd contends that the assessment of treated cases is more appropriate as part of the longitudinal review of progression through the ARCP process and this change aligns the examination with modern educational pedagogy. We argue that the changes to the assessment format make for a fairer and more robust exam.
Common criteria for evaluating assessment utility include validity, reliability, feasibility, educational impact, and acceptability, the first three being most critical for high stakes summative assessments such as MOrth. [1,2] These criteria were more difficult to meet with the old-style examination that used self-selected cases.
The reliability criterion relates to accuracy and reproducibility of scores. This too can be compromised using the old-style exam. When using self-selected cases, candidates each submit different material with at varying levels of complexity. Candidates are then asked different questions that relate to their specific cases. This lack of standardization and equivalence can lead to an “uneven playing field” for candidates. The small number of cases sampled is also likely to impact negatively on reliability.
The self-selected case approach can have a negative or undesirable educational impact, impact if undue time is spent perfecting the presentation of a small select group of cases.
It should be recognised that the examination is only one part of the longitudinal process of assessment which a modern trainee must undertake. This includes entry of the full range of cases treated during training on to the Intercollegiate Surgical Curriculum Programme [3] portal and verified through the Annual Review of Competency Progression [4] process. This is the key area where any contention about case management should rest and not at a single examination point. Such a process allows external validation of the range of cases treated and, assuming it is properly managed, should ensure they fulfil the requirements set out in the training blueprints, ranging from the NHS England Commissioning Standards [5] to the SAC / JCPTD specialty training curricula [6] (the latter currently under revision). This element of the training is overseen by four national Health Education Boards and the blueprint to be covered and assessed by the GDC.
The self-selected case approach can also lead to Feasibility issues where a candidate fails this aspect of their examination but has reached the end of their training. The problem then arises as to how they will be able to provide other appropriate cases prior to the retake sitting.
Acceptability is an area where there is ongoing debate. Some see merit in continuing with self-selected cases and argue that they provide a chance to test technical competence. However, a written report with structured oral would not be considered an appropriate approach for assessing technical skills. These should be directly observed and sampled sufficiently as to generate reliable scores. The self-selected case approach does not support these requirements. Again, the evidence points to this area being addressed more appropriately in the workplace throughout training.
The key question is what have we replaced these cases with? Essentially the treated cases will be replaced by 4 unseen cases related specifically to orthodontic treatment mechanics as below.
The Part B examination will therefore consist of:
4 x 30 minutes structured oral examination related to diagnosis, treatment planning and patient care Please note that there is no change to this component)
An additional 4 x 30 minutes structured oral examination related to diagnosis, treatment planning and management of orthodontic treatment mechanics with four sets of patient records.
A one-hour Communications Examination over four stations, on aspects of communication relating to orthodontic treatment (Please note that there is no change to this component).
This allows RCSEd to better ensure syllabus content is covered at the appropriate level consistently across candidates and that no candidate is disadvantaged. This can be achieved without the loss of the case-based approach which can continue as an assessment more robustly assessed in the workplace.
We thank the editors for this opportunity to highlight the advantages conferred by the new exam format. It is important that all trainees and trainers understand the underpinning rationale for their examinations, and the need to employ current best evidence in their design and delivery. This ensures fair outcomes for the trainees, the profession and for patients and the GDC, provision of safe competent care.
- Professor Philip D. Taylor, Dean of the Faculty of Dental Surgery
References
1. Van Der Vleuten, C.P. and Schuwirth, L.W., 2005. Assessing professional competence: from methods to programmes. Medical education, 39(3), pp.309-317.
2. Norcini, J., Anderson, B., Bollela, V., Burch, V., Costa, M.J., Duvivier, R., Galbraith, R., Hays, R., Kent, A., Perrott, V. and Roberts, T., 2011. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Medical teacher, 33(3), pp.206-214.
3. https://www.rcsed.ac.uk/view?id=ca824300-99bf-40f1-98db-585eed64d21f&type=exam
4. Miller GE Assessment of clinical skills/ competence /performance Acad Med. 1990 Sep;65(9 Suppl):S63
5. https://iscp.ac.uk/iscp/curriculum_2021/
6. https://specialtytraining.hee.nhs.uk/ARCP
7.https://www.england.nhs.uk/commissioning/wpcontent/uploads/sites/12/2015/09/guid-comms-orthodontics.pdf
8.https://www.gdcuk.org/docs/defaultsource/specialistlists/orthodonticcurriculum.pdf?sfvrsn=76eecfed_2