© 2004 by European Society of Cardiology
Copyright © 2004, The European Society of Cardiology
Report on the first written exam held as part of the European Association of Echocardiography Accreditation Process in Adult Transthoracic Echocardiography
European Association of Echocardiography, the Heart House, Nice, France
Received 15 June 2004; accepted after revision 15 June 2004.
* Corresponding author. Chairman of the Accreditation Assessment Committee, Department of Cardiology, Hammersmith Hospitals NHS Trust at Charing Cross Hospital, Fulham Palace Road, London W6 8RF, UK. Tel.: +44 208 846 1917; fax: +44 208 746 8182. k.fox{at}imperial.ac.uk
 |
Introduction |
|---|
 Top Introduction Structure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
In 2003, after widespread consultation, the European Association of Echocardiography (EAE) established a process for accrediting individuals in adult transthoracic echocardiography. To be awarded accreditation an applicant had to demonstrate theoretical knowledge and echocardiographic study interpretation skills by passing a written exam and provide evidence of training and practical experience by preparing a log book of 250 studies undertaken during a period of supervised training. An accredited echocardiographer was aimed to be one who had reached a standard to perform and report general adult transthoracic studies independently (approximately level 2–3 competence in AHA/ACC/ASE criteria).1
Two committees within the EAE were created. The Accreditation Committee provided an overview for the development of accreditation while the Accreditation Assessment Committee was charged with running the accreditation process, in particular the written exam and the assessment of practical experience. Full details of the accreditation process are published on the EAE website at www.euroecho.org.
This report describes the setting, organisation and evaluation of the first written examination held during the EuroEcho 7 meeting in Barcelona on December 6, 2003.
|
Structure of the exam |
|---|
|
TopIntroductionStructure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
The accreditation process and the exam in particular, drew on established accreditation processes including those of the American National Board of Echocardiography, Cardiovascular Credentialling International (an American technologist credentialling body) and the British Society of Echocardiography as well as other processes in place throughout Europe.
A pilot exam was run during the EuroEcho 6 meeting in Munich in December 2003. A group of 31 volunteers of mixed, but mainly extensive, experience answered theory and echo reporting questions which provided valuable information to advise on the optimum format, standard and pass mark.
Integrating the information from other accreditation processes and the pilot exam, multiple choice questions (MCQs) were chosen as the preferred format. MCQs allow rapid, objective and accurate marking, which with modern computer systems also provides detailed feedback on the performance of candidates and questions.
The detailed format chosen was ‘single correct response’ (also called single best answer) which discriminates effectively between candidates. In this format each question has 4 possible responses. One is correct and the other 3 are incorrect (confounders).
The exam had 2 sections. The first section (100 questions over 90min) tested theoretical knowledge including ultrasound physics. The second section was based on reporting of echo case studies shown as a series of digital clips and stills. Ten studies were shown with 5 MCQs related to each study (total 50 MCQs over 90min). Overall therefore 150 MCQs were asked to allow adequate discrimination between pass and fail candidates.
In both the sections correct answers scored 1 mark with no deduction for incorrect responses. It was felt particularly important to keep the exam process as straightforward as possible in view of the varied background and language skills of the candidates, and negative marking can be seen as daunting by candidates and probably does not improve the discrimination of exams.
The University of London Computer Marking Service was used.
|
Setting the exam |
|---|
|
TopIntroductionStructure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
The Accreditation Assessment Committee undertook the setting of the exam questions based on the syllabus agreed by the Accreditation Committee. The questions were focussed not as a test of English, nor as a test of Cardiology or Medicine, but as a test of echocardiography.
Theory section questions were set predominantly by a core group of 3 people from different countries. Approximately 25% of questions were modified directly from questions used in other accreditation processes as a benchmark for the new questions. In addition to the 3 question setters checking each other's questions another examiner checked all the questions before the final theory section was agreed.
The echo reporting section drew on cases from 4 echo digital archives. Questions were checked by 5 individuals before the final paper was agreed. The exam was written in English with a glossary of key terms in Spanish, French, German, Italian and Romanian. Invigilators in the exam room also encompassed Portuguese and Russian.
An observer from the European Board of Accreditation in Cardiology (EBAC) was invited to observe the conduct and process of the examination. The EBAC staff also analysed the candidate feedback questionnaire.
|
The candidates |
|---|
|
TopIntroductionStructure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
One hundred and six candidates sat the exam. The countries of origin of the 97 candidates who registered for the exam before the start of the EuroEcho meeting are shown in Fig. 1 and the first languages, where known, are depicted in Fig. 2.
|
|
Over 80% of pre-registered candidates had been undertaking echocardiographic examinations for at least 5 years. Similar data on experience were not available for the 9 on-site registered candidates but many of those applied have attended the training course that ran during the EuroEcho 7 meeting and therefore are likely to have had much less than 5 years of experience.
|
Candidates' marks |
|---|
|
TopIntroductionStructure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
The candidates' marks are shown in Fig. 3. The mean mark for the theory section was 68.6/100 (SD 12.5). Ten candidates did not complete all 100 questions in the allotted time. The mean mark for the echo reporting section was 36.2/50 (SD 4.1). The on-site registered candidates who as explained above, were likely to have the least experience, performed less well in both sections of the exam compared to the pre-registered candidates. This provides some evidence of the validity of the questions.
|
|
Establishing the pass mark |
|---|
|
TopIntroductionStructure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
The information for candidates stated that the projected pass mark was 65/100 for the theory section and 30/50 for the echo reporting section and questions were set on that basis.
However, it was also pre-arranged that the pass mark would be set by the Accreditation Assessment Committee after analysing the candidates and their marks to take into account the actual difficulty of the exam as reported by candidates and invigilators, the experience of the candidates, and the distribution of marks. The projected pass mark for each section could then be adjusted by integrating a number of factors.
- 1. The absolute standard of knowledge required was estimated to be 65/100 in the theory section and 30/50 in the echo reporting section. As a minimum it was felt that a score of <50 (theory) and <25 (echo reporting) would not be compatible with an adequate standard of knowledge, and any candidate scoring >80 and >40 in the 2 sections should certainly pass. This established a minimum and a maximum pass mark.
- 2. Approximately 80% of the cohort were experienced echocardiographers, the great majority of whom would be expected to be of a standard at least as high as EAE accreditation. Some of the less experienced candidates would also have had substantial theoretical knowledge and so a pass rate of >80% could be expected.
- 3. Some candidates, including those with considerable experience, did not complete all 100 questions in the theory section, which correlated with candidate feedback (see below) that the theory section was the more challenging. This indicated a need for a downward adjustment of the pass mark.
- 4. The clustering of marks indicated a large group that performed similarly and outliers, distinguishable from this main group, who performed less well. Failing candidates by a very small margin of marks from the centre of this main cluster may fail a candidate whose knowledge is adequate but chance, or 1 or 2 poor questions, has separated them from the mean mark.
- 5. The information and factors can be combined in various ways. Combining knowledge of the questions and the candidates, and pre-existing concepts of the standard required together with acceptable minimum and maximum pass marks, can be done with the ‘Hofstee’ procedure whose principles were used here.2
- 2. Approximately 80% of the cohort were experienced echocardiographers, the great majority of whom would be expected to be of a standard at least as high as EAE accreditation. Some of the less experienced candidates would also have had substantial theoretical knowledge and so a pass rate of >80% could be expected.
Taking these factors into consideration the Accreditation Assessment Committee established the pass mark at 55/100 for the theory section and 30/50 for the echo reporting section. Candidates had to pass both sections to pass the overall exam. 91/106 (86%) candidates passed the exam. Fourteen candidates failed the theory section, 7 failed the echo reporting section and 6 failed both sections. 5/9 on-site registered candidates failed compared to 10/97 pre-registered candidates.
|
The performance of the exam questions |
|---|
|
TopIntroductionStructure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
The questions are designed to assess the knowledge and interpretive skills of candidates against an absolute standard and in doing so to discriminate between candidates whose ability is adequate from those whose knowledge is not adequate.
Evidence that the questions performed well as an absolute test comes from the fact that those with more experience fared better than those with less experience and the correlation of scores in the 2 sections also supports this contention (the less likely alternative is that both sections were equally bad at testing ability). The range of difficulty (range of proportion answering questions correctly) was also, as hoped, well distributed.
Discriminatory ability can be tested for MCQ using a number of different formulae. Broadly all methods test how candidates who ultimately scored well in the overall exam performed on a particular question compared to those who ultimately scored less well. A highly discriminating question is one where the overall best candidates all get it right and the worst all get it wrong. The responses can be analysed mathematically in a number of ways to produce a discrimination index. The one used in this exam was the Biserial correlation3.
In the theory section 81% of questions were acceptable (Biserial correlation 0.2–0.35) or very good (>0.35) discriminators. Neither length of question, nor the inclusion of a single negative affected discrimination. However, many of the poor discriminators were questions where both the question and the responses were complex statements.
In the echo reporting section 38/50 (76%) of individual questions were acceptable or very good discriminators. The commonest questions which appeared to be poor discriminators related to the grading of valve lesions and left ventricular function. Consensus, even amongst experts, is not always achieved and in 6 questions relating to gradings, the candidates disagreed with the examiners. Cases were chosen to be clear and also to test common misconceptions about grading and so these 6 questions do not necessarily represent examiner error.
Overall, therefore, the exam performed well in identifying candidates with an appropriate standard of knowledge and interpretive skill.
|
Candidates' feedback |
|---|
|
TopIntroductionStructure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
Data from the anonymised candidate feedback questionnaires are illuminating. One hundred and three questionnaires were returned. Much of the information was provided in the form of subjective linear gradings from 1 to 10 on various issues in the exam. Candidates felt the theory section questions were more difficult than the echo reporting questions. In the feedback questionnaire candidates indicated 1 if questions were felt to be very easy overall and 10 if very difficult for each section. The mean score for the theory section questions was 7.4±1.5. The mean score for the echo reporting questions was 6.2±2. In terms of the process of the exam the echo images were felt to be clear (mean score 7.6/10). Similarly the instructions, printed on the paper and shown on PowerPoint slides at the start of the exam, were generally regarded as clear (mean score 8/10).
In contrast, 41% of candidates felt that the time was short. In 90% of cases these comments related to the theory section. One major factor was whether conducting the exam in English would be a barrier. Clearly candidates in this exam were biased in that they knew the exam was in English before applying but it was still important to record that only 10% felt language was a barrier.
|
Further development of the exam |
|---|
|
TopIntroductionStructure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
Improvements to the exam can and should be made. More time should be allowed, particularly for the theory section where an extra 30min may be appropriate, and the questions were made shorter and clearer wherever possible. While the evidence from the first exam was that this was not a test of speed or English it is important that this concern is kept constantly in mind when the questions for next exam, to take place at the ESC Congress in August 2004, is set. Poorly performing questions should not be reused and a bank of effective questions should be continuously expanded and updated.
|
Conclusions |
|---|
|
TopIntroductionStructure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
This enterprise represents one of the first attempts to organise an exam for health care workers from across Europe as part of a process to establish a common competency. It is to the great credit of the EAE that they have initiated and developed such a process in their area of echocardiography. A multinational cohort of mixed ability candidates successfully completed the exam and the marks and pass rates fulfilled criteria for identifying absolute and relative competence.
Overall, therefore, the first EAE exam as part of Accreditation in Adult Transthoracic Echocardiography was a great success and credit to the large team of people involved. Successful candidates must now complete the practical section to be awarded accreditation. The exam will develop, but the feasibility of a pan-European exam in adult echocardiography has been established.
|
Acknowledgements |
|---|
The committee would like to thank Dr Joanna Ortoli and the staff of the EBAC for their analysis of the candidate feedback questionnaires, the staff of the European Heart House, especially Ms Barbara Lefevre (EAE Administrator) Ms Dawn Appleby (British Society of Echocardiography administrator) and the many question setters and invigilators for the exam.
|
Notes |
|---|
|
TopIntroductionStructure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
1 The Accreditation Assessment Committee of the European Association of Echocardiography.
|
References |
|---|
|
TopIntroductionStructure of the examSetting the examThe candidatesCandidates' marksEstablishing the pass markThe performance of the...Candidates' feedbackFurther development of the...ConclusionsNotesReferences |
|---|
- American College of Cardiology/American Heart Association/American College of Physicians–American Society of Internal Medicine Task Force. American College of Cardiology/American Heart Association Clinical Competence Statement on Echocardiography. Circulation (2003) 107:1068–1089.
[Free Full Text] - Aronson S., Butler A., Subhiyah R., Buckingham R.E., Cahalan M.K., Konstandt S., et al. Development and analysis of a new certifying examination in perioperative transesophageal echocardiography. Anaes Analg (2002) 95:1476–1482.[CrossRef]
- University of London Management Information Services Division. An introduction and guide to the use of multiple choice questions in university examinations. (1986) University of London.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  B. A. Popescu, M. J. Andrade, L. P. Badano, K. F. Fox, F. A. Flachskampf, P. Lancellotti, A. Varga, R. Sicari, A. Evangelista, P. Nihoyannopoulos, et al. European Association of Echocardiography recommendations for training, competence, and quality improvement in echocardiography Eur J Echocardiogr, December 1, 2009; 10(8): 893 - 905. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A.G. Fraser, P.T. Buser, J.J. Bax, W.R. Dassen, P. Nihoyannopoulos, J. Schwitter, J.M. Knuuti, M. Hoher, F. Bengel, and A. Szatmari The future of cardiovascular imaging and non-invasive diagnosis: A joint statement from the European Association of Echocardiography, the Working Groups on Cardiovascular Magnetic Resonance, Computers in Cardiology, and Nuclear Cardiology of the European Society of Cardiology, the European Association of Nuclear Medicine and the Association for European Paediatric Cardiology Eur J Echocardiogr, August 1, 2006; 7(4): 268 - 273. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. G. Fraser, P. T. Buser, J. J. Bax, W. R. Dassen, P. Nihoyannopoulos, J. Schwitter, J. M. Knuuti, M. Hoher, F. Bengel, and A. Szatmari The future of cardiovascular imaging and non-invasive diagnosis: A joint statement from the European Association of Echocardiography, the Working Groups on Cardiovascular Magnetic Resonance, Computers in Cardiology, and Nuclear Cardiology, of the European Society of Cardiology, the European Association of Nuclear Medicine, and the Association for European Paediatric Cardiology. Eur. Heart J., July 1, 2006; 27(14): 1750 - 1753. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||