Copyright © 2006, The European Society of Cardiology
Report on the European Association of Echocardiography Accreditations in Echocardiography: December 2003–September 2006
Chairman EAE Accreditation Assessment Committee, European Association of Echocardiography Administration, European Heart House, 2035, Route des Colles, Les Templiers, BP 179 06903 Sophia Antipolis Cedex, France
* Corresponding author. k.fox{at}imperial.ac.uk
 |
Abstract |
|---|
 Top Abstract IntroductionDescription of the processes...The ExamLog book assessmentStatistics relating to the...The online accreditation projectConclusionsAppendix: Flow chart for...References |
|---|
Aims The European Association of Echocardiography (EAE) launched its Accreditation scheme in echocardiography in December 2003. Currently there are accreditations in Adult Transthoracic Echocardiography and Transesophageal Echocardiography and an Accreditation in Echocardiography for Congenital Disease will be launched in December 2006.
Methods and results Over the past 3years 350 applicants have undertaken the written exams and 157 applicants have completed the process and achieved accreditation.
This report summarises the accreditation process, the details of the numbers of applicants and their progress through the Accreditation scheme. This report also provides data about the performance of the written exam, its reliability and the effectiveness of the questions and data about the reproducibility of the log book marking system.
Conclusion Taken together this report provides evidence that the EAE Accreditation schemes have been effective in establishing a standard for echocardiographic practice and an accreditation that is effectively organised, reliable, robust and successful.
Keywords: European Association of Echocardiography; Accreditation scheme; Echocardiography
|
Introduction |
|---|
|
TopAbstractIntroductionDescription of the processes...The ExamLog book assessmentStatistics relating to the...The online accreditation projectConclusionsAppendix: Flow chart for...References |
|---|
An Accreditation scheme for individuals undertaking echocardiography across the member countries of the EAE was introduced in 2003. It provided a voluntary scheme for individuals to demonstrate and receive validation of their knowledge and skill in echocardiography. The scheme does not affect local statutory requirements for the practice of echocardiography and the reporting of echocardiograms.
It is a primary hope of the EAE that the Accreditation schemes will encourage learning, training and education and raise standards of echocardiography.
The first accreditation was in Adult Transthoracic Echocardiography (TTE) with the first exam held in December 2003 (EE7, Barcelona). This was followed by the establishment, jointly with the European Association of Cardiothoracic Anaesthesiologists, of an accreditation in Transesophageal Echocardiography (TEE) in June 2005 (EACTA meeting, Montpellier). An accreditation in echocardiography for Congenital Heart Disease (CHD), jointly established with the Association of European Paediatric Cardiologists (AEPC), is commencing with a first exam in December 2006 (EE10, Prague). The individual accreditations link closely with the parallel introduction of laboratory accreditation over the coming years.
This paper summarises the numbers and performance of the candidates, and of the exams and the log book marking scheme. A report on the first exam has been published previously.1
|
Description of the processes (see flow chart in Appendix) |
|---|
|
TopAbstractIntroductionDescription of the processes...The ExamLog book assessmentStatistics relating to the...The online accreditation projectConclusionsAppendix: Flow chart for...References |
|---|
All 3 accreditations follow a similar 3 stage process.
Candidates apply to undertake accreditation through the EAE administration at the European Heart House. First the applicants learn to echo with the support and help of a nominated supervisor who must either hold accreditation or be a nationally recognised expert in echocardiography. Second, applicants take a written exam, in English, testing theoretical knowledge and the ability to report on echocardiographic images. Third, candidates submit a log book, in their National language, consisting of a portfolio of reports of echocardiograms they have undertaken, a sample of which are graded by external examiners. Successful completion of all stages leads to the award of accreditation which is valid for 5years, after which candidates need to reaccredit by providing evidence of continued learning and practice.
|
The Exam |
|---|
|
TopAbstractIntroductionDescription of the processes...The ExamLog book assessmentStatistics relating to the...The online accreditation projectConclusionsAppendix: Flow chart for...References |
|---|
The exam must fulfil several requirements. It must be fair, consistent within and between cohorts, test appropriate knowledge and reliably distinguish candidates with an appropriate level of knowledge from those without. A complex system is in place to ensure this.
All questions are checked by at least 3 members of the (approximately 6 members) exam committee prior to use. A bank of questions has been built up. This is based on questions whose difficulty and discriminatory ability has been tested and new questions. Each exam has a balanced proportion of questions on various areas of echocardiography, with a range of difficulty and each exam has approximately 20% new questions. Exams use the multiple choice format in two sections, a theoretical section and a test of image reporting. Both sections must be passed. While a provisional pass mark is set the final pass mark may be adjusted after each exam by the exam committee to ensure a consistent standard against the absolute knowledge level required and between cohorts, taking into consideration the knowledge and experience of the cohort of candidates, the marks obtained and the overall difficulty of the questions. Borderline candidates are individually considered.
The computer analysis of the performance of the questions provides information on the difficulty and discriminatory ability of each question. This information is used to continuously improve the bank of questions and to determine which questions to reuse, modify or discard.
It has been a policy of the Accreditation Committee of the EAE to integrate candidates who have successfully achieved accreditation into the exam committees.
|
Log book assessment |
|---|
|
TopAbstractIntroductionDescription of the processes...The ExamLog book assessmentStatistics relating to the...The online accreditation projectConclusionsAppendix: Flow chart for...References |
|---|
Since echocardiography is a practical skill it is an integral part of the accreditation process that candidates demonstrate practical competency through submission of a portfolio of reports of echo studies performed and reported by the candidate. Equally important, a sample of studies must be reviewed and graded in detail by the applicant's supervisor, prior to log book submission and the details of this grading also submitted.
The grading scheme for log books was devised specifically to support the EAE Accreditation process. A random sample of reports is graded by two independent examiners chosen from a bank built up in coordination with national societies. Candidates must achieve an adequate aggregate score to achieve accreditation and a flow chart provides guidance on action taken in the event of a failure to achieve this score.
Examples are provided in an information pack to examiners (graders) and marks compared to measure reproducibility and reliability of marking.
|
Statistics relating to the accreditation process |
|---|
|
TopAbstractIntroductionDescription of the processes...The ExamLog book assessmentStatistics relating to the...The online accreditation projectConclusionsAppendix: Flow chart for...References |
|---|
The numbers of candidates and their flow through the process
Note: There is a 1year gap allowed taking the exam and submitting the log book explaining the lag between candidates taking the exam and completing the process (Table 1 and Fig. 1).
|
|
Performance of the exam questions
A number of measures of the performance of the exam questions are available. One can also look at the correlation between scores in the two parts of the exam (Table 2). Candidates are generally likely to have balanced knowledge and so a reasonable correlation suggests both sections are identifying the level of knowledge of candidates accurately.
|
With one exception the exams show good correlation between the marks in the two sections.
Computer analysis of the scoring sheets allows detailed assessment of the performance of the questions in the exam. Individual questions are assessed by measurement of their difficulty and their effectiveness in discriminating the stronger candidates from the less strong candidates (Figs. 2 and 3
). In general the number of ‘not effective’ questions (wrong answers, errors in the question) has been <4%. Previous analysis has shown >75% of questions were acceptable or good discriminators.1,2
|
|
The management of the log book section of the accreditation process
Overall there is very little backlog of log books for the accreditation processes (Table 3).
|
Reproducibility of log book grading
For 13 candidates we have analysed total marks (for all 10 cases) and individual marks for each case, comparing the marks of the two examiners for consistency. Each of 15 randomly selected cases from the 250 in the log book is marked on a scale of 1–5.
Fig. 4 shows the standard plot of individual scores versus average score.
|
For 9/13 (70%) candidates’ examiners were within 5 marks of each other and for 10/13 (77%) candidates were within 10 marks i.e. on average 1 mark difference for each case.
In these 13 candidates the observed agreement is 15% compared to 2% expected by chance giving a Kappa coefficient of 0.13 (p=0.0002). The observed agreement of 15% might seem low but the probability of two examiners reaching the same total score over 10 cases is quite small.
Looking at the scores for each case, there is a 50% agreement compared to a 32% agreement expected by chance. This gives a kappa coefficient of 0.27 (p<0.0005).
In terms of overall pass/fail there was agreement in 12 out of 13 cases. So apart from one case, which was resolved by a third examiner, the reliability of marking is reassuring.
|
The online accreditation project |
|---|
|
TopAbstractIntroductionDescription of the processes...The ExamLog book assessmentStatistics relating to the...The online accreditation projectConclusionsAppendix: Flow chart for...References |
|---|
It is an aim of the EAE to develop an on-line solution to registering for and administering the accreditation process.
The global project is composed of two different parts:
- – Registration of the candidates for the accreditation process and the written exam.
- – Practical assessment – log book.
- Submission of the log book.
- Grading of the cases.
- – Practical assessment – log book.
The first part of this process was launched in February 2006. Improvements are being developed to make the system operate very like European Society of Echocardiography congress applications. This new process will consequently provide the following members’ benefits:
- New service available from their ‘MyESC’ environment.
- Simplification of the registration process.
- Better management of the addresses (invoicing, registration, diploma).
- Many registrations (TTE, TEE, CHD) available from a single place.
- Automatic email confirmations.
The second part, to develop an online system for log book management, is in preparation and is hoped to be available in the course of 2007 taking note of the complexities of the log book marking scheme. The aim will be to run a system with similarities to abstract grading so that it will be familiar to markers (examiners).
The move to an online system is more professional, modern and innovative and is likely to be adopted by the other Associations running accreditation processes.
|
Conclusions |
|---|
|
TopAbstractIntroductionDescription of the processes...The ExamLog book assessmentStatistics relating to the...The online accreditation projectConclusionsAppendix: Flow chart for...References |
|---|
These data show that the EAE Accreditation processes are robust, fair, effective and well managed. None the less challenges remain. In comparison to the numbers performing echocardiography the application rates are low. But this is a long term project and the greatest advert for accreditation is successful candidates and therefore numbers undertaking the process are likely to grow with time. As numbers grow it will be increasingly important to recruit and maintain a bank of examiners willing to mark log books. Although the work is not too time consuming (<1h per log book) these are busy people and the development of an on-line assessment process will ease the burden.
Again it is likely that successful candidates will start to fill the ranks of examiners further strengthening the process.
It is equally important to continue to monitor all aspects of the accreditation process and particularly to use candidate's experiences for this purpose.
Currently the exam is in English and the possibility of running the exam in other languages needs to be considered, although in doing so the quality and fairness of the exam must not be compromised. The use of computer based testing rather than holding large exam sittings may also facilitate expansion of the accreditation scheme.
In conclusion the EAE set out to establish a system of accreditations in echocardiography that would help to promote and raise the standard of echocardiographic practice in Europe. The evidence so far is that a system has been developed fulfilling this aim.
|
Appendix: Flow chart for accreditation in adult echocardiography |
|---|
|
TopAbstractIntroductionDescription of the processes...The ExamLog book assessmentStatistics relating to the...The online accreditation projectConclusionsAppendix: Flow chart for...References |
|---|
|
References |
|---|
|
TopAbstractIntroductionDescription of the processes...The ExamLog book assessmentStatistics relating to the...The online accreditation projectConclusionsAppendix: Flow chart for...References |
|---|
- Fox K.F., Flachskampf F., Zamorano J.L., Badano L., Fraser A.G., Pinto F.J. Report on the first written exam held as part of the European Association of Echocardiography Accreditation Process in Adult Transthoracic Echocardiography. Eur J Echo (2004) 5:320–325.[CrossRef]
- University of London Management Information Services Division. An Introduction and Guide to the use of Multiple Choice Questions in University Examinations. (1986) University of London.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  B. A. Popescu, M. J. Andrade, L. P. Badano, K. F. Fox, F. A. Flachskampf, P. Lancellotti, A. Varga, R. Sicari, A. Evangelista, P. Nihoyannopoulos, et al. European Association of Echocardiography recommendations for training, competence, and quality improvement in echocardiography Eur J Echocardiogr, December 1, 2009; 10(8): 893 - 905. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
G. de la Morena, D. Saura, M. J. Oliva, F. Soria, J. Gonzalez, M. Garcia, V. Moreno, J. C. Bonaque, and M. Valdes Real-time three-dimensional transoesophageal echocardiography in the assessment of aortic valve stenosis Eur J Echocardiogr, October 4, 2009; (2009) jep154v1. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||