Copyright © 2005, The European Society of Cardiology
New generation 3-dimensional echocardiography for left ventricular volumetric and functional measurements: Comparison with cardiac magnetic resonance
Department of Cardiology, Academic Unit, The University of Hull, Castle Hill Hospital, Castle Road, Cottingham, Kingston-upon-Hull HU16 5JQ, UK
Received 15 March 2005; received in revised form 21 September 2005; accepted after revision 30 September 2005.
* Corresponding author. Tel.: +44 1482 624144; fax: +44 1482 624071.
n.p.nikitin{at}hull.ac.uk
| Abstract |
|---|
|
|
|---|
Aims Non-invasive assessment of left ventricular (LV) structure and function is important in the evaluation of cardiac patients. This study was designed to test the accuracy and reproducibility of new generation 3-dimensional echocardiography (3DE) in measuring volumetric and functional LV indices as compared with current "gold standard" of non-invasive cardiac imaging, cardiac magnetic resonance (CMR).
Methods and results Sixty-four subjects with good acoustic windows, including 40 cardiac patients with LV ejection fraction (EF)<45%, 14 patients with EF>45% and 10 normal volunteers underwent 3DE using a commercially available Philips Sonos 7500 scanner equipped with a matrix phase-array x4 xMATRIX transducer, and CMR on a 1.5T Signa CV/i scanner (GE Medical Systems). Volumetric assessment was performed with analytical 4D-LV-Analysis software (TomTec) for 3DE and MRI-Mass software (Medis) for CMR.
We found no significant differences in LV end-diastolic volume (EDV), end-systolic volume (ESV) and EF with excellent correlations between the indices measured using 3DE and CMR (r=0.97, r=0.98, and r=0.94, respectively). Bland–Altman analysis showed bias of 7ml for EDV, 3ml for ESV and –1% for EF with 3DE with corresponding limits of agreement (2SD) of 28ml, 22ml and 10%, respectively. Intraobserver and interobserver variabilities were for EDV: 3% and 4% (3DE) vs 2% and 2% (CMR), for ESV: 3% and 6% (3DE) vs 2% and 3% (CMR), and for EF: 4% and 4% (3DE) vs 2% and 4% (CMR), respectively.
Conclusion New generation 3DE provides accurate and reproducible quantification of LV volumetric and functional data in subjects with good acoustic windows as compared with CMR.
Keywords: Imaging; Echocardiography; Left ventricular function; Magnetic resonance imaging
| Introduction |
|---|
|
|
|---|
Accurate and reproducible measurement of indices of left ventricular (LV) morphology and function is an important part of evaluation of cardiac patients. These measurements help to diagnose disease and manage patients and have important prognostic significance.1
Two-dimensional (2D) echocardiography is currently the most widely used non-invasive imaging tool for the assessment of LV volumes and ejection fraction in patients with cardiac disease. Biplane method of disks2 is considered the most accurate 2D echocardiographic approach but it still shows limited accuracy due to the effect of foreshortening and dependence on geometric assumptions, particularly in patients with deformed ventricles.3
Three-dimensional echocardiography (3DE), based on the assessment of a larger number of tomographic views, has the potential to overcome these limitations. Initially, 3-dimensional (3D) methods were based on the use of off-line reconstruction of 2D images obtained in parallel or as a result of fan-like or rotational motion. These methods suffered from numerous limitations, such as unreliable ECG or respiratory gating, long image acquisition times and limited field views produced by bulky transducers that were cumbersome to use. Off-line reconstruction of 3D data sets was also complex and not practical.
The advent of matrix phase-array transducers has allowed real-time 3D volume rendering without the need for ECG or respiratory gating, but until recently has been limited by the number of active elements. Due to recent technological breakthroughs, currently available new generation matrix array transducers have more than 3000 active elements and are capable of producing high-quality 3D images in addition to conventional 2D imaging.
This study was designed to compare the accuracy and reproducibility of volumetric and ejection fraction measurements performed with new generation 3DE as compared to cardiac magnetic resonance (CMR), the current "gold standard" in non-invasive cardiac imaging.
| Methods |
|---|
|
|
|---|
Study population
Ninety-four subjects (including 84 patients with cardiac disease and 10 normal volunteers) were screened. Inclusion criteria were subjects in sinus rhythm with good acoustic windows and no contraindications to CMR (claustrophobia, metallic implants, pacemakers or defibrillators). Sixteen patients were excluded due to poor acoustic windows, six patients were excluded due to atrial fibrillation and eight patients were excluded due to contraindications to CMR. The remaining 64 subjects (including 10 normal volunteers) aged 65±12 years (range 34–85 years, 13 women) comprised the study group. Informed written consent was obtained from all study subjects and the study was approved by the local research ethics committee.
Echocardiography
The study subjects underwent 3DE using a commercially available Philips Sonos 7500 scanner equipped with a matrix array 3D transducer (x4 xMATRIX transducer based on xSTREAM 3D architecture). Full volume (wide angle) ECG-gated 3D data sets were acquired from apical positions. To obtain these data sets, four sectors were scanned during brief breathholds (typically 6–8s) and were automatically integrated into a pyramidal 90° by 90° data sets covering the entire left ventricle (Fig. 1). All the scans were performed by two experienced operators (NPN and CC).
|
The full volume data sets were saved into optical disks and transferred to a workstation running commercially available 4D-LV-Analysis software (TomTec Gmbh, Unterschleissheim, Germany). Off-line data analysis was performed by an experienced operator (NPN) who had no knowledge of CMR-derived data.
In this study volumetric analysis was performed with the use of eight orthogonal long-axis images. LV end-diastolic and end-systolic images were selected by identifying the volumetric frames showing the largest and smallest LV cavity sizes, respectively. In each image the position of the apex and the mitral annulus was defined manually. These points were then connected semi-automatically by curved lines containing numerous setting points and aligning with endocardial surface of the ventricle. The papillary muscles were excluded from the blood pool. LV end-diastolic volume (EDV) and end-systolic volume (ESV) were calculated automatically by the software by summating volumes of a number of smaller volumes defined by adjacent points on the LV contour and the central axis of the ventricle. LV ejection fraction (EF) was calculated using the standard formula.
Cardiac magnetic resonance
Patients underwent CMR on a 1.5T scanner (Signa CV/i, GE Medical Systems) using ECG-triggered breathhold gradient-echo in steady-state acquisition (FIESTA) imaging. After initial localizing scans, cine LV horizontal long-axis, vertical long-axis and contiguous short-axis images covering the left ventricle from apex to base (slice thickness 10mm) were obtained. Each image contained 20 ECG-triggered cine loops.
CMR data were analysed with the use of MRI-MASS software system (MEDIS, Leiden, NL) by an experienced investigator (EIL) who was unaware of the measurements made using 3DE. In contiguous LV short-axis cine data sets, the endocardial borders were traced manually at end diastole and end systole with trabeculations and papillary muscles excluded from the blood volumes (Fig. 1). The software performed summation of the endocardial volumes on a slice-by-slice basis and provided total LV EDV and ESV measurements not based on geometric assumptions.
Statistical analysis
Study results are reported as mean±SD. Echocardiographic and CMR-derived data were compared by using unpaired t tests. Relationship between measurements was assessed by Pearson correlation analysis. The agreement between the two techniques was assessed according to the method described by Altman and Bland.4 A p value less than 0.05 was considered to be significant.
Differences in the measurements of LV EDV, LV ESV, and LV EF made by the same operator (intraobserver variability) and by two independent investigators blinded to each other's results (interobserver variability) were computed for each method (3DE and CMR) in a sample of 20 random subjects and presented as mean±SD. Ninety-five percent confidence intervals of differences were calculated and expressed as absolute values and percentages of the average values of all measurements.
| Results |
|---|
|
|
|---|
Image acquisition with the use of both 3DE and CMR was successful in all 64 subjects who were recruited into the study. The study population included 40 patients with LV systolic dysfunction (ejection fraction<45%) due to ischaemic heart disease or dilated cardiomyopathy, 14 patients with cardiac disease (hypertensive heart disease or diastolic heart failure) and preserved LV systolic function (ejection fraction>45%) and 10 volunteers with normal LV systolic function. Table 1 shows clinical characteristics of the study group.
|
The study subjects represented a broad spectrum of LV dimensions and global function. LV EDV varied from 66 to 371ml (on 3DE) and from 61 to 367ml (on CMR). LV ESV ranged from 15 to 305ml (on 3DE) and from 8 to 310ml (on CMR), and LV EF varied from 12 to 82% (on 3DE) and from 12 to 88% (on CMR). Mean measurements of LV EF, LV EDV, LV ESV and corresponding volumetric indices corrected for body surface area obtained with the use of both techniques are shown in Table 2. There were no significant differences in any of these indices.
|
There was a strong linear correlation between measurements performed with the use of 3DE and CMR (r=0.98 for both LV EDV and LV ESV and r=0.94 for LV EF). Corresponding scatterplots are shown in Fig. 2.
|
Bland–Altman analysis of LV volumetric data and EF measurements showed good agreement between the two methods (Fig. 3). Limits of agreement expressed as mean difference between 3DE and CMR measurements (2SD) were 7±28ml for LV EDV, 3±22ml for LV ESV and –1±10 absolute units for LV EF.
|
The study of inter- and intraobserver variabilities of LV EDV, LV ESV and LV EF measurements performed on 20 random scans demonstrated good reproducibility of the measurements with the use of both imaging techniques (Table 3).
|
| Discussion |
|---|
|
|
|---|
Accurate non-invasive assessment of LV morphology and function is important for the assessment and management of patients with potential cardiac disease. LV volumes, particularly ESV, have been shown to be strong prognostic indicators after myocardial infarction.5,6 LV EF which is derived from volumes is prognostically important in patients with chronic heart failure.7
Echocardiography is a well established and widely available method of non-invasive cardiac imaging. However, measurements of LV volumes and EF by both M-mode and 2D echocardiography have been marred by insufficient reproducibility and accuracy compared to reference methods such as CMR or radionuclide ventriculography.3
CMR is an alternative non-invasive technique that is very well suited for the assessment of LV geometry and function. Its main advantages are superior quality of cardiac images not limited by the acoustic windows or operator skills. The technique is not restricted by imaging planes and allows acquiring 3D data sets. As a result, measurements of LV volumes, EF and myocardial mass are not based on geometric assumptions and are accurate and reproducible.8 The method is a strong contender for being the current "gold standard" in the non-invasive assessment of cardiac structure and function.9 However, CMR is not free of limitations. It is a costly technique which is still not widely available. A proportion of patients cannot undertake a CMR investigation due to claustrophobia, metal in the eye or in the brain, pacemakers and other implants.
3DE is a logical step in the development of echocardiography which has been made possible due to recent technological advances. It has the potential to overcome some important limitations of conventional echocardiography while still retaining its principal advantages of moderate cost and patient friendliness. An important feature of 3DE is its independence of geometric assumptions. It has been shown to improve accuracy of measurements of LV volumes and ejection fraction.10 Unlike 2D echocardiography, which is very dependent on acquiring quality standard views, 3DE is less dependent on operator skills and can help to compensate for sonographer inexperience.10
The method used in this study was a new generation 3DE based on matrix phase-array transducer technology. Many previous 3DE solutions were limited by very labour- and time-intensive post-processing. The last generation systems coupled with advanced analytical software deliver faster image acquisition and less laborious and lengthy off-line analysis which make the technique practical for routine clinical use. Currently available new generation systems are capable of providing real-time 3D imaging. However, ECG gating is still required for wide angle or "full volume" acquisition which is used for volumetric analysis.
In this study we performed volumetric analysis based on eight long-axis cross-sections as using a larger number of views make analysis more laborious and time-consuming but does not result in a significant improvement in the accuracy or reproducibility of volumetric measurements.11,12
We found good agreement between new generation 3DE and CMR measurements in patients with good acoustic windows. Compared with CMR, 3DE slightly overestimated LV volumes and underestimated ejection fraction, probably mainly due to differences in the planimetric approaches to LV contour tracing (tracing contours in long-axis views using 3DE vs tracing contours in short-axis views using CMR).
Our data obtained in a substantial patient population adds to results of previous smaller studies that showed good agreement between CMR and 3DE based on technological solutions of various generations.13–17 Even earlier, good correlation was reported in studies in which first generation 3DE was compared with radionuclide ventriculography.18,19
Our results also showed a much better agreement between CMR-based and 3DE-based measurements than previously reported agreement between CMR and M-mode or 2D echocardiography. For example, limits of agreement calculated as mean difference±1SD for LV EF between CMR and 3DE found in this study was –1±5% compared to –2±10% agreement between CMR and 2D echocardiography reported in CHRISTMAS study.3
Good agreement between 3DE and CMR found in this study despite methodological differences (see Limitations) suggests that volumetric assessment free of geometric assumptions (which is a feature of both methods) is probably the most essential element of accurate image data analysis. In contrast, it has been previously shown that data analysis based on the use of biplane data sets enhanced by geometric models is less accurate especially in distorted ventricles.15
This study also demonstrated excellent intra- and interobserver reproducibilities of volumetric and EF measurements performed with new generation 3DE which was better compared to some previous reports based on experimental 20,21 and clinical studies.16,17
Good agreement in accuracy and reproducibility between the methods makes new generation 3DE a potentially more practical alternative to CMR not only in clinical practice, but also in clinical trials. Further work is needed to show whether it will allow the demonstration of small changes in volumetric indices or LV EF and therefore limit the number of study participants required to deliver statistical power.
| Limitations |
|---|
|
|
|---|
Any echo-based technique is dependent on the acoustic window. Measurements performed in patients with suboptimal quality of echocardiographic images are less reliable. We had to exclude 22 patients who were found to have insufficient image quality to ensure reliable volumetric analysis. The decision to exclude unsuitable patients was made by the experienced operators who performed 3DE investigations for this study. The prevalence of patients with suboptimal quality of echocardiographic images in our population which included a small proportion of normal volunteers was 23% which generally agrees with 31% reported in CHRISTMAS study3 which dealt with patients with LV systolic dysfunction. Use of contrast agents may be necessary to ensure accurate volume and EF measurements using 3DE in patients with suboptimal image quality.
The comparison performed in this study was limited not only to subjects with good acoustic windows (3DE) but also to subjects with no contraindications to CMR.
Claustrophobia, metal in the eye and in the brain, pacemakers and defibrillators are the most common contraindications to CMR. A new design of more open scanners and introduction of CMR-compatible implants and devices should help to solve the problem in the future.
We also excluded patients with atrial fibrillation which may lead to degradation of the quality of ECG-gated imaging (both 3DE and CMR). ECG gating still remains a necessity in many modern imaging techniques and any factors affecting it (such as arrhythmias or problems with breathhold) can make quantitative analysis less accurate and less reproducible.
Analysis of CMR data performed in this study was based on manual border tracing in contiguous short-axis views, using the summation of discs method. Analysis of 3DE data was performed using semi-automated endocardial border detection in multiple orthogonal long-axis views. Each of these methods has advantages and limitations.
The method based on the use of multiple short-axis data sets suffers from certain difficulties in defining the borders of LV cavity in most apical and basal views. In the most apical slice, endocardial border tracing may be difficult. However, the volumes derived from this slice contribute least to the total LV volumes. In the basal view it is important to correct for mitral annular motion.
The method based on the use of multiple long-axis view data is optimal for following mitral annular displacement but may be limited by difficulties in delineating papillary muscles and ventricular trabeculations and accurately defining the apex in heavily trabeculated ventricles.
| Conclusions |
|---|
|
|
|---|
New generation 3DE based on matrix phase-array transducer technology provides accurate and reproducible quantification of LV volumetric and functional data in patients with good acoustic windows as compared with current "gold standard" of non-invasive cardiac imaging, CMR. The method can be recommended for routine assessment of patients with cardiac disease.
Further studies on other groups of cardiac patients and also looking at inter-centre reproducibility of new generation 3DE are required to confirm the findings of this study. 3DE is a fast evolving technique and there are little doubts that future technological developments will further improve the accuracy and reproducibility of the technique.
| Acknowledgements |
|---|
We thank Philips Medical Systems for the loan of the echocardiographic equipment and Tomtec Gmbh for their support with analysis software. NPN and EIL were partly supported by grants from the Wellcome Trust and British Heart Foundation.
| References |
|---|
|
|
|---|
- Cowburn P.J., Cleland J.G., Coats A.J., Komajda M. Risk stratification in chronic heart failure. Eur Heart J (1998) 19:696–710.
[Free Full Text] - Schiller N.B., Acquatella H., Ports T.A. Left ventricular volume from paired biplane two-dimensional echocardiography. Circulation (1979) 60:547–555.
[Abstract/Free Full Text] - Bellenger N.G., Burgess M., Ray S.G., Lahiri A., Coats A.J., Cleland J.G., et al. Comparison of left ventricular ejection fraction and volumes in heart failure by two-dimensional echocardiography, radionuclide ventriculography and cardiovascular magnetic resonance: Are they interchangeable? Eur Heart J (2000) 21:1387–1396.
[Abstract/Free Full Text] - Bland J.M., Altman D.G. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet (1986) 1:307–310.[CrossRef][Web of Science][Medline]
- Migrino R.Q., Young J.B., Ellis S.G., White H.D., Lundergan C.F., Miller D.P., et al. End-systolic volume index at 90 to 180 minutes into reperfusion therapy for acute myocardial infarction is a strong predictor of early and late mortality. Circulation (1997) 96:116–121.
[Abstract/Free Full Text] - White H.D., Norris R.M., Brown M.A., Brandt P.W., Whitlock R.M., Wild C.J. Left ventricular end-systolic volume as the major determinant of survival after recovery from myocardial infarction. Circulation (1987) 76:44–51.
[Abstract/Free Full Text] - Cohn J.N., Johnson G.R., Shabetai R., Loeb H., Tristani F., Rector T., et al. Ejection fraction, peak exercise oxygen consumption, cardiothoracic ratio, ventricular arrhythmias and plasma norepinephrine as determinants of prognosis in heart failure. Circulation (1993) 87(Suppl. VI):VI5–VI16.[Medline]
- Bellenger N.G., Davies L.C., Francis J.M., Coats A.J., Pennell D.J. Reduction in sample size for studies of remodelling in heart failure by the use of cardiovascular magnetic resonance. J Cardiovasc Magn Reson (2000) 2:271–278.[Web of Science][Medline]
- Buser P.T. MR studies of left ventricular remodeling. MAGMA (1998) 6:150–151.[CrossRef][Medline]
- Chuang M.L., Parker R.A., Riley M.F., Reilly M.A., Johnson R.B., Korley V.J., et al. Three-dimensional echocardiography improves accuracy and compensates for sonographer inexperience in assessment of left ventricular ejection fraction. J Am Soc Echocardiogr (1999) 12:290–299.[CrossRef][Web of Science][Medline]
- Kuhl H.P., Franke A., Merx M., Hoffmann R., Puschmann D., Hanrath P. Rapid quantification of left ventricular function and mass using transoesophageal three-dimensional echocardiography: validation of a method that uses long-axis cutplanes. Eur J Echocardiography (2000) 1:213–221.[CrossRef]
- Nosir Y.F., Stoker J., Kasprzak J.D., Lequin M.H., Dall'Agata A., Ten Cate F.J., et al. Paraplane analysis from precordial three-dimensional echocardiographic data sets for rapid and accurate quantification of left ventricular volume and function: a comparison with magnetic resonance imaging. Am Heart J (1999) 137:134–143.[CrossRef][Web of Science][Medline]
- Shiota T., McCarthy P.M., White R.D., Qin J.X., Greenberg N.L., Flamm S.D., et al. Initial clinical experience of real-time three-dimensional echocardiography in patients with ischemic and idiopathic dilated cardiomyopathy. Am J Cardiol (1999) 84:1068–1073.[CrossRef][Web of Science][Medline]
- Hibberd M.G., Chuang M.L., Beaudin R.A., Riley M.F., Mooney M.G., Fearnside J.T., et al. Accuracy of three-dimensional echocardiography with unrestricted selection of imaging planes for measurement of left ventricular volumes and ejection fraction. Am Heart J (2000) 140:469–475.[CrossRef][Web of Science][Medline]
- Chuang M.L., Hibberd M.G., Salton C.J., Beaudin R.A., Riley M.F., Parker R.A., et al. Importance of imaging method over imaging modality in noninvasive determination of left ventricular volumes and ejection fraction: assessment by two- and three-dimensional echocardiography and magnetic resonance imaging. J Am Coll Cardiol (2000) 35:477–484.
[Abstract/Free Full Text] - Schmidt M.A., Ohazama C.J., Agyeman K.O., Freidlin R.Z., Jones M., Laurienzo J.M., et al. Real-time three-dimensional echocardiography for measurement of left ventricular volumes. Am J Cardiol (1999) 84:1434–1439.[CrossRef][Web of Science][Medline]
- Mannaerts H.F.J., van der Heide J.A., Kamp O., Papavassiliu T., Marcus J.T., Beek A., et al. Quantification of left ventricular volumes and ejection fraction using freehand transthoracic three-dimensional echocardiography: comparison with magnetic resonance imaging. J Am Soc Echocardiogr (2003) 16:101–109.[CrossRef][Web of Science][Medline]
- Nosir Y.F.M., Fioretti P., Vletter W.B., Boersma E., Salustri A., Postma J.T., et al. Accurate measurements of left ventricular ejection fraction: radionuclide angiography versus three-dimensional echocardiography. Circulation (1996) 94:460–466.
[Abstract/Free Full Text] - Gopal A.S., Sapin P.M., Shen Z., Keller A.M., Schnellbaecher M.J., Leibowitz D.W., et al. Assessment of cardiac function by three-dimensional echocardiography compared with conventional noninvasive methods. Circulation (1995) 92:842–852.
[Abstract/Free Full Text] - Schmidt M.A., Freidlin R.Z., Ohazama C.J., Jones M., Laurienzo J.M., Brenneman C.L., et al. Anatomic validation of a novel method for left ventricular volume and mass measurements with use of real-time 3-dimensional echocardiography. J Am Soc Echocardiogr (2001) 14:1–10.[CrossRef][Web of Science][Medline]
- Siu S.C., Levine R.A., Rivera J.M., Xie S.W., Lethor J.P., Handschumacher M.D., et al. Three-dimensional echocardiography improves noninvasive assessment of left ventricular volume and performance. Am Heart J (1995) 130:812–822.[CrossRef][Web of Science][Medline]
This article has been cited by other articles:
![]() |
K.Y. E. Leung and J. G. Bosch Automated border detection in three-dimensional echocardiography: principles and promises Eur J Echocardiogr, February 6, 2010; (2010) jeq005v1. [Abstract] [Full Text] [PDF] |
||||
![]() |
H. C. Kim, D. M. Yang, W. Jin, and S. H. Lee Relation Between Total Renal Volume and Renal Function: Usefulness of 3D Sonographic Measurements With a Matrix Array Transducer Am. J. Roentgenol., February 1, 2010; 194(2): W186 - W192. [Abstract] [Full Text] [PDF] |
||||
![]() |
A Bhan, S Kapetanakis, and M J Monaghan Three-dimensional echocardiography Heart, January 15, 2010; 96(2): 153 - 163. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Muraru, L. P. Badano, G. Piccoli, P. Gianfagna, L. Del Mestre, D. Ermacora, and A. Proclemer Validation of a novel automated border-detection algorithm for rapid and accurate quantitation of left ventricular volumes based on three-dimensional echocardiography Eur J Echocardiogr, January 6, 2010; (2010) jep217v2. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. Ren, J.-W. Tian, X.-P. Leng, H.-M. Wang, Y. Wang, and Z.-Z. Wang Assessment of Global and Regional Left Ventricular Function After Surgical Revascularization in Patients With Coronary Artery Disease by Real-time Triplane Echocardiography J. Ultrasound Med., September 1, 2009; 28(9): 1175 - 1184. [Abstract] [Full Text] [PDF] |
||||
![]() |
V. Mor-Avi, L. Sugeng, and R. M. Lang Real-Time 3-Dimensional Echocardiography: An Integral Component of the Routine Echocardiographic Examination in Adult Patients? Circulation, January 20, 2009; 119(2): 314 - 329. [Full Text] [PDF] |
||||
![]() |
H. C. Kim, D. M. Yang, S. H. Lee, and Y. D. Cho Usefulness of Renal Volume Measurements Obtained by a 3-Dimensional Sonographic Transducer With Matrix Electronic Arrays J. Ultrasound Med., December 1, 2008; 27(12): 1673 - 1681. [Abstract] [Full Text] [PDF] |
||||
![]() |
S J Copley and T P Mathew Advances in echocardiography Imaging, September 1, 2006; 18(3): 160 - 165. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||








