The emerging role of radiation therapists in the contouring of organs at risk in radiotherapy: analysis of inter-observer variability with radiation oncologists for the chest and upper abdomen
Simona Arculeo1,2, Eleonora Miglietta1, Fabrizio Nava1, Anna Morra1, Maria Cristina Leonardi1, Stefania Comi3, Delia Ciardo1, Massimo Sarra Fiore1, Marianna Alessandra Gerardi1, Matteo Pepa1, Simone Giovanni Gugliandolo1, Lorenzo Livi4, Roberto Orecchia5, Barbara Alicja Jereczek-Fossa1,2 and Samantha Dicuonzo1
1Division of Radiation Oncology, European Institute of Oncology IRCCS (IEO), Via Ripamonti 435, 20141 Milan, Italy
2Department of Oncology and Hemato-oncology, University of Milan, Via Festa del Perdono 7, 20122, Milan, Italy
3Unit of Medical Physics, European Institute of Oncology IRCCS (IEO), 20141 Milan, Italy
4Radiation Oncology Unit, Oncology Department, Azienda Ospedaliero-Universitaria Careggi, University of Florence, Largo Piero Palagi 1, 50139 Florence, Italy
5Scientific Directorate, European Institute of Oncology IRCCS (IEO), 20141 Milan, Italy
Abstract
Aims: To compare the contouring of organs at risk (OAR) between a clinical specialist radiation therapist (CSRT) and radiation oncologists (ROs) with different levels of expertise (senior–SRO, junior–JRO, fellow–FRO).
Methods: On ten planning computed tomography (CT) image sets of patients undergoing breast radiotherapy (RT), the observers independently contoured the contralateral breast, heart, left anterior descending artery (LAD), oesophagus, kidney, liver, spinal cord, stomach and trachea. The CSRT was instructed by the JRO e SRO. The inter-observer variability of contoured volumes was measured using the Dice similarity coefficient (DSC) (threshold of ≥ 0.7 for good concordance) and the centre of mass distance (CMD). The analysis of variance (ANOVA) was performed and a p-value < 0.01 was considered statistically significant.
Results: Good overlaps (DSC > 0.7) were obtained for all OARs, except for LAD (DSC = 0.34 ± 0.17, mean ± standard deviation) and oesophagus (DSC = 0.66 ± 0.06, mean ± SD). The mean CMD < 1 cm was achieved for all the OARs, but spinal cord (CMD = 1.22 cm). By pairing the observers, mean DSC > 0.7 and mean CMD < 1 cm were achieved in all cases. The best overlaps were seen for the pairs JRO-CSRT(DSC = 0.82; CMD = 0.49 cm) and SRO-JRO (DSC = 0.80; CMD = 0.51 cm).
Conclusions: Overall, good concordance was found for all the observers. Despite the short training in contouring, CSRT obtained good concordance with his tutor (JRO). Great variability was seen in contouring the LAD, due to its difficult visualization and identification of CT scans without contrast.
Keywords: contouring, inter-observer variability, organs at risk (OAR), radiation therapist
Correspondence to: Maria Cristina Leonardi
Email: cristina.leonardi@ieo.it
Published: 06/01/2020
Received: 12/07/2019
Publication costs for this article were supported by ecancer (UK Charity number 1176307).
Copyright: © the authors; licensee ecancermedicalscience. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Introduction
Conformal radiotherapy (RT), based on three-dimensional technique and, more recently, on intensity-modulated RT (IMRT), is widely used as a modern approach to postoperative treatments for breast cancer. Conformal RT allows for dosimetric optimization and increases organs at risk (OAR) sparing, which represents major goals of treatment planning [1], and more tailored dose distributions to the target volume are obtained with IMRT [2]. The accuracy of dose delivery is provided by means of image-guided RT, which ensures the reproducibility of patient positioning at each treatment fraction.
Conformal RT is based on complex procedures which start from the accurate delineation of the clinical target volume (CTV) and OARs [3, 4]. The contouring task is a fundamental pre-requisite for high-quality plans and a source of potential inaccuracy. Great efforts are spent to lower the inter-observer variability by using atlases, auto-contouring software, and attending live contouring workshops [5].
Furthermore, in the RT planning process, volume delineation is the most time-consuming task for clinicians as it relies more and more on registration and interpretation of multimodality imaging [6]. The inter-observer variability on contouring might have implications in terms of dosimetric distribution and clinical outcomes and has been the subject of several studies [7, 8].
In clinical practice, the contouring of target volumes and OARs is performed by radiation oncologists (ROs) [9]. In the literature, most of the contouring studies are focused on assessing the variability among physicians (ROs and radiologists) [10–14]. Since in recent years, the role of clinical specialist radiation therapists (CSRTs) has gained ground in RT planning, treatment delivery and patient care [15], appropriately trained CSRTs might substitute ROs in contouring OARs [10, 16, 17], with the aim of improving the workflow.
The purpose of the current study is to quantify the inter-observer variability in OARs delineation in the chest and upper abdomen between ROs, with different levels of expertise, and a trained CSRT.
Materials and methods
This study was carried out within the research project entitled ‘Adjuvant radiation treatments with IMRT and/or hypofractionated schedules for breast cancer’, notified to the European Institute of Oncology Ethics Committee (May 26, 2016, Milan, Italy). All patients gave written informed consent for the treatment and anonymous use of their data for educational and research purposes. The non-contrast 2.5-mm-thick computed tomography (CT) of the chest and upper abdomen of ten patients receiving RT to the breast for breast cancer (BC) using TomoTherapy® Hi-Art System (Tomotherapy Inc., Madison, WI), acquired in the supine position, were used for OARs delineation. OARs were contoured using the treatment planning system RayStation® (version 6.1.1.2, RaySearch Laboratories AB, Stockholm, Sweden).
The observers were given written documentation regarding delineation, according to clinical daily practice. In addition, the CSRT was presented the delineation software and CT scans. The group of observers included a senior RO (SRO) with more than 10-year experience, a junior RO (JRO) with 2-year experience, a second-year training fellow RO (FRO) with a 3-month experience in the thoracic district contouring, and a CSRT with 500-hour working experience in RT department. The CSRT received a 100-hour training course in contouring by the JRO under the supervision of the SRO. In total, the CSRT contoured approximately 110 OARs from ten patients during the training period.
The contoured OARs were contralateral breast, heart, oesophagus, ipsilateral kidney, left anterior descending artery (LAD), liver, spinal cord, stomach, and trachea. Lung was not considered because the contouring was automatically generated. The delineation of the heart (starting inferior to the left pulmonary artery to the apex, including the fatty tissue within the pericardium and excluding the superior vena cava), LAD (from the left main coronary artery, running between the left atrium and ventricle and extending to the apex), oesophagus (from the cricoid cartilage to the gastroesophageal junction), spinal cord (including all the spaces within the bony canal from the cricoid cartilage to the bottom of L2), and contralateral breast (breast gland, excluding pectoralis and intercostal muscles and rib plane) was based on the Atlas for breast volumes contouring developed by the Italian Association of Radiation Oncology (AIRO) and available online [18]. The kidney and liver were contoured with the exclusion of the hilum, and the stomach was contoured from the cardiac region to the pyloric antrum and trachea from the bottom of the larynx to the carina. The observers independently contoured the OARs.
The variations between observers were quantified by means of the Dice similarity coefficient (DSC) and the centre of mass distance (CMD).
The DSC was used to evaluate the overlap between the delineated contours. DSC represents the intersection between two observers’ contours, for the same OAR, normalized to their intersection. Since the DSC varies from 0 (no overlap) to 1 (perfect overlap), a threshold value of 0.7 was considered as the minimum level of good concordance [19]. The CMD represents the distance between the centre of mass of two delineated contours for the same OAR and was measured in centimetres. DSC and CMD were calculated for each OAR by RayStation®.
The distributions of the evaluated parameters were assessed using the analysis of variance (ANOVA); the significance p-value threshold was set to 0.01. The analysis was performed using Matlab (v. R2014a, The Mathworks Inc.).
For the purpose of the study, we compared the inter-observer variability of contouring of each OAR and, in every couple of observers, we evaluated the consistency of contouring with each other considering all the OARs as a whole.
The observers were paired into sets of two: JRO versus FRO, FRO versus CSRT, SRO versus CSRT, SRO versus JRO, SRO versus FRO and JRO versus CSRT.
Results
All CT scans were acquired in the period between April and June 2017. In total, 360 contours were generated by the four observers.
Inter-observer variability
Table 1 shows the inter-observer variability of the OAR delineation. The mean DSC between all the couples of observers was > 0.7 and the mean CMD was < 1 cm. The best concordance was found between CSRT and JRO (DSC = 0.83 cm and CMD = 0.49 cm), who were taken as the reference pair. Comparing each other sets of observers with the reference pair, no statistical difference was found with the pair SRO-JRO with regard to both parameters (DSC = 0.80 cm and CMD = 0.51 cm) (Figure 1a) and with the pair SRO-CSRT with respect to CMD alone (Figure 1b). The FRO reported the highest variability when compared to the other observers for both parameters.
OAR contour analysis
The largest variability in contouring affected the LAD, which presented a mean DSC of 0.34 ± 0.17, followed by the oesophagus with a mean DSC of 0.66 ± 0.06 (Table 2). Regarding the CMD, a poor value (>1 cm) was reported only for the spinal cord (CMD = 1.22 ± 1.00 cm). All other OARs showed a mean DSC > 0.7. In particular, stomach, spinal cord, and trachea showed a mean DSC of about 0.80, whereas heart, liver, breast, and kidney had a mean DSC in the range of 0.88–0.93 (Table 2). The best overlapping was obtained for the liver which scored a DSC of 0.93 and CMD of 0.19 cm. Considering these values as a reference, no statistically different values of DSC were found for heart and contralateral breast, while the remaining six OARs had lower values (Figure 2a). Regarding CMD, only the heart and the ipsilateral kidney were not statistically different from the liver (Figure 2b).
Table 1. Mean and SD of DSC and CDM for each couple of observers.
Figure 1. (A) DCA and (B) CMD of each couple of observers. The black dashed line indicates the couple that is significantly different from the red-underlined reference couple.
Table 2. Mean and SD of DSC and CMD for each OAR obtain from all observers for all patients.
Figure 2. (A) DCA and (B) CMD for each analysed organ at risk (OAR). The black dashed line indicates the OARs that are significantly different from the red-underlined reference (liver).
Discussion
Accurate delineation of tumour volumes and surrounding normal structures represents critical aspects of conformal RT. Several atlases, based on expert consensus, have been published mainly to help the delineation of target volumes [18–22], while potentially critical OARs, such as heart, LAD and brachial plexus have received less attention [23–25], even though their impact is on toxicity. In addition, the contouring process is burdened by time-consuming procedures which may slow the entire workflow. Apart from the introduction of atlas-based auto-contouring programs in the clinical practice, which proved to be beneficial in time reduction compared to the manual contouring [26], the possibility of expanding the role of CSRTs in the pre-planning phase represents a major asset. Several studies have already confirmed the great potential of CSRTs in normal tissue contouring in the head and neck and the pelvic districts [10–13, 27].
This study reported on the variability in delineating chest and upper abdomen OARs by observers from a different background and showed that small variations can be achieved between CSRT and ROs, for all the structures, through adequate contouring training. A 100-hour training course was therefore sufficient to provide the basic contouring skills, but we are aware that continuing contouring workshops are mandatory. Although on the whole, the inter-observer variability was low and all the results were above the thresholds for good concordance, the FRO showed the smallest concordance when compared with the other observers probably due to the short experience in thoracic structures. This observation can strengthen the concept that providing detailed indications might not be sufficient to ensure a consistent way of contouring, particularly in the case of unfavorable patients’ anatomy [26] and clinical practice is crucial to improve skills. In addition, Li et al [28] reported that the inter-observer variability might be related to differences in opinion regarding volume boundaries. Efforts to improve contouring consistency and reduce inter-observer variability were successful to fill the gap in education, knowledge and expertise among the observers. In fact, the study showed that a well-trained CSRT acquires site-specific skills of contouring which bring him on the same level of capabilities as those displayed by senior and junior ROs.
Comparing the different OAR delineation, six out of nine OARs showed both parameters above the acceptable threshold and all of them reported at list one parameter above the threshold. The possibility of training CSRTs for helping ROs in the contouring of OARs is appealing, given the increased contouring workload for ROs in the light of the growing complexity of radiation treatment [29]. The evaluation of OAR delineation consistency between CSRT and ROs has been previously studied. Fitzpatrick et al [17] evaluated 39 sample prostate plans where the CSRT delineated OARs and concluded that, after proper training, contouring of OARs was successfully carried out by CSRT.
On the other hand, we believe that the CTV delineation remains the responsibility of the ROs, even if some studies addressed the role of CSRT in this field [9, 12, 13]. Batumalai et al [9] used the concordance index as an assessment tool to quantify inter-observer variability between four ROs and four CSRTs regarding breast CTV delineation. The analysis revealed high consistency between the different professional figures: the mean concordance index was 0.81 and 0.84 for the RO and CSRT, respectively. Similar results were found in other studies [16, 30].
Regarding the single OAR (Table 2), the LAD presented the largest variability of contouring probably due to the small volume and the difficulty of visualization and identification on the CT imaging, especially without any contrast medium [31]. Similarly, the oesophagus faced the same challenge of being identified along the entire length and presented a borderline DSC. Structures with small volumes are a source of uncertainties and may cause inaccurate dosimetric evaluation, which is exacerbated by highly conformal RT planning techniques such as IMRT and stereotactic RT [6]. For all these reasons, it is recommendable that the RO reviews the contours drawn by the CSRT, especially when dealing with small volumes and complex organs.
The spinal cord presented a good value of DSC (>0.80, Table 2), but a poor CMD (1.22 cm, Table 2) due to the variability in the contouring in cranio-caudal direction. In fact, although the upper and lower limits of the spinal cord contouring were previously established, some observers did not carefully follow the given indications.
This study presented a number of limitations. First, it included only one CSRT and in general a small number of observers. Other studies, like the one conducted by Struikmans et al [32], included few observers, but they all were physicians and, therefore, had a common background. Second, only chest and upper abdomen OARs were the objects of investigation, and none of the patients had a complex anatomic conformation which could decrease the adherence to the written indications and the uniformity of contouring [26]. Third, the lack of gold standard OAR delineation may be seen as a limitation [10–12], but the aim of the study was to compare the contouring skills of the CSRT with those of ROs. Other studies on inter-observer variability did not incorporate a gold standard set of contours [7, 28, 33]. Finally, this study did not assess the dosimetric impact on inter-observer variability, although there is a paucity of data on variations in dosimetric planning [28].
Conclusion
Despite the short training dedicated to learn the contouring of the chest and upper abdomen OARs, the CSRT has achieved a good concordance with the ROs (both JRO and SRO). This study demonstrates that CSRT can successfully accomplish the task of contouring OARs and may represent a reliable resource to enhance departmental planning process efficiencies. Moreover, a good/excellent agreement between ROs and CSRTs in electronic portal imaging (EPI) registration in BC RT [34] and in prostate image-guided RT procedures using cone-beam CT [35] has been documented, encouraging the extension of the CSRT role and their growing involvement in all departmental activities. Further developments will include a larger number of CSRTs and the set up of a multicentric study, in order to increase the robustness of the investigation and at the same time strengthen and validate these preliminary results.
Authors’ contributions
Eleonora Miglietta is the co-first author. Barbara Alicja Jereczek-Fossa is the co-last author.
Conflicts of interest
The authors declared that there is no conflict of interest.
Funding declaration
This study was partially supported by the Associazione Italiana per la Ricerca sul Cancro (AIRC), project IG-14300 ‘Carbon ions boost followed by pelvic photon intensity-modulated RT for high-risk prostate cancer’, registered at ClinicalTrials.gov (NCT02672449), approved by IEO R86/14- IEO 98 and by a research grant from Accuray Inc. entitled ‘Data collection and analysis of Tomotherapy and CyberKnife breast clinical studies, breast physics studies and prostate study’. The Sponsors did not play any role in the study design, collection, analysis and interpretation of data nor in the writing of the manuscript and in the decision to submit the manuscript for publication.
References
1. Balaji K, Subramanian B, and Yadav P, et al (2016) Radiation therapy for breast cancer: literature review Med Dosim 41(3) 253–257 https://doi.org/10.1016/j.meddos.2016.06.005 PMID: 27545009
2. Zhang F, Wang Y, and Xu W, et al (2015) Dosimetric evaluation of different intensity-modulated radiotherapy techniques for breast cancer after conservative surgery Technol Cancer Res Treat 14(5) 515–523 https://doi.org/10.1177/1533034614551873
3. International Commission on Radiation Units and Measurements (1999) Prescribing, recording, and reporting photon beam therapy (Supplement to ICRU Report 50) (ICRU Report 62) Bethesda: ICRU
4. Alasti H, Cho YB, and Catton C, et al (2017) Evaluation of high dose volumetric CT to reduce inter-observer delineation variability and PTV margins for prostate cancer radiotherapy Radiother Oncol 125(1) 118–123 https://doi.org/10.1016/j.radonc.2017.08.012 PMID: 28859933
5. Vinod SK, Min M, and Jameson MG, et al (2016) A review of interventions to reduce inter-observer variability in volume delineation in radiation oncology J Med Imaging Radiat Oncol 60(3) 393–406 https://doi.org/10.1111/1754-9485.12462 PMID: 27170216
6. Vinod SK, Jameson MG, and Min M, et al (2016) Uncertainties in volume delineation in radiation oncology: a systematic review and recommendations for future studies Radiother Oncol 121(2) 169–179 https://doi.org/10.1016/j.radonc.2016.09.009 PMID: 27729166
7. Allozi R, Li XA, and White J, et al (2010) Tools for consensus analysis of experts’ contours for radiotherapy structure definitions Radiother Oncol 97(3) 572–578 https://doi.org/10.1016/j.radonc.2010.06.009 PMID: 20708285 PMCID: 3116271
8. Fiorino C, Reni M, and Bolognesi A, et al (1998) Intra- and inter-observer variability in contouring prostate and seminal vesicles: implications for conformal treatment planning Radiother Oncol 47(3) 285–292 https://doi.org/10.1016/S0167-8140(98)00021-8 PMID: 9681892
9. Batumalai V, Koh ES, and Delaney GP, et al (2011) Interobserver variability in clinical target volume delineation in tangential breast irradiation: a comparison between ROs and radiation therapists Clin Oncol (R Coll Radiol) 23(2) 108–113 https://doi.org/10.1016/j.clon.2010.10.004
10. Song WY, Chiu B, and Bauman GS, et al (2006) Prostate contouring uncertainty in megavoltage computed tomography images acquired with a helical tomotherapy unit during image-guided radiation therapy Int J Radiat Oncol Biol Phys 65(2) 595–607 https://doi.org/10.1016/j.ijrobp.2006.01.049 PMID: 16690441
11. Kepka L, Bujko K, and Garmol D, et al (2007) Delineation variation of lymph node stations for treatment planning in lung cancer radiotherapy Radiother Oncol 85(3) 450–455 https://doi.org/10.1016/j.radonc.2007.10.028 PMID: 18022265
12. Breen SL, Publicover J, and De Silva S, et al (2007) Intraobserver and interobserver variability in GTV delineation on FDG-PET-CT images of head and neck cancers Int J Radiat Oncol Biol Phys 68(3) 763–770 https://doi.org/10.1016/j.ijrobp.2006.12.039 PMID: 17379435
13. Caravatta L, Macchia G, and Mattiucci GC, et al (2014) Inter-observer variability of clinical target volume delineation in radiotherapy treatment of pancreatic cancer: a multi-institutional contouring experience Radiat Oncol 9 198 https://doi.org/10.1186/1748-717X-9-198 PMID: 25199768 PMCID: 4261525
14. Steenbakkers RJ, Duppen JC, and Fitton I, et al (2006) Reduction of observer variation using matched CT-PET for lung cancer delineation: a three dimensional analysis Int J Radiat Oncol Biol Phys 64(2) 435–448 https://doi.org/10.1016/j.ijrobp.2005.06.034
15. Pötter R, Eriksen JG, and Beavis AW, et al (2012) Competencies in radiation oncology: a new approach for education and training of professionals for radiotherapy and oncology in Europe Radiother Oncol 103(1) 1–4 https://doi.org/10.1016/j.radonc.2012.03.006 PMID: 22480609
16. Lee G, Fyles A, and Cho BCJ, et al (2012) Evaluation of variability in seroma delineation between clinical specialist radiation therapist and RO for adjuvant breast irradiation Pract Radiat Oncol 2 114–121 https://doi.org/10.1016/j.prro.2011.07.002 PMID: 24674087
17. Fitzpatrick K, Kelly C, and Thirion P (2005) Is there a role for radiation therapists in the delineation of organs-at-risk in conformal radiotherapy for prostate cancer? Radiother Oncol 76 S99–S100 [abstract] https://doi.org/10.1016/S0167-8140(05)81181-8
18. Breast Cancer Atlas [www.rtog.org] Date accessed 1/12/19
19. Dice LR (1945) Measures of the amount of ecologic association between species Ecology 26(3) 297–302 https://doi.org/10.2307/1932409
20. Italian Association of Radiation Oncology (2013) Gruppi di studio. Mammella. La Radioterapia Dei Tumori Della Mammella. Indicazioni e Criteri Guida pp 75-81 [www.radioterapiaitalia.it] Date accessed: 1/12/19
21. Offersen BV, Boersma LJ, and Kirkove C, et al (2016) ESTRO consensus guideline on target volume delineation for elective radiation therapy of early stage breast cancer, version 1.1 Radiother Oncol 118(1) 205–208 https://doi.org/10.1016/j.radonc.2015.12.027 PMID: 26791404
22. Offersen BV, Boersma LJ, and Kirkove C, et al (2015) ESTRO consensus guideline on target volume delineation for elective radiation therapy of early stage breast cancer Radiother Oncol 114(1) 3–10 https://doi.org/10.1016/j.radonc.2014.11.030 PMID: 25630428
23. Duane F, Aznar MC, and Bartlett F, et al (2017) A cardiac contouring atlas for radiotherapy Radiother Oncol 122(3) 416–422 https://doi.org/10.1016/j.radonc.2017.01.008 PMID: 28233564 PMCID: 5356506
24. Feng M, Moran JM, and Koelling T, et al (2011) Development and validation of a heart atlas to study cardiac exposure to radiation following treatment for breast cancer Int J Radiat Oncol Biol Phys 79(1) 10–18 https://doi.org/10.1016/j.ijrobp.2009.10.058
25. Kong FM, Ritter T, and Quint DJ, et al (2011) Consideration of dose limits for organs at risk of thoracic radiotherapy: atlas for lung, proximal bronchial tree, esophagus, spinal cord, ribs, and brachial plexus Int J Radiat Oncol Biol Phys 81(5) 1442–1457 https://doi.org/10.1016/j.ijrobp.2010.07.1977
26. Ciardo D, Argenone A, and Boboc GI, et al (2017) Variability in axillary lymph node delineation for breast cancer radiotherapy in presence of guidelines on a multi-institutional platform Acta Oncol 56(8) 1081–1088 https://doi.org/10.1080/0284186X.2017.1325004 PMID: 28534430
27. Cante D, Petrucci E, and Piva C, et al (2016) Delineation of the larynx as organ at risk in radiotherapy: a contouring course within “Rete Oncologica Piemonte-Valle d’Aosta” network to reduce inter- and intraobserver variability Radiol Med 121(11) 867–872 https://doi.org/10.1007/s11547-016-0668-8 PMID: 27422528
28. Li XA, Tai A, and Arthur DW, et al (2009) Variability of target and normal structure delineation for breast cancer radiotherapy: an RTOG multi-institutional and multiobserver study Int J Radiat Oncol Biol Phys 73(3) 944–951 https://doi.org/10.1016/j.ijrobp.2008.10.034 PMID: 19215827 PMCID: 2911777
29. Orecchia R, Rojas DP, and Cattani F, et al (2018) Hypofractionated postmastectomy radiotherapy with helical tomotherapy in patients with immediate breast reconstruction: dosimetric results and acute/intermediate toxicity evaluation Med Oncol 35(3) 39 https://doi.org/10.1007/s12032-018-1095-6 PMID: 29442173
30. Lee G, Dinniwell R, and Liu FF, et al (2016) Building a new model of care for rapid breast radiotherapy treatment planning: evaluation of the advanced practice radiation therapist in cavity delineation Clin Oncol 28(12) e184–e191 https://doi.org/10.1016/j.clon.2016.07.013
31. Lee J, Hua KL, and Hsu SM, et al (2017) Development of delineation for the left anterior descending coronary artery region in left breast cancer radiotherapy: an optimized organ at risk Radiother Oncol 122(3) 423–430 https://doi.org/10.1016/j.radonc.2016.12.029 PMID: 28087071
32. Struikmans H, Warlam-Rodenhuis C, and Stam T, et al (2005) Interobserver variability of clinical target volume delineation of glandular breast tissue and of boost volume in tangential breast irradiation Radiother Oncol 76(3) 293–299 https://doi.org/10.1016/j.radonc.2005.03.029 PMID: 16165237
33. Darby S, McGale P, and Correa C, et al (2011) Effect of radiotherapy after breast-conserving surgery on 10-year recurrence and 15-year breast cancer death: meta-analysis of individual patient data for 10,801 women in 17 randomised trials Lancet 378(9804) 1707–1716 https://doi.org/10.1016/S0140-6736(11)61629-2 PMID: 22019144 PMCID: 3254252
34. Jereczek-Fossa BA, Santoro L, and Colangione SP, et al (2013) Electronic portal imaging registration in breast cancer radiotherapy verification: analysis of inter-observer agreement among different categories of health practitioners Neoplasma 60(3) 302–308 https://doi.org/10.4149/neo_2013_040 PMID: 23374000
35. Jereczek BA, Pobbiati C, and Santoro L, et al (2014) Prostate positioning using cone-beam computer tomography based on manual soft-tissue registration: interobserver agreement between radiation oncologists and therapists Strahlenther Onkol 190(1) 81–87 https://doi.org/10.1007/s00066-013-0387-1
