A Study to Assess the Impact of an Artificial Intelligence (AI) System on Chest X-ray Reporting

NCT05489471 | Unknown DMC

• 1 other identifier: R2757
• Study Type: observational
• Target: 20,000
• Locations: 0 countries
• Sites: N/A

Brief Summary

The study has an initial short retrospective component but is predominately a prospective study with two main parts. Initially during a 1 month period whilst reporters are familiarising themselves with the software two local databases will be reviewed by the AI software:

• A training set of 100 chest X-rays (CXR) some of which contain nodules and is used as a training tool with previously documented radiologist performance.
• A set of previously reported radiographs in patients referred by the reporter for CT, ground truth created from the prior CT report and review by two radiologists if required. This will allow comparison of stand-alone radiologist and AI performance This is followed by a 6 month period involving multiple groups of reporters and approximately 20,000 cases looking at the impact of an AI system which assesses 10 abnormalities on chest X-ray and reporting on the sensitivity for detection of lesions and its impact on reporter confidence. Specifically the investigators would look at:
• Missed finding by AI, but detected by reporter
• Correctly detected finding by AI
• Missed finding by the reporter but detected by AI
• Finding detected by AI but disputed by the reporter ■ AI's impact on
• Radiological report
• Further recommended imaging
• Altering patient management
• improvement in report confidence as perceived by reporter A subsequent 3 month period looking at the impact of AI produced worklists on report turnaround times and the patient pathway from chest X-ray to CT. the investigators would specifically look at:
• number of nodules detected
• number of CXRs recommended for follow up CT
• time taken from CXR to CT
• number of lung cancers detected after CT\[1\]
• Time to report, measured as previously from PACS and reporting software data The population to be studied will be all patients over 16 years of age referred by their General Practitioner to Hull University Hospitals NHS Trust for a chest radiograph and any chest radiograph performed in the Hull Royal Infirmary ED radiology for patients over 16 years of age during the 6 month study period. The ED department images patients from the emergency department and in-patients within the hospital. All radiographs will be reviewed initially without review of the AI information and then using the additional images. Reporters will mark the effect of the AI on their decision. All disagreements between the reporter and the AI will be reviewed by senior reporters and a consensus decision made.

Conditions

Artificial Intelligence

Keywords

Chest Radiograph, AI assisted reporting; Radiology; AI assisted reporting

Outcome Measures

Primary Outcomes (1)

• Radiologist performance review

  To demonstrate AI can help to improve the radiologist performance in terms of missed finding by radiologist detected by AI ( as a percentage error rate)

  six months

Secondary Outcomes (3)

• Lung cancer detection

  six months

• Lung cancer pathway improvement

  three months

• Report turnaround times improvement

  three months

Study Arms (1)

Adult Chest Radiographs

All chest X-rays for patients over 16 years from either a GP referral or performed in the Emergency department (ED) of the acute hospital, which includes Accident and Emergency attendances and in-patient studies.

Other: Artificial intelligence review

Interventions

Artificial intelligence review OTHER

The AI looks for ten different abnormalities on each chest X-ray and produces a heat map and percentage confidence score if it detects an abnormality.

Adult Chest Radiographs

Eligibility Criteria

• Age: 16 Years+
• Sex: all
• Age Groups: Child (0-17), Adult (18-64), Older Adult (65+)
• Sampling Method: Non-Probability Sample

Study Population

The chest radiographs of patients, aged 16 years of age or over which have either been requested by a General Practitioner or taken in the radiology unit in the Emergency Department Hull University Teaching Hospitals NHS trust.

You may qualify if:

• patient 16 years or older
• Posterior-anterior and Anterior-posterior chest radiographs
• Requested by General Practitioners or performed in the Emergency Department radiology unit

You may not qualify if:

• Patients under 16 years of age
• lateral films
• Chest radiographs which are of suboptimal quality, to an extent that it is deemed uninterpretable by the reporter

Sponsors & Collaborators

• Hull University Teaching Hospitals NHS Trust: lead
• Lunit Inc.: collaborator

Related Publications (21)

• Turkington PM, Kennan N, Greenstone MA. Misinterpretation of the chest x ray as a factor in the delayed diagnosis of lung cancer. Postgrad Med J. 2002 Mar;78(917):158-60. doi: 10.1136/pmj.78.917.158.

  PMID: 11884698 BACKGROUND

• Jang S, Song H, Shin YJ, Kim J, Kim J, Lee KW, Lee SS, Lee W, Lee S, Lee KH. Deep Learning-based Automatic Detection Algorithm for Reducing Overlooked Lung Cancers on Chest Radiographs. Radiology. 2020 Sep;296(3):652-661. doi: 10.1148/radiol.2020200165. Epub 2020 Jul 21.

  PMID: 32692300 BACKGROUND

• Nam JG, Park S, Hwang EJ, Lee JH, Jin KN, Lim KY, Vu TH, Sohn JH, Hwang S, Goo JM, Park CM. Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs. Radiology. 2019 Jan;290(1):218-228. doi: 10.1148/radiol.2018180237. Epub 2018 Sep 25.

  PMID: 30251934 BACKGROUND

• Hwang EJ, Park S, Jin KN, Kim JI, Choi SY, Lee JH, Goo JM, Aum J, Yim JJ, Cohen JG, Ferretti GR, Park CM; DLAD Development and Evaluation Group. Development and Validation of a Deep Learning-Based Automated Detection Algorithm for Major Thoracic Diseases on Chest Radiographs. JAMA Netw Open. 2019 Mar 1;2(3):e191095. doi: 10.1001/jamanetworkopen.2019.1095.

  PMID: 30901052 BACKGROUND

• Hwang EJ, Lee JH, Kim JH, Lim WH, Goo JM, Park CM. Deep learning computer-aided detection system for pneumonia in febrile neutropenia patients: a diagnostic cohort study. BMC Pulm Med. 2021 Dec 7;21(1):406. doi: 10.1186/s12890-021-01768-0.

  PMID: 34876075 BACKGROUND

• Jones CM, Danaher L, Milne MR, Tang C, Seah J, Oakden-Rayner L, Johnson A, Buchlak QD, Esmaili N. Assessment of the effect of a comprehensive chest radiograph deep learning model on radiologist reports and patient outcomes: a real-world observational study. BMJ Open. 2021 Dec 20;11(12):e052902. doi: 10.1136/bmjopen-2021-052902.

  PMID: 34930738 BACKGROUND

• Kim JH, Kim JY, Kim GH, Kang D, Kim IJ, Seo J, Andrews JR, Park CM. Clinical Validation of a Deep Learning Algorithm for Detection of Pneumonia on Chest Radiographs in Emergency Department Patients with Acute Febrile Respiratory Illness. J Clin Med. 2020 Jun 24;9(6):1981. doi: 10.3390/jcm9061981.

  PMID: 32599874 BACKGROUND

• Jang SB, Lee SH, Lee DE, Park SY, Kim JK, Cho JW, Cho J, Kim KB, Park B, Park J, Lim JK. Deep-learning algorithms for the interpretation of chest radiographs to aid in the triage of COVID-19 patients: A multicenter retrospective study. PLoS One. 2020 Nov 24;15(11):e0242759. doi: 10.1371/journal.pone.0242759. eCollection 2020.

  PMID: 33232368 BACKGROUND

• Hwang EJ, Kim KB, Kim JY, Lim JK, Nam JG, Choi H, Kim H, Yoon SH, Goo JM, Park CM. COVID-19 pneumonia on chest X-rays: Performance of a deep learning-based computer-aided detection system. PLoS One. 2021 Jun 7;16(6):e0252440. doi: 10.1371/journal.pone.0252440. eCollection 2021.

  PMID: 34097708 BACKGROUND

• Hwang EJ, Kim H, Yoon SH, Goo JM, Park CM. Implementation of a Deep Learning-Based Computer-Aided Detection System for the Interpretation of Chest Radiographs in Patients Suspected for COVID-19. Korean J Radiol. 2020 Oct;21(10):1150-1160. doi: 10.3348/kjr.2020.0536. Epub 2020 Jul 17.

  PMID: 32729263 BACKGROUND

• Kim JH, Han SG, Cho A, Shin HJ, Baek SE. Effect of deep learning-based assistive technology use on chest radiograph interpretation by emergency department physicians: a prospective interventional simulation-based study. BMC Med Inform Decis Mak. 2021 Nov 8;21(1):311. doi: 10.1186/s12911-021-01679-4.

  PMID: 34749731 BACKGROUND

• Yoo H, Kim KH, Singh R, Digumarthy SR, Kalra MK. Validation of a Deep Learning Algorithm for the Detection of Malignant Pulmonary Nodules in Chest Radiographs. JAMA Netw Open. 2020 Sep 1;3(9):e2017135. doi: 10.1001/jamanetworkopen.2020.17135.

  PMID: 32970157 BACKGROUND

• Nam JG, Hwang EJ, Kim DS, Yoo SJ, Choi H, Goo JM, Park CM. Undetected Lung Cancer at Posteroanterior Chest Radiography: Potential Role of a Deep Learning-based Detection Algorithm. Radiol Cardiothorac Imaging. 2020 Dec 10;2(6):e190222. doi: 10.1148/ryct.2020190222. eCollection 2020 Dec.

  PMID: 33778635 BACKGROUND

• Yoo H, Lee SH, Arru CD, Doda Khera R, Singh R, Siebert S, Kim D, Lee Y, Park JH, Eom HJ, Digumarthy SR, Kalra MK. AI-based improvement in lung cancer detection on chest radiographs: results of a multi-reader study in NLST dataset. Eur Radiol. 2021 Dec;31(12):9664-9674. doi: 10.1007/s00330-021-08074-7. Epub 2021 Jun 4.

  PMID: 34089072 BACKGROUND

• Lee JH, Sun HY, Park S, Kim H, Hwang EJ, Goo JM, Park CM. Performance of a Deep Learning Algorithm Compared with Radiologic Interpretation for Lung Cancer Detection on Chest Radiographs in a Health Screening Population. Radiology. 2020 Dec;297(3):687-696. doi: 10.1148/radiol.2020201240. Epub 2020 Sep 22.

  PMID: 32960729 BACKGROUND

• Koo YH, Shin KE, Park JS, Lee JW, Byun S, Lee H. Extravalidation and reproducibility results of a commercial deep learning-based automatic detection algorithm for pulmonary nodules on chest radiographs at tertiary hospital. J Med Imaging Radiat Oncol. 2021 Feb;65(1):15-22. doi: 10.1111/1754-9485.13105. Epub 2020 Oct 8.

  PMID: 33090731 BACKGROUND

• Nam JG, Kim M, Park J, Hwang EJ, Lee JH, Hong JH, Goo JM, Park CM. Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs. Eur Respir J. 2021 May 20;57(5):2003061. doi: 10.1183/13993003.03061-2020. Print 2021 May.

  PMID: 33243843 BACKGROUND

• Kim EY, Kim YJ, Choi WJ, Lee GP, Choi YR, Jin KN, Cho YJ. Performance of a deep-learning algorithm for referable thoracic abnormalities on chest radiographs: A multicenter study of a health screening cohort. PLoS One. 2021 Feb 19;16(2):e0246472. doi: 10.1371/journal.pone.0246472. eCollection 2021.

  PMID: 33606779 BACKGROUND

• Hwang EJ, Nam JG, Lim WH, Park SJ, Jeong YS, Kang JH, Hong EK, Kim TM, Goo JM, Park S, Kim KH, Park CM. Deep Learning for Chest Radiograph Diagnosis in the Emergency Department. Radiology. 2019 Dec;293(3):573-580. doi: 10.1148/radiol.2019191225. Epub 2019 Oct 22.

  PMID: 31638490 BACKGROUND

• Hwang EJ, Hong JH, Lee KH, Kim JI, Nam JG, Kim DS, Choi H, Yoo SJ, Goo JM, Park CM. Deep learning algorithm for surveillance of pneumothorax after lung biopsy: a multicenter diagnostic cohort study. Eur Radiol. 2020 Jul;30(7):3660-3671. doi: 10.1007/s00330-020-06771-3. Epub 2020 Mar 11.

  PMID: 32162001 BACKGROUND

• Jin KN, Kim EY, Kim YJ, Lee GP, Kim H, Oh S, Kim YS, Han JH, Cho YJ. Diagnostic effect of artificial intelligence solution for referable thoracic abnormalities on chest radiography: a multicenter respiratory outpatient diagnostic cohort study. Eur Radiol. 2022 May;32(5):3469-3479. doi: 10.1007/s00330-021-08397-5. Epub 2022 Jan 1.

  PMID: 34973101 BACKGROUND

Study Officials

• Gerard Avery

  Hull University Teaching Hospitals NHS Trust

  PRINCIPAL INVESTIGATOR

Central Study Contacts

gerard avery, FRCR

CONTACT

+441482623205; ged.avery1@nhs.net

Oliver Byass, FRCR

CONTACT

+441482623205; oliver.byass@nhs.net

Study Design

• Study Type: observational
• Observational Model: CASE ONLY
• Time Perspective: PROSPECTIVE
• Sponsor Type: OTHER GOV
• Responsible Party: SPONSOR

Study Record Dates

• First Submitted: May 19, 2022
• First Posted: August 5, 2022
• Study Start: May 1, 2023
• Primary Completion: July 1, 2023
• Study Completion: July 1, 2023
• Last Updated: March 31, 2023

Record last verified: 2023-03

Data Sharing

• IPD Sharing: Will not share