Breast Cancer Decision Support: Expert Systems vs. Machine Learning – have we thrown the baby out with the bathwater?
Main Article Content
Abstract
Background and Aim: Machine learning enabled clinical decision support systems offer the potential to enhance the efficiency of decision-making processes in breast cancer multidisciplinary team meetings. We examine the circumstances where a traditional rule-based expert system may have advantages over machine learning enabled clinical decision support systems.
Methods: We compared the concordance of an expert system (Deontics) and a machine learning enabled clinical decision support system (Watson for Oncology) with the treatment recommendations of a gold standard consensus panel of breast surgeons and medical oncologists for 208 non-metastatic breast cancer patients, and for 165 patients deemed eligible for triage to an agreed standard of care, and ‘not for discussion at multidisciplinary team meeting’.
Results: The overall concordance between the Deontics clinical decision support system treatment plan recommendations and the gold standard consensus panel was 98% compared to 92% for the machine learning enabled clinical decision support system. Using a clinical decision tree, 79% of patients were eligible for triage to a standard of care and ‘not for discussion at multidisciplinary team meetings’; for these patients the concordance between the Deontics clinical decision support system and gold standard consensus panel was 98.8% (95% CI: 95.6-99.8%), whilst for the machine learning enabled clinical decision support system concordance was 78.8% (95% CI: 71.7-84.8%).
Conclusion: The high level of agreement between the Deontics clinical decision support system and clinical consensus suggests it may be acceptable for use in the breast multidisciplinary team pathway, whereas the level of disagreement observed for the machine learning enabled clinical decision support system would result in a clinically unsafe error rate if used to triage patients away from the multidisciplinary team meetings. These findings, if replicated in prospective studies in a routine clinical setting, could improve the efficiency of UK breast cancer diagnostic and treatment pathways.
Article Details
The Medical Research Archives grants authors the right to publish and reproduce the unrevised contribution in whole or in part at any time and in any form for any scholarly non-commercial purpose with the condition that all publications of the contribution include a full citation to the journal as published by the Medical Research Archives.
References
2. Patkar V, Acosta D, Davidson T, et al. Cancer multidisciplinary team meetings: evidence, challenges, and the role of clinical decision support technology. Int J Breast Cancer. 2011;2011:831605.
3. Prades J, Remue E, van Hoof E, Borras JM. Is it worth reorganising cancer services on the basis of multidisciplinary teams (MDTs)? A systematic review of the objectives and organisation of MDTs and their impact on patient outcomes. Health Policy. 2015;119(4):464-474.
4. Eaker S, Dickman PW, Hellstrom V, et al. Regional differences in breast cancer survival despite common guidelines. Cancer Epidemiol Biomarkers Prev. 2005;14(12):2914-2918.
5. De Ieso PB, Coward JI, Letsa I, et al. A study of the decision outcomes and financial costs of multidisciplinary team meetings (MDMs) in oncology. Br J Cancer. 2013;109(9):2295-2300.
6. Sibering M. Improving the Efficiency of Breast Multidisciplinary Team Meetings: A Toolkit for Breast Services 2022.
7. Streamlining multi-disciplinary team meetings: guidance for cancer alliance. NHS England and NHS Improvement.
8. Mazo C, Kearns C, Mooney C, Gallagher WM. Clinical Decision Support Systems in Breast Cancer: A Systematic Review. Cancers (Basel). 2020;12(2).
9. Sutton RT, Pincock D, Baumgart DC, et al. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med. 2020;3:17.
10. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293 (10):1223-1238.
11. Roshanov PS, Fernandes N, Wilczynski JM, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657.
12. Pawloski PA, Brooks GA, Nielsen ME, Olson-Bullis BA. A Systematic Review of Clinical Decision Support Systems for Clinical Oncology Practice. J Natl Compr Canc Netw. 2019;17(4):331-338.
13. Sloane EB JSR. Artificial intelligence in medical devices and clinical decision support systems. Clinical Engineering Handbook 2020:556-568.
14. Martin M, Kristeleit, H., Ruta, D., et al. Augmentation of a multidisciplinary team meeting with a clinical decision support system to triage breast cancer patients in the United Kingdom. Future Medicine AI. 2023;1(1):1-12.
15. Scott IA. Machine Learning and Evidence-Based Medicine. Ann Intern Med. 2018;169(1):44-46.
16. Greenes RA, Bates DW, Kawamoto K, et al. Clinical decision support models and frameworks: Seeking to address research issues underlying implementation successes and failures. J Biomed Inform. 2018;78:134-143.
17. Klarenbeek SE, Weekenstroo HHA, Sedelaar JPM, et al. The Effect of Higher Level Computerized Clinical Decision Support Systems on Oncology Care: A Systematic Review. Cancers (Basel). 2020;12(4).
18. Johnson KB, Wei WQ, Weeraratne D, et al. Precision Medicine, AI, and the Future of Personalized Health Care. Clin Transl Sci. 2021; 14(1):86-93.
19. Singh S, Kumar R, Payra S, Singh SK. Artificial Intelligence and Machine Learning in Pharmacological Research: Bridging the Gap Between Data and Drug Discovery. Cureus. 2023;15(8):e44359.
20. Amann J, Vetter D, Blomberg SN, et al. To explain or not to explain?-Artificial intelligence explainability in clinical decision support systems. PLOS Digit Health. 2022;1(2):e0000016.
21. Tonekaboni S, Joshi, S., McCradden, M., Goldenberg, A. What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. Proceedings of Machine Learning Research. 2019:1-21.
22. Bradley A, van der Meer R, McKay C. Personalized Pancreatic Cancer Management: A Systematic Review of How Machine Learning Is Supporting Decision-making. Pancreas. 2019;48 (5):598-604.
23. Gradishar WJ, Moran MS, Abraham J, et al. Breast Cancer, Version 3.2024, NCCN Clinical Practice Guidelines in Oncology. J Natl Compr Canc Netw. 2024;22(5):331-357.
24. Early and locally advanced breast cancer: diagnosis and management 2024.
25. Yu SH, Kim MS, Chung HS, et al. Early experience with Watson for Oncology: a clinical decision-support system for prostate cancer treatment recommendations. World J Urol. 2021;39(2):407-413.
26. Pan H, Tao J, Qian M, et al. Concordance assessment of Watson for Oncology in breast cancer chemotherapy: first China experience. Transl Cancer Res. 2019;8(2):389-401.
27. Zhuang YD, Zhou MC, Liu SC, et al. Effectiveness of personalized 3D printed models for patient education in degenerative lumbar disease. Patient Educ Couns. 2019;102(10):1875-1881.
28. Zhou N, Zhang CT, Lv HY, et al. Concordance Study Between IBM Watson for Oncology and Clinical Practice for Patients with Cancer in China. Oncologist. 2019;24(6):812-819.
29. Nafees A, Khan M, Chow R, et al. Evaluation of clinical decision support systems in oncology: An updated systematic review. Crit Rev Oncol Hematol. 2023;192:104143.
30. Saab K, Tu, T., Weng, WH., et al. Capabilities of Gemini Models in Medicine. arXiv. 2024.