The Feasibility and Integration of a Clinical Decision Support System with an Artificial Intelligence Algorithm that Optimises the Care Pathway of Nursing Home Residents: Results of the INTEL@MED-FAISA Study

Caroline Gayot, PhD 1, 2, 3; Cécile Laubarie-Mouret, MD1, 2; Delphine Marchesseau MD 1,2; Noëlle Cardinaud MD 1, 2; Achille Tchalla, MD, PhD 1, 2, 3

Abstract

Background: Nursing home residents often exhibit both polypathology and a dependency that require medical care. However, in medical deserts, access to care is difficult, sometimes associated with hospitalisation or disruption of the care pathway. Clinical decision support systems with Artificial Intelligence algorithms have been validated in various clinical fields but none yet takes the holistic approach required when caring for nursing home residents.

Aim: We explored the feasibility of integrating a clinical decision support systems with Artificial Intelligence algorithm tool into nursing home care. We sought to improve holistic gerontological care.

Methods: We included nursing home residents with medical events requiring the attendance of a general practitioner. Nurses and residents completed interviews using the clinical decision support systems with Artificial Intelligence algorithm tool incorporated into a tablet. Next, reports were sent to remote physicians. We compared the diagnostic severity of the medical event and the aetiological diagnostic hypotheses suggested by the tool and the remote physician. We also evaluated user acceptability.

Results: Eighteen medical events were reported. The clinical decision support systems with Artificial Intelligence algorithm tool was unable to provide reports on four occasions because details were lacking, but diagnostic severity was always assessed. Sixteen missed diagnoses specific to the elderly were identified. The concordances between the on-site and remote physician diagnostic severity levels and aetiological hypotheses were 66.7% and 71.4% respectively. Fourteen users (residents and professionals) of the tool completed the acceptability questionnaire. Nurses and physicians found that the tool was convenient, useful, and simple, but also rather time-consuming because of poor between-software interoperability. Some remote physicians did not trust their diagnoses because medical histories were not available to them. Residents reported that evaluations using the tool and remote physicians were acceptable.

Conclusion: This Intel@Med-Faisa study identified how the clinical decision support systems with Artificial Intelligence algorithm tool can be better adapted to reflect the characteristics of nursing home residents and the needs of different users. The next step is proof-of-concept evaluation. Clinicaltrials.gov number: NCT04242043

Keywords

• Artificial Intelligence
• Clinical Decision Support System
• Nursing Home Residents
• Holistic Care
• Feasibility Study

Introduction

As ‘baby boomers’ age and life expectancies increase, more people are moving into nursing homes (NHs). Today, about 4% of all U.S. and 1% of French seniors live in NHs. These proportions are expected to double over the next three decades. Nursing home residents are at higher risk of events requiring care than are others. Such residents exhibit age-related impairments, multimorbidity, and polypharmacy. A comprehensive, holistic, and coordinated approach to care must consider the interactions between intrinsic capacity (cognitive impairment, functional disability, denutrition, sensor deficits, depressive symptoms and frailty) and chronic diseases (multimorbidity) that are associated with a polymedication risk. However, in areas with few general practitioners (GPs), access to care is difficult, creating inequalities in primary care that disrupt care pathways and increase healthcare costs. Up to 67% of resident hospitalisations could have been avoided had primary care been available.

New technologies may aid the creation of novel models of elderly care. Over the past few decades, artificial intelligence (AI) has learnt how to simulate physician reasoning and thinking, and has been incorporated into clinical decision support systems (CDSSs) that improve medical decision-making and patient outcomes. These tools find applications in diagnostic imaging (identifying lesions and other features that cannot be detected with the naked eye), cancer screening, emergency triage, dementia screening, treatment decision-making, and prognostic assessments. Artificial Intelligence shows promise when used to deal with the care challenges posed by those with Alzheimer’s disease. Artificial Intelligence facilitates early detection of frailty and allows patients to maintain their autonomy at home. Artificial Intelligence has already been used to assist elderly populations. Ma et al. reviewed the field. Artificial Intelligence is used to control robots and exoskeletal devices and to create intelligent homes. Artificial Intelligence finds applications in “smart health” devices and wearables, voice-activated systems, and virtual reality. However, no AI-based CDSS has yet been used to take a holistic approach toward elderly care. A French AI-based CDSS tool termed MedVir developed by the Medical Intelligence Service exhibited an 87% accuracy rate when used for emergency triage of a general population (not just older adults). How might MedVir assist older adults with multimorbidity and polypharmacy?

We developed a research pipeline that combined AI with a CDSS tool and Telemedicine (TML) to coordinate the health care of seniors in NHs remotely. This is a new care paradigm. This Intel@Med-Faisa trial is the first study to evaluate the feasibility of AI-based CDSS implementation in NHs in a medical desert.

Methods

The Intel@Med-Faisa study is an academic, multicentre, open-label, single arm feasibility study. The protocol was approved by the French Ethics Committee. The Clinicaltrials.gov registration number is NCT04242043. The trial was planned to run from December 2019 to March 2020, but was extended for 3 months, initially to May 27 2020 and then to June 21 2020, because of a gastroenteritis epidemic followed by the Covid-19 health crisis. The nursing home coordinating physicians (the study investigators), nurses, and remote practitioners (RPs) underwent good clinical practice and AI-CDSS tool training.

The study proceeded in two NHs (one public, one private) located near the Limoges Hospital Centre. All residents aged 65 years or over who agreed to take part in the study and who presented with symptoms requiring GP attention were included. The principal objective was to verify that the diagnostic parameters of the artificial intelligence algorithm considered the clinical characteristics of an elderly population (multimorbidity and polypharmacy). The secondary objectives were assessment of the diagnostic and aetiological concordances between the AI-CDSS tool and the RPs, and user perceptions of the assessments and tool feasibility.

Procedure

When a resident experienced a medical event requiring a general physician’s attention, the nurse used the AI-CDSS tool incorporated into a tablet to perform a medical survey. Based on the answers provided by the resident and/or the nurse, the tool listed possible diagnoses and graded the event and the appropriate triage into one of three levels ranging from an emergency to a minor ailment. The nurses then sent the anonymised reports to the Clinical Research and Innovation Unit in Gerontology of CHU Limoges. The study coordinator masked all information on event severity, the diagnosis, and the intervention time, and then sent the data to a remote physician (RP) (a geriatrician). The RP independently diagnosed the event severity and pathology using the data in the AI-CDSS tool report. Finally, the study coordinator explored whether the AI-CDSS and RP proposals matched. To ensure that the care offered by the GP was respected, the GP was contacted by the nurse before the nurse used the AI-CDSS tool.

The Technology

We used the MedVir CDSS (Medical Intelligence Service, Paris, France). MedVir employs an artificial neural network running a fuzzy-logic-based AI algorithm to help clinicians to triage patients based on urgency. MedVir was trained on information in the Pubmed, HAS, PNDS, Medline, and Cochran databases, and on proprietary ontologies that define and classify the symptoms of 1,200 different database diagnoses. The algorithm assesses the characteristics and severity of clinical symptoms by reference to the five levels of the Clinical Classification of Emergency Diseases and then recodes these into three severity levels using the criteria for hospitalization of geriatric patients (Table 1). The algorithm also proposes diagnoses and suggests a level of event severity (from 1 to 5) even if no diagnostic hypothesis is made.

Analysis

To meet the principal objective, we listed all diagnoses and parameters that the Clinical decision support systems with artificial intelligence tool should have reported but did not. To explore concordance, the severity and etiological diagnoses were binarized as follows: “YES” if they were shared by the AI-CDSS tool and the remote physicians but “NO” otherwise. We calculated the “YES” proportions. Quantitative variables are presented as medians with interquartile ranges. Qualitative variables are given as numbers with percentages and 95% confidence intervals. The exact method was used to handle small numbers.

Results

Study Population

Between January and June 2020, 18 nursing home residents were included in the study. The study was interrupted twice; once because of a gastroenteritis epidemic and once because of the Covid-19 pandemic. Of the 18 residents, 78% were in NH1. Fourteen residents were women. Their mean age was 88.72 ± 7.73 years and their mean “Groupe Iso-Resource” (GIR) score, which measures physical dependence on a scale of 6 (totally independent) to 1 (totally dependent), was 2.6 ± 1.29. The principal chronic diseases were neurocognitive disorders (66.7%), cardiovascular risk factors (66.7%), and a history of falls in the year prior to inclusion (44.4%). All residents exhibited polypharmacy (more than four medications).

Four AI-CDSS reports lacked sufficient descriptive elements for a diagnostic proposal. However, the RPs developed diagnostic hypotheses for three of these cases based on information in the AI-CDSS reports.

Table 2: Characteristics of the study population.

* Physical dependence: 6 completely independent to 1 completely dependent.

Severity Level Concordance

Severity was graded on three levels. The concordance between the AI-CDSS tool and the RPs (all three levels) was 66.7%. The AI-CDSS tool and the RPs diagnosed the severity of all cases. Including non-severe events, in 83.3% of all cases, the AI-CDSS hypothesised a severity in the range of 1 to 5.

Diagnostic Concordance

The main symptom analysed by the AI-CDSS tool was the major symptom of the resident, the accompanying signs were the associated symptoms, and the diagnostic hypotheses were listed from most to least likely. The diagnostic concordance (at least one hypothesis of both the AI-CDSS tool and the RP) was 10/14 across the applicable interrogations (71.4%). A table showing all diagnostic concordances between the RPs and the AI-CDSS tool can be found in Appendix 1.

Table 3: The severity concordance between the AI-CDSS tool and the RPs.

Acceptability (Figure 1)

Fourteen AI-CDSS tool users completed the acceptability questionnaire: two nursing home directors, three residents, three nurses, two coordinating physicians, and four remote physicians. The two directors considered that the staff found the tool attractive and that the tool added value. However, they emphasised the need to train nurses how to use the technology and that NH procedures had to be re-organised when integrating the AI CDSS tool into health management. Specifically, NH-wide Wifi was required.

The three residents indicated that the questions were understandable, but the three nurses disagreed, reporting that they had to rephrase the questions. Two of the three residents would have preferred face-to-face examination by a GP.

The nurses required training on the AI-CDSS tool, even though they reported that the “app” was ergonomic and data entry was intuitive. If the technology was not used daily, refresher training was required. The nurses also thought that the app could be simplified for 75% of professionals. The data entry time was too long (average 15 min), especially if the nurse was alone when delivering the usual care and entering data into the AI-CDSS tool. At the end of the study, a nurse from NH1 reported that the tool entry time had fallen to 5–7 min, showing that training experience was essential to master the tool fully. However, all nurses considered that AI helped to optimise resident care and aided evolution of their nursing practice.

The coordinating physicians of both NHs considered that the diagnostic tool could optimise resident care both in terms of timeliness and care quality. However, neither was prepared to blindly trust AI. They insisted on seeing all residents and making their own diagnoses.

The four remote physicians required more information on residents (concomitant treatments, medical histories, and all clinical parameters). The AI reports mentioned only information that AI considered relevant in terms of a medical event requiring a general physician’s attention. Two of the four remote physicians tended to think that AI can be trusted, one thought that regular practice of AI-CDSS is essential before trusting AI, and the last remote physician expressed no opinion.

Technical Review

Nurses identified bugs or inconsistencies. Some boxes could not be unchecked. If ticked by mistake, they could not be unticked and questioning had to be restarted. This problem is inherent to the use of Internet radio buttons. It is often circumvented by adding an additional radio button when making non-binary choices that codes “none of these”, equivalent to “none of these propositions”. Also, it was impossible to enter a symptom/sign duration of greater than 9 days or an age over 100 years. These deficiencies have been corrected in version 3. Finally, if the reported interview duration was wrong, it was impossible to estimate the real time taken by the nurse. In the new version, timing commences only after the first page has been completed, which records patient age and sex.

Discussion

To the best of our knowledge, this is the first study to explore the utility of a holistic AI-CDSS tool in terms of improving the health of elderly NH residents. Despite the low number of participants (18 residents), the study population is representative of that of French NHs.

After receiving the first AI reports, all RPs immediately asked how the AI-CDSS tool worked, even if they had been trained by the tool developer. The RPs understood that only data selected by nurses were included in the AI report. This gave (a perhaps erroneous) impression of an “incomplete report”. In fact, only relevant elements were mentioned in the artificial intelligence reports.

For example, a nurse stated ‘for the box marked “Fever”, a check means “yes, the patient has a fever”; if we don’t check it, it means “the patient has no fever”. The remote physician does not know whether the question has been asked but the answer is no, or whether the question has not been asked and there remains some doubt as to whether the patient has a fever’.

In an effort to understand how the AI operated, we asked the RPs to re-play the NH cases on the platform. Data re-entry allowed the RPs to visualise the entire interrogation (including questions to which the nurses gave negative answers). The RPs then understood how the AI reasoned. The RPs identified missing data and the important clinical signs. They appreciated the added value that they (the RPs) imparted. When the RPs accepted ownership of the AI-CDSS tool, most reports could be produced without re-play, and were accurate.

The severities of functional complaints reported by the AI-CDSS tool were in agreement with those of the RPs in 66% of cases. In four AI-CDSS reports classified as “non-applicable” the tool did not suggest diagnostic hypotheses, but did output severity levels that agreed with those of the RPs (all four cases). The tool suggested a severity level for all cases, and this was accurate. This is important because, if a general physician is unavailable, NH professionals will know what to do even when a diagnosis is uncertain. This may avoid hospitalisations that may accelerate loss of independence in terms of the activities of daily living and reduce the likelihood of recovery from a disability.

The diagnostic concordance between the decisions of the AI-CDSS tool and the RPs was 71%. This is encouraging. The AI-CDSS tool was previously validated only for a general population, not specifically for older adults. The diagnostic concordance could be improved by integrating 16 older-adult-specific diagnoses considered by the RPs that were not on the list of the AI-CDSS tool. Removal of technical bugs would also enhance accuracy. The AI-CDSS tool used in this study is a “stand-alone” tool. To improve diagnostic matching between the AI-CDSS tool and the RPs, it is essential to allow the RPs to access medical histories by linking to the NH database. Also, nurses (not residents) should enter the principal symptoms.

This feasibility study highlights the need for unanimous AI tool acceptance. The Technology Acceptance Model of Venkatesh et al. emphasises the positive effects of perceived usefulness, the potential to improve care coordination, and ease of application. These are the foundations of technological acceptance. We found no psychosociological barrier to the implementation of our AI-based CDSS in NHs. This was also the case when an AI-based CDSS was employed in emergency departments. However, in our study, the resident is cared for by the nursing home staff and then by the RP. They do not communicate directly with the physician. As found in the literature, residents believe in the use of CDSS by the staff, but residents still express their wish to maintain a physical connection with their physician who treats them. The RPs stated that the data provided by the AI-based CDSS were essential in terms of decision-making. Nurse turnover can be viewed as a weakness because constant training is required for the nursing home staff, a threat because personalised care is lacking, but also an opportunity because the technology may assist non-geriatric nurses positively impact certain aspects of their performance and care outcomes. Turnover requires careful attention because this can be associated with poor-quality care.

We previously examined the efficacy and cost-effectiveness of a preventative gerontological telemedicine program in terms of reducing both unplanned hospitalisations and falls of NH residents in a medical desert, but the need for attending GPs and the number of unplanned emergency department visits, without admission, seemed to increase. In the near future, we will combine the IA-CDSS tool with a telemedicine technology that provides the RP, and potentially also GPs, with all clinical information required to come to a correct decision.

Conclusions

The Intel@Med-Faisa study showed that it was necessary to improve the integration and efficacy of the AI-CDSS tool used to coordinate the care of NH residents before moving to proof-of-concept. The study revealed the challenges posed by a future large-scale integration of AI-CDSS. The ethical and legal dimensions must be considered, and the trust of all players must be assured.

Conflicts of Interest Statement

The authors have no conflicts of interest to declare.

Funding Statement

Each institution (two nursing homes and one hospital) funded the study from in-house resources.

Acknowledgements

This paper is based on research supported by the Limoges University Hospital Centre. We thank the Medical Intelligence Service (MedVir) for the AI-CDSS tool. The views and opinions expressed in this paper are those of the authors alone and do not necessarily reflect those of any of the participant organisations. We also thank the two nursing homes where the study took place: EHPAD du Roussillon, Centre Communale d’Action Sociale (CCAS) Ville de Limoges and EHPAD des Bayles, Commune de Isle at Limoges, France.

References

1. U.S. Department of Health and Human Services. Profile of Older Americans. U.S. Department of Health and Human Services; 2022.
2. A. Évrard, C. Tchobanian. En 2019, 1,6 Million de Personnes Vivent En Communauté : Ehpad, Internat, Foyer de Travailleurs…; 2022.
3. Gordon AL, Franklin M, Bradshaw L, Logan P, Elliott R, Gladman JRF. Health status of UK care home residents: a cohort study. Age Ageing. 2014;43(1):97-103. doi:10.1093/ageing/aft077
4. Jokanovic N, Tan ECK, Dooley MJ, Kirkpatrick CM, Bell JS. Prevalence and Factors Associated With Polypharmacy in Long-Term Care Facilities: A Systematic Review. J Am Med Dir Assoc. 2015;16(6):535.e1-535.e12. doi:10.1016/j.jamda.2015.03.003
5. Beard JR, Officer A, de Carvalho IA, et al. The World report on ageing and health: a policy framework for healthy ageing. Lancet Lond Engl. 2016;387(10033):2145-2154. doi:10.1016/S0140-6736(15)00516-4
6. Ouslander JG, Lamb G, Perloe M, et al. Potentially avoidable hospitalizations of nursing home residents: frequency, causes, and costs: [see editorial comments by Drs. Jean F. Wyman and William R. Hazzard, pp 760-761]. J Am Geriatr Soc. 2010;58(4):627-635. doi:10.1111/j.1532-5415.2010.02768.x
7. Spector WD, Limcangco R, Williams C, Rhodes W, Hurd D. Potentially Avoidable Hospitalizations for Elderly Long-stay Residents in Nursing Homes. Med Care. 2013;51(8):673-681. doi:10.1097/MLR.0b013e3182984bff
8. Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med. 2020;3:17. doi:10.1038/s41746-020-0221-y
9. Gulshan V, Peng L, Coram M, et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA. 2016;316(22):2402. doi:10.1001/jama.2016.17216
10. Liu C, Liu X, Wu F, Xie M, Feng Y, Hu C. Using Artificial Intelligence (Watson for Oncology) for Treatment Recommendations Amongst Chinese Patients with Lung Cancer: Feasibility Study. J Med Internet Res. 2018;20(9):e11087. doi:10.2196/11087
11. Fernandes M, Vieira SM, Leite F, Palos C, Finkelstein S, Sousa JMC. Clinical Decision Support Systems for Triage in the Emergency Department using Intelligent Systems: a Review. Artif Intell Med. 2020;102:101762. doi:10.1016/j.artmed.2019.101762
12. Tolonen A, Rhodius-Meester HFM, Bruun M, et al. Data-Driven Differential Diagnosis of Dementia Using Multiclass Disease State Index Classifier. Front Aging Neurosci. 2018;10:111. doi:10.3389/fnagi.2018.00111
13. Ritter Z, Vogel S, Schultze F, et al. Using Explainable Artificial Intelligence Models (ML) to Predict Suspected Diagnoses as Clinical Decision Support. In: Séroussi B, Weber P, Dhombres F, et al., eds. Studies in Health Technology and Informatics. IOS Press; 2022. doi:10.3233/SHTI220529
14. Butler M, McCreedy E, Nelson VA, et al. Does Cognitive Training Prevent Cognitive Decline?: A Systematic Review. Ann Intern Med. 2018;168(1):63. doi:10.7326/M17-1531
15. Ma B, Yang J, Wong FKY, et al. Artificial intelligence in elderly healthcare: A scoping review. Ageing Res Rev. 2023;83:101808. doi:10.1016/j.arr.2022.101808
16. Kamal S. Usage de La Médecine Numériques Aux Urgences. Hassan II; 2017.
17. Gayot C, Ribet J, Trimouillas J, Micallef L, Tchalla A. Which place for Artificial Intelligence in the concept of “the nursing home of the future”? Intel@Care project in the region of Nouvelle Aquitaine in France. Trayectorias Humanas Trascontinentales. 2021;(12). doi:10.25965/trahs.4229
18. Etienne L, Faux F, Roecker O. Modeling and Representation by Graphs of the Reasoning of an Emergency Doctor: Symptom Checker MedVir. In: Tucker A, Henriques Abreu P, Cardoso J, Pereira Rodrigues P, Riaño D, eds. Artificial Intelligence in Medicine. Vol 12721. Lecture Notes in Computer Science. Springer International Publishing; 2021:418-427. doi:10.1007/978-3-030-77211-6_49
19. Fourestié V, Roussignol E, Elkharrat D, Rauss A, Simon N. Classification clinique des malades des urgences : Définition et reproductibilité. Réanimation Urgences. 1994;3(5):573-578. doi:10.1016/S1164-6756(05)80760-1
20. Balavoine A. Premiers Résultats de l’enquête EHPA 2019. DREES; 2022.
21. Bazin M, Muller M. Le personnel et les difficultés de recrutement dans les Ehpad. Published online June 2018. https://drees.solidarites-sante.gouv.fr/sites/default/files/er_1067.pdf
22. Loyd C, Markland AD, Zhang Y, et al. Prevalence of Hospital-Associated Disability in Older Adults: A Meta-analysis. J Am Med Dir Assoc. 2020;21(4):455-461.e5. doi:10.1016/j.jamda.2019.09.015
23. Grechuta K, Shokouh P, Alhussein A, et al. Benefits of Clinical Decision Support Systems for the Management of Noncommunicable Chronic Diseases: Targeted Literature Review. Interact J Med Res. 2024;13:e58036. doi:10.2196/58036
24. Venkatesh V, Davis FD. A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies. Manag Sci. 2000;46(2):186-204. doi:10.1287/mnsc.46.2.186.11926
25. Fujimori R, Liu K, Soeno S, et al. Acceptance, Barriers, and Facilitators to Implementing Artificial Intelligence–Based Decision Support Systems in Emergency Departments: Quantitative and Qualitative Evaluation. JMIR Form Res. 2022;6(6):e36501. doi:10.2196/36501
26. Zondag AGM, Rozestraten R, Grimmelikhuijsen SG, et al. The Effect of Artificial Intelligence on Patient-Physician Trust: Cross-Sectional Vignette Study. J Med Internet Res. 2024;26:e50853. doi:10.2196/50853
27. Mebrahtu TF, Bloor K, Ledward A, et al. Effects of computerised clinical decision support systems (CDSS) on nursing and allied health professional performance and patient outcomes. Cochrane Effective Practice and Organisation of Care Group, ed. Cochrane Database Syst Rev. Published online March 23, 2021. doi:10.1002/14651858.CD014699
28. Shen K, McGarry BE, Gandhi AD. Health Care Staff Turnover and Quality of Care at Nursing Homes. JAMA Intern Med. 2023;183(11):1247. doi:10.1001/jamainternmed.2023.5225
29. Gayot C, Laubarie-Mouret C, Zarca K, et al. Effectiveness and cost-effectiveness of a telemedicine programme for preventing unplanned hospitalisations of older adults living in nursing homes: the GERONTACCESS cluster randomized clinical trial. BMC Geriatr. 2022;22(1):991. doi:10.1186/s12877-022-03575-6
30. Kumar P, Chauhan S, Awasthi LK. Artificial Intelligence in Healthcare: Review, Ethics, Trust Challenges & Future Research Directions. Eng Appl Artif Intell. 2023;120:105894. doi:10.1016/j.engappai.2023.105894