Speaker
Description
In the rapidly changing field of prevention and rehabilitation, adaptive and personalized Artificial Intelligence (AI) systems play a crucial role as facilitators of patient-centric care. These systems, by analyzing comprehensive data on user behavior and situational context, offer support customized to the specific needs of individual patients [1, 2, 3, 4]. However, patients dealing with ailments or recovery from incidents like knee surgeries may face cognitive challenges. These challenges can result in the creation of data that is incomplete, inaccurate, or biased, significantly hampering the patient's ability to make informed decisions, understand complex medical information, or effectively communicate their symptoms and concerns [5, 6, 7, 8]. Recognizing the significant influence of these cognitive limitations, it is essential to leverage AI healthcare systems designed to mitigate these gaps, ensuring care that aligns perfectly with each patient’s distinct needs and conditions [1, 2].
The exploration of mental models [9] in therapeutic settings and their integration into AI systems introduces a promising path to improve patient care and treatment outcomes. Mental models are cognitive structures that encompass a patient's perceptions and presumptions about their therapy and rehabilitation journey [10, 11, 12, 13, 14]. Existing research highlights the need for accurately capturing and understanding these models, especially in therapeutic and recovery scenarios [15, 16, 17, 18].
Our research investigates the potential of using meta-representations of patients’ mental models called artificial mental models (AMM) within healthcare AI systems to enhance patient support, especially in making informed choices amidst the uncertainty and discomfort typical in rehabilitation phases. For instance, Carlos is an amateur soccer player recovering from knee surgery and eager to return to the field. His AMM acts as his representative by communicating directly with his physiotherapist to develop a personalized exercise plan that aligns with his goals and recovery status. The AMM schedules these exercises throughout the week, adjusting the intensity based on feedback from Carlos’s wearable devices that track his pain and performance. It also arranges video consultations with his therapist if it detects issues or deviations in his recovery progress, ensuring Carlos remains on the safest and quickest path to full activity.
In this work, we investigate the elicitation and individualization of AMMs for patients in rehabilitation situations. Large Language Models (LLMs) are used for the generation of AMMs. Training data are acquired by indirect observations, a quantitative study and direct observation. Our research is driven by the following research questions (RQ):
• RQ1: Can we elicit an individual AMM for a specific patient by using LLMs?
• RQ2: Are predictions of the AMM regarding expected pain in exercises comparable with expectations of the patient?
• RQ3: Are predictions of the AMM regarding expected pain in exercises accurate compared with assessment by the therapist?
For tackling the research questions, we specified a research design according to Design Science Research (DSR) [19, 20] covering a prospective study with two phases: elicitation and individualization. Within each phase, an artifact is created and evaluated [21]. Objective of the elicitation phase is the generation of a discrimination- and bias-free domain-specific basis AMM in the domain of knee rehabilitation. Here, large scale data are acquired on the one hand by a quantitative study (n=150) on personality traits (Big Five [22]) and expected pain in exercises (e.g., squats). On the other hand, indirect observation of patients with knee issues is applied in terms of a twofold scraping approach. Additionally, large corpora of healthcare dialogues are filtered, analyzed, and prepared for model training, e.g., HealthCareMagic [23], ChatDoctor [24], medAlpaca [25]. The resulting artifact – a discrimination- and bias-free domain-specific basis AMM is evaluated in a technical experiment (ablation study). The individualization phase uses this AMM basis model for fine-tuning an AMM for an individual patient. Here, direct observation of a specific patient is used for training the model. Curated data like medication, rehabilitation therapy plans, data on injury, surgical procedure and duration, complications, self-reported pain scales (e.g., Visual Analog Scale (VAS) [26], Wong-Baker FACES Pain Rating Scale [27]) are integrated as well as non-curated data like movement data, sleep, fitness status etc. of the patient. The resulting artifact – an AMM for a specific patient - is evaluated by an action research approach with the patient.
As result of the elicitation phase, a technically evaluated domain-specific basis AMM in the domain of knee rehabilitation is generated. For building this model, open source LLMs like BLOOM, Mistral7B or Falcon are applied. For ensuring fairness and mitigating bias in the basis AMM, discrimination and bias checks are used. Here a combination of methods is designed, i.e., counterfactual fairness testing [28], adversarial debiasing [29], fairness metrics such as equality of opportunity [30] or demographic parity [31], and interpretability. The basis AMM is used for fine-tuning the AMM for a specific patient in the individualization phase (RQ1). This patient AMM is evaluated in a real-world rehabilitation situation with the patient as part of a research intervention, evaluating its effect on the real-world situation. For the experimental setting an A/B test is planned with focus on anticipation respectively prediction of pain in a therapy unit. Predictions are generated in two variants: (A) by the patient, and (B) by the AMM. For ground truthing, predictions are mirrored with a subsequent actual pain assessment by the patient in a therapy unit. Thus, overlapping of predictions of patient and AMM in concrete cases as well as their accuracy with respect to ground truth are measured (RQ 2 & 3). Contribution of this research is the support of personalized patient care in knee rehabilitation by leveraging LLMs to create and individualize patient-specific AMM for AI systems in healthcare.
This research emphasizes the role of Artificial Intelligence (AI) systems in enhancing patient-centric care in the field of prevention and rehabilitation. It particularly focuses on the development and implementation of Artificial Mental Models (AMM) within healthcare AI systems in knee rehabilitation. These models are personalized to account for individual patients' mental models regarding their therapy and rehabilitation journeys. In this work, the potential of using Large Language Models (LLMs) to generate and fine-tune these AMMs, aiming for more accurate, bias-free, and personalized patient care, are highlighted.
By creating systems that better understand and predict patient needs and responses, healthcare providers can offer more targeted and effective interventions in knee rehabilitation. This could lead to quicker recoveries and improved overall patient outcomes. Additionally, the methodological approach of using AMMs could be expanded to other areas of healthcare, suggesting a broad potential impact on how AI is integrated into patient care systems and routine care.
References:
1. Khera, R., Butte, A.J., Berkwits, M., Hswen, Y., Flanagin, A., Park, H., Curfman, G., Bibbins-Domingo, K.: Ai in medicine—jama’s focus on clinical outcomes, patient-centered care, quality, and equity. Jama (2023)
2. Wan, T.T., Wan, H.S.: Predictive analytics with a transdisciplinary framework in promoting patient-centric care of polychronic conditions: Trends, challenges, and solutions. AI 4(3), 482–490 (2023)
3. Bollos, L.A.C.L., Zhao, Y., Soriano, G.P., Tanioka, T., Otsuka, H., Locsin, R.: Technologies, physician’s caring competency, and patient centered care: A systematic review. The Journal of Medical Investigation 70(3.4), 307–316 (2023)
4. Li, W., Ge, X., Liu, S., Xu, L., Zhai, X., Yu, L.: Opportunities and challenges of traditional chinese medicine doctors in the era of artificial intelligence. Frontiers in Medicine 10 (2023)
5. Timm, A., Schmidt-Wilcke, T., Blenk, S., Studer, B.: Altered social decision making in patients with chronic pain. Psychological Medicine 53(6), 2466–2475 (2023)
6. Meyer, A.N., Giardina, T.D., Khawaja, L., Singh, H.: Patient and clinician experiences of uncertainty in the diagnostic process: current understanding and future directions. Patient Education and Counseling 104(11), 2606–2615 (2021)
7. Dildine, T.C., Amir, C.M., Parsons, J., Atlas, L.Y.: How pain-related facial expressions are evaluated in relation to gender, race, and emotion. Affective Science pp. 1–20 (2023)
8. Dildine, T.C., Necka, E.A., Atlas, L.Y.: Confidence in subjective pain is predicted by reaction time during decision making. Scientific reports 10(1), 21373 (2020)
9. Norman, D.A.: Some observations on mental models, pp. 241–244. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1987)
10. Jones, N.A., Ross, H., Lynam, T., Perez, P., Leitch, A.M.: Mental models: an interdisciplinary synthesis of theory and methods. Ecology and Society 16(1), 46 (2011), https://api.semanticscholar.org/CorpusID:38976887
11. LaMere, K., M¨antyniemi, S., Vanhatalo, J., Haapasaari, P.: Making the most of mental models: Advancing the methodology for mental model elicitation and documentation with expert stakeholders. Environmental Modelling & Software 124, 104589 (2020)
12. Borders, J., Klein, G., Besuijen, R.: Mental model matrix: Implications for system design and training. Journal of Cognitive Engineering and Decision Making p. 15553434231226317 (2024)
13. Johnson-Laird, P.N.: Mental models. The MIT Press (1989)
14. Hermans, A., Muhammad, S., Treur, J.: You feel so familiar, you feel so different: A controlled adaptive network model for attachment patterns as adaptive mental models. In: Mental Models and Their Dynamics, Adaptation, and Control: A SelfModeling Network Modeling Approach, pp. 321–346. Springer (2022)
15. Hermans, A., Muhammad, S., Treur, J.: You feel so familiar, you feel so different: A controlled adaptive network model for attachment patterns as adaptive mental models. In: Mental Models and Their Dynamics, Adaptation, and Control: A SelfModeling Network Modeling Approach, pp. 321–346. Springer (2022)
16. Gabbas, M., Kim, K.: Gamified user interface design for dysphagia rehabilitation based on common mental models. In: DRS2022, Bilbao, Spain (2022)
17. Barber, T., Crick, K., Toon, L., Tate, J., Kelm, K., Novak, K., Yeung, R.O., Tandon, P., Sadowski, D.C., Veldhuyzen van Zanten, S., et al.: Gastroscopy for dyspepsia: Understanding primary care and gastroenterologist mental models of practice: A cognitive task analysis approach. Journal of the Canadian Association of Gastroenterology 6(6), 234–243 (2023)
18. Naik, A.D.: Collaborative decision-making: Identifying and aligning care with the health priorities of older adults. In: Geriatric Medicine: A Person Centered Evidence Based Approach, pp. 1–21. Springer (2023)
19. Hevner, A.R., March, S.T., Park, J., Ram, S.: Design science in information systems research. MIS Quaterly 28, 75–105 (2004), https://api.semanticscholar.org/CorpusID:13553735
20. Peffers, K., Tuunanen, T., Rothenberger, M.A., Chatterjee, S.: A design science research methodology for information systems research. Journal of Management Information Systems 24, 45 – 77 (2007), https://api.semanticscholar.org/CorpusID:17511997
21. Peffers, K., Rothenberger, M., Tuunanen, T., Vaezi, R.: Design science research evaluation. In: Design Science Research in Information Systems. Advances in Theory and Practice: 7th International Conference, DESRIST 2012, Las Vegas, NV, USA, May 14-15, 2012. Proceedings 7. pp. 398–410. Springer (2012)
22. Oliver P. John, Laura P. Naumann, Christopher J. Soto: (2008) Paradigm Shift to the Integrative Big Five Trait Taxonomy. Handbook of Personality Theory and Research. 3. Auflage. S. 114–117
23. https://huggingface.co/datasets/RafaelMPereira/HealthCareMagic-100k-Chat-Format-en
24. Li Y, Li Z, Zhang K, et al. (June 24, 2023) ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge. Cureus 15(6): e40895. doi:10.7759/cureus.40895
25. https://github.com/kbressem/medAlpaca
26. Crichton, N.: Visual analogue scale (vas). J Clin Nurs 10(5), 706–6 (2001)
27. Wong, D.L., Baker, C.M.: Wong-baker faces pain rating scale. Pain Management Nursing (2012)
28. Kusner, M.J., Loftus, J., Russell, C., Silva, R.: Counterfactual fairness. In: Advances in Neural Information Processing Systems. vol. 30 (2017).
29. Lemoine, B., Zhang, B., Mitchell, M.: Mitigating unwanted biases with adversarial learning. In: Proceedings of the ACM Conference (2018)
30. Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. In: Advances in Neural Information Processing Systems. vol. 29 (2016).
31. Zafar, M.B., Valera, I., Rogriguez, M.G., Gummadi, K.P.: Fairness constraints: Mechanisms for fair classification. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 54, pp. 962–970. PMLR (2017).
