91��Ƭ��

UMSAEP Activities Report November 2024

Enhancing Health IT Integration using Implementation Science.

Principal Investigator (PI): Johan Breytenbach, Ph.D. Information Systems, Economic and Management Sciences, University of Western Cape. jbreytenbach@uwc.ac.za

Co-Principal Investigator (Co-PI): Mark Patterson, Department of Pharmacy Practice and Administration, UMKC School of Pharmacy. pattersonmar@umkc.edu.

Executive Summary

This project bridges Health IT and implementation science to address barriers to Artificial Intelligence (AI) adoption in South African healthcare and other sectors. With Health IT identified as a national research priority in South Africa (2021–2025), the study applies frameworks like COM-B to investigate individual behaviors and organizational factors influencing AI adoption and integration. The COM-B model identifies psychological capabilities, physical/social opportunities, and motivation as critical determinants of AI adoption, offering actionable insights for implementation strategies.

Key activities included developing a survey tool grounded in the COM-B model to explore stakeholder attitudes, capabilities, and motivations related to AI use. The survey was piloted among South African educators, revealing variations in attitudes toward AI infrastructure, accuracy, and workplace support. Analysis highlighted critical adoption barriers, including infrastructure limitations and ethical concerns, as well as enablers, such as AI infrastructure readiness.

Collaborative efforts with local and international researchers enhanced the study's scope, culminating in panel discussions and seminars on AI adoption strategies. The findings are currently being finalized for peer-review manuscripts, focusing on our COM-B survey building methodology and our preliminary findings. Future goals include refining the survey, expanding its deployment, and fostering partnerships to develop evidence-based interventions, ultimately aiming to inform strategies that can be used to enhance Health IT adoption across diverse job sectors.

Background and Project Goals

Scientific Background

Health information technology, including information systems and AI tools, is a key public health research priority in South Africa (2021–2025). Despite its potential to improve decision-making, communication, and health outcomes, its effectiveness is hindered by challenges like interoperability, workflow integration, low digital literacy, insufficient training, and ethical concerns.

As AI technologies advance, integrating them into healthcare systems necessitates understanding factors influencing adoption, including attitudes, behaviors, and alignment with clinical needs. The project addresses organizational and behavioral factors related to the adoption of Artificial Intelligence (AI) for clinical decision-making, and more broadly, for decision making in general across diverse job sectors within South Africa.

Implementation science frameworks offer an alternative to some health IT evaluation models by introducing more robust behavioral and attitude concepts that can be used to better understand AI-specific adoption behaviors. The COM-B model is particularly suited to AI, as it considers psychological capabilities, physical and social opportunities, and motivation, all of which drive user behavior. In the AI context, capabilities may include AI training, physical opportunities may refer to workplace infrastructure, and social opportunities might include an innovative work environment.

Studies have shown that psychological capability, physical and social opportunities, and motivation are key factors influencing behavior change (Michie et al., 2014), all of which are critical to AI adoption. This research seeks to explore how the COM-B frameworks can be applied to identify key domains and issues for improving AI adoption in professional decision- making.

Leveraging PI Breytenbach's expertise in Information Systems and Co-PI Patterson's specialization in implementation science, the project aims to identify barriers and facilitators related to AI adoption, which can then be used to inform and evidence-based implementation strategies to facilitate the uptake of AI decision tools both in clinical environment in addition to other industry sectors within South Africa.

Previous Work and Expertise

Our work leverages the complementary expertise of PI Johan Breytenbach, with a background in Information Systems (IS), and Co-PI Mark Patterson, who specializes in implementation science and project evaluation, which together enable us to use behavioral and evaluation frameworks to identify factors related to AI adoption.

• Breytenbach: Our work builds on Dr. Breytenbach’s recent research, which involved validating various machine learning algorithms by calculating their ROC curves for a tool that predicts post-operative complications. This tool is intended for use by surgeons in South Africa. Through discussions on my training approach, we identified the critical need to improve adoption rates and pinpointed the factors and barriers affecting implementation within clinics.

• Patterson: My background in implementation science uniquely positions me to address and investigate the barriers surgeons face when adopting AI tools. Building on my previous work assessing barriers and facilitators for integrating community health workers into vaccination education within community pharmacies, I can apply similar methods—such as surveys and assessment tools—to promote AI tool adoption among surgeons.

  Goal of the Project

  Our research project aims to bridge the gap between Health IT and Implementation Science by collaboratively developing Health IT implementation strategies that integrate concepts from Information Systems, Computer Science, Artificial Intelligence systems, and Implementation Science.

  Specific Aims

• Identify shared and distinct barriers, facilitators, and implementation strategies influencing the successful adoption of diverse health IT tools across different settings and purposes by comparing Implementation Science studies conducted at UMKC and UWC.

• Foster collaboration between UMKC and UWC, working collaboratively to share knowledge and disseminate findings, establishing a connection between the fields of health IT and implementation science.

  Rationale

  By combining interdisciplinary expertise in qualitative analysis, health IT tool design, and technology adoption in healthcare, we can leverage implementation science methods to identify barriers and facilitators to AI adoption for decision-making. Additionally, we can recommend strategies to enhance AI adoption across various job sectors. Utilizing implementation science frameworks such as TICD and COM-B, we will develop a systematic approach to pinpoint organizational and individual-level barriers, informing strategies for interventions that improve AI adoption rates.

  OutcomesPreliminary Research Completed Prior to Trip

• Implementation Framework Selection: In order to identify determinants (e.g., barriers and facilitators) to AI tool adoption, we initially selected a framework called the Tailored Implementation for Chronic Diseases (TICD) framework to focus on organizational and individual factors affecting AI tool use.

  Figure 1: TICD Domains1

  Domain	Definition
  Individual Health Professional Factors	Characteristics of the healthcare providers, including knowledge, skills, and attitudes, influencing implementation.
  Patient Factors	Attributes of patients that impact implementation, such as preferences, resources, and beliefs.
  Guideline Factors	Features of the clinical guidelines themselves, including clarity, relevance, and feasibility.
  Incentives and Resources	External incentives and material resources that facilitate or hinder implementation.
  Capacity for Organizational Change	The organization's readiness and ability to implement changes, including leadership and culture.
  Social, Political, and Legal Factors	External factors such as policies, regulations, and societal attitudes affecting implementation.
  Health Professional Interactions	The dynamics and communication between health professionals and their teams.

• Literature Review: To understand the organizational, societal, and individual factors influencing AI tool use for decision-making in South Africa, research assistants conducted systematic literature reviews using TICD domains. This identified multi-level adoption factors and provided the foundation for developing an evaluation survey

• Adoption of the COM-B Model: Further research revealed that barriers to AI adoption were primarily due to end-users' beliefs, perceptions, and fears rather than infrastructure or organizational factors. Consequently, we prioritized attitudes and perceptions as key domains. Traditional health IT models like TAM did not adequately address these areas. We transitioned to the COM-B (Capability, Opportunity, Motivation - Behavior) model to better understand individual behaviors influencing AI adoption, including attitudes, skills, perceptions, fears, and workflow burdens. Utilizing the TICD framework as a foundation, we mapped broader issues to COM-B domains, leading to the development of a survey for AI stakeholders.

1 Flottorp, S. A., Oxman, A. D., Krause, J., Musila, N. R., Wensing, M., Godycki-Cwirko, M., ... & Eccles, M. P. (2013). A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and implementation research. Implementation Science, 8(1), 35.

Figure 2: COM-B Model2

Activities During Visit Exchange

Dr. Patterson participated in an exchange visit, staying in Cape Town, South Africa from August 6th to August 21st during which he worked together with PI Breytenbach on the following activities and deliverables:

• Seminar Delivery: Dr. Patterson presented a seminar titled “Attitudes Toward AI and Implementation Science Approaches for Evaluation” to UWC faculty and postgraduate students. The seminar highlighted the relevance of the COM-B model over traditional IT evaluation models like TAM, emphasizing the significance of measuring individual’s capabilities, opportunities, and motivation driving AI adoption behavior.

• Panel Discussion Participation: Participated in a panel with a university task force developing the South African AI Maturity Framework, which includes metrics to track SA’s ability to design, implement, and maintain robust IT systems. Over 70 preliminary domains for the framework overlapped with our proposed COM-B survey domains, allowing us to contribute based on our work. Insights from the panel reinforced our approach, affirming that our COM-B survey measures concepts aligned with existing AI maturity framework research.

• Survey Preparation: Finalized and expanded the survey tool to a broader AI stakeholder audience, incorporating feedback and ensuring alignment with the COM-B model.

• Stakeholder Engagement: Established contacts with local healthcare providers and secured a pathway to disseminate the survey to over 400 stakeholders via the South African Association for Institutional Research (SAAIR) research network.

• Research Paper Initiation: Began drafting a methods paper highlighting the unique application of the COM-B model in studying AI adoption, with a preliminary draft submitted to the journal Global Implementation Research and Applications (GIRA) as part of a special invitation issue focused on implementing interventions in practice.

2 Michie, S., van Stralen, M. M., & West, R. (2011). The behavior change wheel: A new method for characterizing and designing behavior change interventions. Implementation Science, 6(1), 42.

Post-visit deliverables and activities

• Survey Deployment: Deployed the survey in mid-October using Qualtrics, collecting preliminary results from stakeholders in the education sector through the SAAIR network

  Figure 3: COM-B Survey Questions with Associated COM-B Domains

  Question	COM-B Domain
  I am concerned about relying too much on AI tools for my professional decisions	Automatic and Reflective Motivation
  I have adequate skills to run AI tools in my industry	Capability
  When using AI tools in my industry, I understand and am confident in the results and/or output	Capability & Reflective Motivation
  I am prepared to address my clients' ethical issues regarding AI tools in my decision-making	Capability & Social Opportunity
  My workplace has adequate technological infrastructure to effectively use AI tools	Physical Opportunity
  Our current computer systems easily integrate AI tools	Physical Opportunity
  My workplace has adequate support systems to effectively implement AI tools	Physical Opportunity
  Our AI tools comply with regulation and privacy laws	Physical Opportunity
  I can integrate AI tools into my job tasks with minimal effort and time	Physical Opportunity
  The costs of infrastructure and resources limit our ability to use AI tools	Physical Opportunity
  AI tools benefit our clients	Reflective Motivation
  Our AI tools are accurate enough to inform our professional decisions	Reflective Motivation
  The precision of AI tools impacts my willingness to use AI tools for professional decisions	Reflective Motivation
  We are concerned that clients may have ethical concerns about our use of AI tools in decision-making	Reflective Motivation & Social Opportunity
  Integrating AI tools aligns with our industry’s best practices and standards	Social Opportunity
  Our workplace culture supports and rewards innovation	Social Opportunity

• Manuscript Preparation: Continued developing the methods paper, detailing the survey development process and initial findings. Submit to GIRA

  Figure 4: Abstract submitted to GIRA. October 31, 2024

  Title: Enhancing AI Adoption: A Pilot Survey on Capability, Opportunity, and Motivation in South African Universities

  Patterson, ME, Coffman I, Breytenbach J

• BACKGROUND: Artificial Intelligence (AI) drives innovation but faces challenges like limited expertise, resources and legal or ethical concerns. Identifying the factors influencing AI adoption is key to developing effective implementation strategies. The Capability- Opportunity-Motivation Behavior (COM-B) model provides a framework for understanding AI adoption. This study developed and piloted a COM-B-based survey among South African university educators to assess adoption attitudes and evaluate its broader feasibility.
• METHODS: A COM-B-based survey was designed to assess AI adoption attitudes amongst a broad diverse sector of AI stakeholders in South Africa. Barriers and facilitators were identified through a literature review and mapped to the COM-B domains. A pilot survey was administered electronically to South African university educators to gauge feasibility. The results were analysed using medians, interquartile ranges (IQRs), and the Coefficient of Quartile Variation (CQV). High median scores with CQVs < 0.25 indicated facilitators, while low scores indicated barriers. Items with mid-range CQVs were considered mixed indicators.
• RESULTS: Median scores from 33 respondents ranged from 5 to 7.5, and CQVs from 0.17 and 0.73. Respondents strongly agreed on adequate AI infrastructure (median=7.5; CQV=0.17). AI tool accuracy received mixed opinions (median=5.0; CQV=0.40), indicating neutral to moderate agreement. Support systems and regulatory compliance (median=5; CQV=0.6) showed moderate disagreement.

• CONCLUSIONS: Although respondents show confidence in AI infrastructure, neutral and varying scores on AI accuracy may reduce adoption motivation according to the COM-B model. This suggests a need for training sessions and resource allocation to improve adoption rates. Mixed views on support systems and regulatory compliance indicate potential to enhance workplace environments that shape the user's context—such as implementing financial incentive reward systems. Further research is necessary to expand the tool to other industries beyond education, explore workplace innovation factors, and align strategies and policies to support AI adoption.

  Figure 5: Preliminary finding from survey respondents (N=32)

  Question	Min	Max	Median	IQR	CQV	Variability
  I am concerned about relying too much on AI tools for my professional decisions	1	10	7	5	0.71	medium
  I have adequate skills to run AI tools in my industry	1	10	6	3	0.5	high
  When using AI tools in my industry, I understand and am confident in the results and/or output	1	10	6	3.25	0.54	medium
  I am prepared to address my clients' ethical issues regarding AI tools in my decision-making	2	10	7	2.25	0.32	medium
  My workplace has adequate technological infrastructure to effectively use AI tools	1	10	7.5	1.25	0.17	low
  Our current computer systems easily integrate AI tools	2	10	7	3	0.43	medium
  My workplace has adequate support systems to effectively implement AI tools	1	9	5	3	0.6	high
  Our AI tools comply with regulation and privacy laws	1	10	5	3	0.6	high
  I can integrate AI tools into my job tasks with minimal effort and time	1	10	6	4	0.67	medium
  The costs of infrastructure and resources limit our ability to use AI tools	2	10	6	4.25	0.71	high
  AI tools benefit our clients	2	10	7	2	0.29	medium
  Our AI tools are accurate enough to inform our professional decisions	1	9	5	2	0.4	medium
  The precision of AI tools impacts my willingness to use AI tools for professional decisions	2	10	7	3	0.43	medium
  We are concerned that clients may have ethical concerns about our use of AI tools in decision-making	2	10	7	2	0.29	high
  Integrating AI tools aligns with our industry’s best practices and standards	3	10	6.5	3	0.46	medium
  Our workplace culture supports and rewards innovation	2	10	5.5	4	0.73	high

  Future Directions

  Short-term Goals

  • Manuscript submission to JHIR: Plan to submit the AI attitudes findings to the Journal of Health Informatics Research (JHIR) - Human Factors, following a rejection from the GIRA thematic issue. The focus of the new submission will be on the methods used to develop the COM-B survey, along with the preliminary results from the subgroup of education stakeholders. The emphasis will be on insights gained from the education subgroup

  • Algorithm Paper Submission: Complete and submit a paper focused on calculating the ROC of various machine learning algorithms used to predict post-operative complications.

    Long-term Goals

  • Survey Development and Refinement: Validate and potentially copyright the COM-B- based survey instrument by assessing its psychometric properties.
  • Expanded Deployment: Deploy the survey to a larger, more diverse cohort across various sectors and countries. Conduct subgroup analyses to evaluate differing attitudes toward AI adoption.
  • Behavior Change Wheel Integration: Utilize the Behavior Change Wheel to map COM-B domains to evidence-based intervention strategies, informing policies aimed at improving AI adoption.
  • Collaborative Network Building: Establish partnerships with other researchers to utilize the survey tools and data repositories, enhancing understanding of the factors driving AI adoption and decision-making in the workplace.

Conclusion

This collaborative project successfully initiated the development of a standardized approach to understanding and enhancing AI adoption in healthcare settings using implementation science frameworks. By emphasizing individual behaviors through the COM-B model and engaging with stakeholders, the project establishes a solid foundation for future research and practical interventions. The ultimate objective is to improve Health IT integration, enhancing healthcare delivery in South Africa and potentially in other culturally diverse contexts.