Validation of Self-Efficacy in Pediatric ECHO Training
Validation and reliability of the self-efficacy scale to assess the competence of pediatricians in ECHO-Pediatrics course
Le Hong Nhung¹*, Nguyen Phuong Lan¹, Piter Martinez Benitez², Pham Van Hoc², Pham Duc Han², Matthew F Bouchonville³
- Vietnam National Children’s Hospital, Hanoi, Vietnam.
- General Hung Vuong Hospital, Phu Tho, Vietnam.
- University of New Mexico Health Sciences Center, Albuquerque, New Mexico, US.
OPEN ACCESS
PUBLISHED: 31 October 2024
CITATION: Nhung, LH., Lan, NP., et al., 2024. “Validation and reliability of the self-efficacy scale to assess the competence of pediatricians in ECHO-Pediatrics course”. Medical Research Archives, [online] 12(10).
https://doi.org/10.18103/mra.v12i10.5771
COPYRIGHT: © 2024 European Society of Medicine. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
DOI https://doi.org/10.18103/mra.v12i10.5771
ISSN 2375-1924
ABSTRACT
Background: Self-assessment of competency in CME/CPD has been fully approved by the American Medical Association (AMA) in its program of retention of practice certificates. To be accepted and applicable in the CME/CPD course in Vietnam, the scale of self-efficacy or self-assessment of competency requires modification. It is helpful for the future studies related to the ECHO courses that the scale of self-efficacy or self-assessment of competency are applied with its respect to validity and reliability.
Aims: In the current study, we aimed to introduce the theoretical framework for developing a set of scales to facilitate self-assessment of professional capacity among PHWs participating in project ECHO. Additionally, we proposed a viable method for creating a self-efficacy scale applicable to the continuing medical training programs. We hypothesized that the scale of efficacy would integrate into the research and evaluation procedure at VNCH.
Methods: We applied a cross-sectional study design, implementing mixed methods including quantitative and qualitative in order to adjust the scale of self-efficacy to assess the professional capacity improvement of the healthcare workers to ensure the validity and reliability of the scale. The study proceeds in the following steps:
- Reviewing the sets of scales for self-assessment of competency (self-efficacy) according to the ECHO model in the world;
- Collecting the decisions from the panel of experts to determine the content of primary scale to improve professional capacity through continuing medical training;
- Test the scale on a sample of learners participating in a continuing medical training program using the ECHO model;
- Evaluate the surface validity, the content of validity including the convergent value and discriminant value and the structural reliability with Cronbach’s Alpha internal consistency index.
Results: The 22-item self-assessment of competency scale was developed based on the results of qualitative and quantitative research, and is considered to be a reliable scale applicable to further studies on the ECHO model in Vietnam.
Conclusion: The 22-item self-assessment of competency scale was developed based on the results of qualitative and quantitative research, and is considered to be a reliable scale applicable to further studies on the ECHO model in Vietnam.
Keywords: self-efficacy, CME, Delphi method, reliability, validity.
Introduction
Extension for Community Healthcare Outcomes (ECHO) is a platform for academic health centers to expand healthcare worker capacity in underserved populations to meet the need of medical healthcare service¹. This program uses the model of “hub and spoke” through a video conferencing platform to connect experts at Vietnam National Children’s Hospital (VNCH, the hub) with the healthcare workers at different provincial hospital (spokes)². The ECHO model has been developed and replicated in many different countries to address health problems and treat common and complex medical conditions, such as HIV/AIDS, substance use disorders, palliative care, rheumatism, chronic diseases and other diseases³⁻⁹. During the implementation process, the ECHO model is regarded as a continuing medical education to improve the professional capacity and professional collaboration of health workers in practicum.
Self-efficacy or self-assessment of competency is a concept that has shaped medical training since the 1990s, demonstrating the desire of healthcare workers to make self-determination and selfregulation based on the realities faced in patient care. This desire depends on the readiness of healthcare workers to self-assess their knowledge, skill and practice based on patient outcomes¹⁰.
Self-assessment of competency in continuing medical education has been fully approved by the American Medical Association (AMA) in its program of retention of practice certificates¹¹. To evaluate clinician competence, confidence and professional collaboration in clinical practice in the workplace are significant goals in studies assessing the effectiveness of the ECHO model according to Moore’s method¹²⁻¹⁵. Results from scale development inform future studies related to the ECHO model with respect to validity and reliability.
The first ECHO program, developed by Dr. Arora at the University of New Mexico for hepatitis C, used the self-assessment of competency scale named “self-efficacy”, laying the foundation for the construction of the scale later in the authors’ studies on different topics¹⁶. The theoretical basis of the self-efficacy scale included the combination of social cognitive theories of Bandura¹⁷, Vygotsky’s situation-based learning theory¹⁸ and the practice community of Lave¹⁹. The term of “self-efficacy” refers to “self-confidence or the perception that one has the ability to organize and take the actions necessary to succeed when given a task.” (Bandura, 1997)²⁰. Self-efficacy is based on three sources of information: an individual’s previous experience, observational experiences in practice, and verbal persuasion on the psychological status of the learner²¹.
Later, many ECHO programs were developed, but all were based on these three theories. A number of studies have shown that synergies in learning, coaching and mentoring by experts, among colleagues, have formed “learning loops”²². Self-efficacy was assessed at three stages: before training, after training and 3–6 months following completion of training²³. Later studies employed the concept of “self-assessment of competence” developed for the ECHO hypertension program²⁴.
When studying ECHO programs from the perspective of continuing medical education (CME), researchers found that the scale of “self-efficacy” or “self-assessment of competence” had also been developed and applied in different aspects in the implementation process. These included assessing participants’ satisfaction with implementing best practices in the workplace, known as the scale of “job satisfaction” or “professional satisfaction.”
In the current study, we aimed to introduce the theoretical framework for developing a set of scales to facilitate self-assessment of professional capacity among healthcare workers participating in project ECHO. Additionally, we proposed a viable method for creating a self-efficacy scale applicable to the continuing medical training program. We hypothesized that the scale of efficacy would be applied to the research and evaluation procedure at VNCH.
Methods
STUDY SETTING
The study was conducted at the VNCH and its satellite hospitals from January to December 2020. Recruitment of participants occurred from July to September 2020. The study proposal was approved by the research ethics committees at the Hanoi University of Public Health (261/2020/YTCCHD3) and the VNCH (883/BVNTW-VNCSKKTE). VNCH was one of the leading pediatric hospitals in Vietnam, located in the North of Vietnam. The hospital played the role as a center of pediatric healthcare directives and provided the Continuing Medical Education (CME) programs of pediatrics for the province-level hospitals. The administrative function of the hospital included updating professional referrals and treatment guidelines for the province-level hospitals, organizing CME courses to update knowledge, and supporting expertise resources in difficult cases of diseases at the local.
Project ECHO for pediatricians was launched in January 2020 to improve the capacity of physicians at province-level hospitals, following which the program recruited participants from July to September 2020 and organized online courses after the recruitment period. The anatomy of each session comprised two parts: the theoretical part (didactics) and the practical part (case study discussion). The online course included 8 to 10 sessions depending on the specific course, in which each session lasted two hours with 30 minutes for theoretical presentation and the remaining 90 minutes for discussing a patient case. A typical online course was conducted twice per week for 3–5 weeks. In this study, we only focused on learners who participated in the ECHO Immunology course in 2020. The hub was VNCH and the spokes were learners from 18 province-level hospitals specializing in pediatric faculty to participate in the ECHO course. The learners were sponsored by the national grant to pay fees during the period of studying.
STUDY DESIGN AND SAMPLING
Applying a mixed method of qualitative and quantitative research, the study was performed through a three-round Delphi iterative consultation process with experts²⁵. This technique is widely utilized in the research context²⁶ and its validity for questionnaire development has already been described²⁷. Furthermore, the final questionnaire was tested in the field, then validated following the Campbell & Fiske method with constructive validity and inconsistent reliability²⁸.
PARTICIPANTS
The panel of expert was made up of six researchers from six units including Hanoi Medical University, Hanoi University of Public Health, and the Agency of Science, Technology and Training, at Ministry of Health. Eligible participants were academic training managers or professors in the areas of both teaching and research relating to the study topic. Intentional sampling²⁹ was used for those panelists who complied with the aforementioned criteria, who were contacted by e-mail. The procedure below began to implement after their confirmation of participation.
PROCEDURE
The construction of the scale “self-assessment of the physician’s professional competence” took place in three distinct stages: Qualitative study with Delphi method (included preliminary stage and exploratory stage):
- Preliminary stage: Set up the focus group discussion by research team to find a consensus on the content of the initial scale.
- Exploratory stage: Invited six experts on the CME implementation as panelists to comment on the revised scale through the roundtable discussion.
- The research team called or met in person, wrote emails to discuss with the experts about the purpose of collecting the expert’s opinion, and instructed the experts to focus on the content of items in the scale (Appendix 1).
- Collected comments from six experts and continued to discuss with the research team, revised the items of scale according to the expert’s comments in each round. The submission was done in 3 rounds.
- Collected the expert’s opinion based on Delphi method following the issues:
▪ Experts commented on whether the items were appropriate to measure the “self-assessment of competency” before and after participating the virtual training following the ECHO model.
▪ Experts made any possible suggestions with adding or removing items or rewording the items on the scale.
▪ Experts evaluated the instruction of scale.
▪ Experts evaluated the format of the scale.
Inclusion Criteria:
- Items with a consensus rate (surface validity) above 0.78³⁰.
Exclusion Criteria:
- Items with a consensus rate lower than 0.5.
- Items with a consensus rate from 0.5–0.78 continued to be evaluated in the following round after revision based on the experts’ suggestion.
Final stage: Quantitative study to test the adjusted scale in the field with Campbell & Fiske method
- Chose the convenience sample with 38 learners who joined in the ECHO course at VNCH.
- Invited learners to participate in the study with the answers to the questions of the scale, collected feedbacks from learners who were evaluated by the data.
- Calculated the internal consistency reliability Cronbach’s Alpha, the convergent value and the discriminant value based on the learner’s feedback.
STUDY TOOLS AND DATA COLLECTION
The initial scale after being translated into Vietnamese then being revised based on focus group’s discussions that included 28 items naming “self-assessment of the physician’s professional competence” (Appendix 1), divided into 4 domains including:
- Section 1: self-assessment of competence in knowledge and professional practice before training with six items;
- Section 2: self-assessment of competence in knowledge and professional practice after training with six items;
- Section 3: self-assessment of professional satisfaction in clinical practice before training with eight items;
- Section 4: self-assessment of professional satisfaction in clinical practice after training with eight items.
The final adjusted scale (Table 6) was developed after three rounds of expert voting and was adjusted based on their comments. The experts were requested to rate from 1 to 4 on a Likert scale with level 1: Very unsuitable; level 2: Not suitable; level 3: Suitable; level 4: Very suitable.
DATA COLLECTION AND ANALYSIS
- Data collection on the expert opinions was carried out three times, counting on the consensus rate (face validity) and the median value in each rounds to select subitems in the scale. All data were analyzed in 2020.
- The revised scale with 28 items was sent to the experts for rating from 1 to 4 point to show the degree of approval. With the point 3 and 4 obtained, collecting the experts’ opinions continue until the face validity reached above 0.78, considered high approval.
- The final adjusted scale with high approval would be tested for reliability using Cronbach’s Alpha internal consistency index and validated following the Campbell & Fiske method.
Results
PRELIMINARY STAGE RESULT: Based on the literature review and the focus group discussion, we removed unsuitable items/subitems, leaving the initial scale with 28 items (Appendix 1).
The revised scale was sent to six experts for judgement process.
Exploratory stage result on the assessment of six experts through roundtable discussions
Table 1: the first roundtable discussion result
Section I, II: Scale evaluate the self-assessment of competency in knowledge, professional practice following Likert 5 BEFORE and AFTER training
(Learner tick the point from 1 to 5 under each question 1:Very poor;2:Poor; 3: Average;4: Good; 5: Excellent)
| Item | Median (Q1, Q3) | Consensus rate (%) |
|---|---|---|
| Sentence 1: Have the ability to detect symptoms of patients who need to be screened | 3.0 (2.75;3.0) | 83.3% |
| Sentence 2: Have the ability to detect patients suitable for treatment | 3.0 (2.0;3.0) | 66.7% |
| Sentence 3: Have the ability to assess the extent of damage done to relatable organs in patients | 2.5 (1.75;3.00) | 50% |
| Sentence 4: Have the ability to treat the patients and manage one’s side effects | 3.0 (2.0;3.0) | 66.7% |
| Sentence 5: Have the ability to educate and motivate the patients | 2.5 (2.0;3.0) | 50% |
| Sentence 6: Have the ability as a consultant in my clinic as well as locally regarding medical problems | 2.5 (2.0;3.0) | 50% |
Section III, IV: Scale evaluate the self-efficacy of job satisfaction following Likert 5 BEFORE and AFTER training
Sentence 1: I feel isolated when working professionally at my workplace | 2.5 (2.0;3.0) | 50%
Sentence 2: I could easily form close relationships with my coworkers | 2.5 (1.75–3.0) | 50%
Sentence 3: I could easily access a doctor when I needed their professional feedback or their help | 3.0 (2.0;3.0) | 66.7%
Sentence 4: I could easily access all sources and documents for my career development (improvement in my knowledge) | 2.0 (1.0;2.25) | 16.7%
Sentence 5: When I needed the clinical doctors’ help or support, I was able to contact the experts in due time | 3.0 (2.0;3.0) | 66.7%
Sentence 6: I was able to have the opportunity to share my clinical experiences with my co-workers at regular intervals | 3.0 (2.0;3.0) | 66.7%
Sentence 7: I’m generally satisfied with my job | 3.0 (2.0;3.0) | 66.7%
Sentence 8: I’m confident that I could improve the quality of medical service delivery in my facility | 3.0 (2.75;3.0) | 83.3%
In the first round, sentence 5 of section I and section II was revised: “Ability to advise and educate patients on self-care and disease prevention”.
Table 2: the second roundtable discussion result
Section I, II: Scale evaluate the self-assessment of competency in knowledge, professional practice following Likert 5 BEFORE and AFTER training
Sentence 1: Ability to detect symptoms of patients who need to be screened | 3.00 (3.00;3.00) | 100%
Sentence 2: Ability to analyze and synthesize clinical and subclinical manifestation to make an appropriate diagnosis | 3.00 (2.75–4.00) | 83.3%
Sentence 3: Ability to apply standardized treatment protocol and control side effects when prescribing | 3.50 (2.75–4.00) | 83.3%
Sentence 4: Ability to handle emergency situations according to standard protocols | 3.00 (2.75;3.25) | 83.3%
Sentence 5: Ability to advise and educate patients on self-care and disease prevention | 3.50 (2.75–4.00) | 83.3%
Sentence 6: Ability to manage complications of the disease | 3.00 (2.00;3.00) | 66.7%
Section III, IV: Scale evaluate the self-efficacy of job satisfaction following Likert 5 BEFORE and AFTER training
Sentence 1: Ability to resolves independently professional issues at the workplace | 3.00 (2.75;3.25) | 83.3%
Sentence 2: Ability to exchange clinical experiences among colleagues | 3.00 (2.75;3.00) | 83.3%
Sentence 3: Ability to get experts’ support to solve illness cases at the workplace | 3.00 (2.75;3.25) | 83.3%
Sentence 4: Ability to finish the assigned work according to one’s expertise in the workplace | 3.00 (2.75–4.00) | 83.3%
Sentence 5: Ability to improve the quality of the treatment and examination services according to one’s professional department in the workplace | 3.00 (2.75–4.00) | 83.3%
In the second round, two additional items were added and continued to be reviewed for approval by the experts. The result of the second round included 22 items with six couples of items regarding confidence in professional knowledge and practical expertise before and after ECHO model training and five couples of items regarding one’s ability to coordinate professionally in clinical practices.
Table 3: the third roundtable discussion
Section I: Scale evaluate the self-assment of competency in knowledge, professional practice following Likert 5 BEFORE and AFTER training
(Learners tick the given score from 1 to 5 beneath each question. 1: Very poor; 2: Poor; 3: Average; 4: Good; 5: Excellent (Students can skip this section if they deem it unsuitable))
| Item | Median (Q1, Q3) | Consensus rate (%) |
|---|---|---|
| Sentence 1: Ability to examine and detect clinical symptoms | 3.0 (3.0;3.0) | 100% |
| Sentence 2: Ability to analyze and synthesize the clinical and paraclinical characteristics to present the most appropriate diagnosis | 3.00 (2.75;4.00) | 83.3% |
| Sentence 3: Ability to apply treatments per standard protocols and control the side effects when prescribing | 3.5 (2.75;4.0) | 83.3% |
| Sentence 4: Ability to handle emergencies according to standard protocols | 3 (2.75;3.25) | 83.3% |
| Sentence 5: Ability to advise and educate patients on self-care and disease prevention following standard protocols | 3.5 (3.0;4.0) | 100% |
| Sentence 6: Ability to manage the risks involved, detect and smartly solve the complications of the illness | 3 (2.75;3.25) | 83.3% |
Section II: Scale evaluate the self-efficacy of job satisfaction following Likert 5 BEFORE and AFTER training
(Students tick the given score from 1 to 5 beneath each question. 1: Very poor; 2: Poor; 3: Average; 4: Good; 5: Excellent (Students can skip this section if they deem it unsuitable))
| Item | Median (Q1, Q3) | Consensus rate (%) |
|---|---|---|
| Sentence 1: Ability to resolves independently the professional issues at the workplace | 3.00 (2.75;4.00) | 100% |
| Sentence 2: Ability to exchange experiences in examination skills, diagnosis skills, and treatment skills among co-workers at the workplace | 3.00 (3.00;3.25) | 100% |
| Sentence 3: Ability to get senior experts’ support to solve illness cases at the workplace | 3.00 (2.75;3.25) | 83.3% |
| Sentence 4: Ability to finish the assigned work according to one’s expertise in the workplace | 3.00 (2.75;4.00) | 83.3% |
| Sentence 5: Ability to improve the quality of the treatment and examination services according to one’s professional department in the workplace | 3.0 (2.75;4.00) | 83.3% |
As the result of the third round, sentence 6 in section I and II was revised:
“Ability to manage the risks involved, detect and smartly solve the complications of the illness”.
Table 4: The final adjusted scale
Section I: Scale evaluate the self-assessment of competency in knowledge, professional practice following Likert 5.
(Students tick the given score from 1 to 5 beneath each question. 1: Very poor; 2: Poor; 3: Average; 4: Good; 5: Excellent (Students can skip this section if they deem it unsuitable))
BEFORE training | AFTER training
Sentence 1: Ability to examine and detect clinical symptoms.
Sentence 2: Ability to analyze and synthesize the clinical and paraclinical characteristics to present the most appropriate diagnosis.
Sentence 3: Ability to apply treatments per standard protocols and control the side effects when prescribing.
Sentence 4: Ability to handle emergencies according to standard protocol(s).
Sentence 5: Ability to manage the risks involved, detect and solve the complications of the illness.
Sentence 6: Ability to advise and educate patients on self-care and disease prevention following standard protocol(s).
Section II: Scale evaluate the self-efficacy of job satisfaction following Likert 5
(Students tick the given score from 1 to 5 beneath each question. 1: Very poor; 2: Poor; 3: Average; 4: Good; 5: Excellent (Students can skip this section if they deem it unsuitable))
BEFORE training | AFTER training
Sentence 1: Ability to independently resolves professional issues at the workplace.
Sentence 2: Ability to exchange experiences in examination skills, diagnosis skills, and treatment skills among co-workers at the workplace.
Sentence 3: Ability to get senior experts’ support to solve illness cases at the workplace.
Sentence 4: Ability to finish the assigned work according to one’s expertise in the workplace.
Sentence 5: Ability to improve the quality of the treatment and examination services according to one’s professional department in the workplace.
Finally, the adjusted scale of self-assessment of competence measured the change of a physician’s knowledge and professional practice before and after ECHO model training, which consisted of four categories:
Section I: “Scale evaluate the self-assessment of competency in knowledge, professional practice following Likert 5 BEFORE training” consisted of six items.
Section II: “Scale evaluate the self-assessment of competency in knowledge, professional practice following Likert 5 AFTER training” consisted of six items.
Section III: “Scale evaluate the self-efficacy of job satisfaction following Likert 5 BEFORE training” consisted of five items.
Section IV: “Scale evaluate the self-efficacy of job satisfaction following Likert 5 AFTER training” consisted of five items.
According to Lawshe’s suggestion, consensus rate at 0.78 and above were considered satisfactory³⁰. The research team had already carefully examined the final scale based on the three rounds of experts’s judgements.
Final Stage. Quantitative study to test the adjusted scale in the field with Campbell & Fiske method.
Table 5: Characteristics of item statistics
| Scale | Items | Missing value (%) | Mean interval | Standard deviation Interval (SD) |
|---|---|---|---|---|
| Self-assessment of competency in knowledge, professional practice before training | 6 | 0 | 2.18–2.37 | 0.68–0.75 |
| Self-assessment of competency in knowledge, professional practice after training | 6 | 0 | 3.55–3.68 | 0.47–0.6 |
| Self-efficacy of job satisfaction before training | 5 | 0 | 2.37–2.55 | 0.82–0.47 |
| Self-efficacy of job satisfaction after training | 5 | 0 | 3.45–3.63 | 0.49–0.56 |
The mean score and standard deviations of the items in each domain before training ranged from 2.18±0.68 to 2.55±0.47; the mean score and standard deviation of the items in each domain after training ranged from 3.45±0.49 to 3.68±0.6.
The construct validity including convergent and discriminant validity
Table 6: results of multi-point matrix correlation
| Scale | Convergent validity | Discriminant validity |
|---|---|---|
| Self-assessment of competency in knowledge, professional practice before training | 0.68–0.94 | 0.25–0.75 |
| Self-assessment of competency in knowledge, professional practice after training | 0.42–0.75 | 0.23–0.73 |
| Self-efficacy of job satisfaction before training | 0.55–0.88 | 0.23–0.73 |
| Self-efficacy of job satisfaction after training | 0.33–0.82 | 0.25–0.75 |
The construct validity was evaluated by calculating the item’s convergent with the value fluctuate from 0.33–0.94 and discriminant validity from 0.23–0.75.
Internal consistency reliability
Table 7: Cronbach Alpha reliability
| Scale | Inter-scale correlation | Cronbach Alpha |
|---|---|---|
| Self-assessment of competency in knowledge, professional practice before training | 1 | 0.829 |
| Self-assessment of competency in knowledge, professional practice after training | 0.744 | 0.818 |
| Self-efficacy of job satisfaction before training | 0.789 | 0.84 |
| Self-efficacy of job satisfaction after training | 0.764 / 0.84 / 0.813 / 1 | 0.875 |
Internal consistency reliability was tested by Cronbach’s Alpha coefficient for each domain in the range of 0.72–0.87.
Discussion
Sanjeev Arora is well-known for his work on the Project ECHO model, which focuses on expanding access to specialty care in underserved areas through telemedicine and collaborative learning. While his primary work has been on the ECHO model, which emphasizes knowledge sharing and mentorship, self-efficacy has been a key concept in evaluating the outcomes of such programs.
In particular, self-efficacy scales have been used to assess the impact of the ECHO model on healthcare providers’ confidence and abilities. Although Arora himself may not have explicitly developed or published studies solely focused on self-efficacy scales, his research and the evaluations of the ECHO model often incorporate self-efficacy as a measure of success³¹⁻³².
Self-efficacy, a concept developed by psychologist Albert Bandura, refers to an individual’s belief in their ability to successfully execute tasks or achieve goals. In the context of Continuing Medical Education (CME), applying a self-efficacy scale can help assess and improve healthcare professionals’ confidence in their abilities related to their practice³³⁻³⁴.
The theoretical foundation of applying the “self-efficacy” assessment method in the ECHO model has been built on 3 learning theories: Bandura’s social cognition¹⁷, Lave’s situation-based learning theory¹⁹, and Vygotsky’s learning community theory¹⁸. In particular, Bandura’s Social Cognitive Theory is key, which describes three key factors that increase a person’s confidence to change their behavior to help them actively assess their own abilities through participation in the learning process that the ECHO model uses. It is called the “All teach all learn” method, exchanging experiences through cases that are accumulated gradually and regularly.
The “self-efficacy” scale using in the ECHO-Pediatrics program in Vietnam was developed from the research related to the ECHO-Hepatitis C model by S Arora³⁵. In addition to self-assess the capacity improvement of healthcare providers, there has been a new point of the scale that evaluate the healthcare provider’s ability to meet the job satisfaction at workplace after being educated and how they concretize the ability to coordinate professionally with the team at their workplace.
This research used the Delphi method to adjust the “self-efficacy” scale using in the ECHO-Pediatrics with face validity. This method was first introduced in a RAND report published in 1953, which outlined its use for obtaining expert opinions and reaching consensus on complex issues, reflecting the method’s aim to provide a structured process for predicting future events and gaining insights through expert judgment. The Delphi method can indirectly address face validity by refining instruments, questionnaires, or assessment tools through expert feedback³⁵.
Face validity is a form of validity that is concerned with the appearance or subjective judgment of the measure’s relevance and appropriateness, thereby leading to more accurate and relevant insights³⁶. In this study, in the exploratory stage, the face validity (consensus rate) attained at 0.78 and above were considered satisfactory³⁰. The researcher had already carefully examined the final scale based on the three rounds of experts’ judgements.
In addition to calculating the face validity according to the Delphi method, this study calculated the construct validity including convergent validity and discriminant validity according to the Campbell Fiske method. Convergent validity refers to the degree to which two measures that are supposed to be related are actually related. In other words, if two different methods are used to measure the same construct, they should show high correlations with each other. Discriminant validity (or divergent validity) refers to the degree to which a measure is not related to measures of different constructs. It’s essentially the opposite of convergent validity. If a measure is supposed to assess a specific construct, it should not be strongly related to measures of other, distinct constructs²⁸.
The final scale (22 items) after being validated with face validity was distributed by the research team to all learners participating in the online training program following the ECHO-Immunology course at the VNCH. None of the learners were disqualified. The survey was distributed to learners in an online session at the end of the course. The final adjusted scale provided instructions on the objective of the scale, how to respond to the questions of the scale, and ensuring anonymity. The items included demographic characteristics, the name of the program, and the learner’s qualifications for those who participated in the ECHO-Immunology course.
The quality and completeness of the feedback from learners’ response to the scale were assessed. The mean and standard deviation were calculated based on the points in each item category. The calculated percentage for missing items or domains with accepted values were less than 20%. A high number of missing items or a high percentage of missing data throughout the scale indicated that the items were confusing or that the scale layout was problematic³⁷. Spearman’s correlation was used in this study with a non-normal distribution.
THE CONSTRUCT VALIDITY INCLUDING CONVERGENT AND DISCRIMINANT VALIDITY
The construct validity was evaluated by calculating the item’s convergent and discriminant validity. The correlation of each item with its own total score was considered to be satisfactory if it valued > 0.30³⁸. Theoretically, the convergent value in the same item was higher than the correlation value in other items. The discriminant validity of the item assumed that in the tool with more than one domain, the correlations between items in the same domain were expected to be significantly higher than those in other domains. The scale success rate was calculated, as suggested by McHorney et al.³⁹, as the percentage of items in each domain that met the criteria for convergent validity and discriminant validity. Following the matrix model of Campbell & Fiske, the result was interpreted at the diagonal values²⁸,³⁸.
INTERNAL CONSISTENCY RELIABILITY
Internal consistency reliability was tested by Cronbach’s Alpha coefficient for each domain and for the whole scale with an acceptable value of Cronbach Alpha ≥ 0.70, in the range of 0.72–0.87⁴¹. However, a Cronbach Alpha value ≥ 0.60 was considered to be accepted in the newly developed scale⁴². Cronbach’s Alpha was also checked when individual items were deleted. Items that reduced the Cronbach’s Alpha value in the domain were considered for exclusion.
Limitation
The number of experts invited to participate in the quality assessment of the scale were less than previous studies according to the Delphi method⁴³. However, they were the leading experts whose academic qualifications were PhDs and Professors in the divisions of training management at the Department of Science and Technology Training and Research – Ministry of Health, Vietnam Universities of Medicine and Public Health. Therefore, the value of judgements achieved were scientific and useful. A similar study in Vietnam invited five experts to participate in the evaluation of the questionnaire⁴⁴.
Additionally, to calculate the construct validity, there is a similar study in Vietnam using exploratory factor analysis (EFA), which is a widely used statistical analysis method. EFA helps researchers reduce the number of sub-items of a scale by removing sub-items with low relevant values and providing the optimal component structure of the scale, so it is appropriate to apply to a scale with a large number of sub-items⁴⁴.
Conclusion
The scale to assess the physician professional capacity improvement has been adjusted accordingly to each ECHO program in the world since 2007. In our program, we developed “the scale self-assessment of competency” using the Delphi method to evaluate the surface validity (consensus rate), combining with testing the scale in the field in order to calculate the construction validity and reliability following the Campbell & Fiske method. Therefore, this scale was valid and highly reliable that would be widely applied in evaluating the effectiveness of CME courses following the ECHO model in Vietnam.
We recommend that the scale continue to be developed depending on the disease state and customised to each organization’s capacity. In our study, this scale was built for general use in the ECHO courses with different specialties in VNCH, and the medical terminology in the domains were adjusted to fit an ECHO program for CME purpose.
Competing of interests
The authors declare that they have no competing interests.
Financial Disclosure statement
The authors declare that they have no financial interest. There are no funder or granter support for the research.
Acknowledgements:
The authors thank the hospital managers and healthcare providers who participated in this study. The views expressed in this article are solely those of the authors and do not represent the official positions of the organizations the authors are affiliated with.
Ethics approval and consent to participate
The study proposal was approved by research ethics committees at the Hanoi University of Public Health (261/2020/YTCC-HD3) and the Vietnam National Children’s Hospital (883/BVNTW-VNCSKKTE). The study was conducted in accordance with relevant guidelines and regulations. Informed consent was obtained from all participants.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
Authors’ contributions
LHN and NPL developed the study concept and design, wrote the main manuscript text. LHN, PDH, PVH contributed to the data collection and analysis. NPL, PMB contributed to revise and improve the manuscript. MFB contributed his experience with the ECHO model to revise the manuscript. All authors read and approved the final manuscript.
References
1. Faherty LJ, Rose AJ, Chappel A, Taplin C, Martineau M, Fischer SH. Assessing and Expanding the Evidence Base for Project ECHO and ECHO-Like Models: Findings of a Technical Expert Panel. Journal of general internal medicine. Mar 2020;3 5(3):899-902. doi:10.1007/s11606-019-05599-y
2. Arora S, Thornton K, Murata G, et al. Outcomes of treatment for hepatitis C virus infection by primary care providers. The New England journal of medicine. Jun 9 2011;364(23):2199-207. doi:10 .1056/NEJMoa1009370
3. Salvador JG, Bhatt SR, Jacobsohn VC, et al. Feasibility and acceptability of an online ECHO intervention to expand access to medications for treatment of opioid use disorder, psychosocial treatments and supports. Substance Abuse. 2020:1-8.
4. Pollack TM, Nhung VTT, Vinh DTN, et al. Building HIV healthcare worker capacity through telehealth in Vietnam. BMJ global health. 2020;5(4) :e002166.
5. Eaton LH, Godfrey DS, Langford DJ, Rue T, Tauben DJ, Doorenbos AZ. Telementoring for improving primary care provider knowledge and competence in managing chronic pain: A randomised controlled trial. Journal of telemedicine and telecare. Jan-Feb 2020;26(1-2):21-27. doi:10.1 177/1357633×18802978
6. Holmes CM, Keyser-Marcus, L., Dave, B., & Mishra, V Project ECHO and opioid education: a systematic review. Current Treatment Options in Psychiatry. 2020:14.
7. Carlin L, Zhao J, Dubin R, Taenzer P, Sidrak H, Furlan A. Project ECHO telementoring intervention for managing chronic pain in primary care: Insights from a qualitative study. Pain Medicine. 2018;19 (6):1140-1146.
8. Johnson KL, Hertz D, Stobbe G, et al. Project Extension for Community Healthcare Outcomes (ECHO) in multiple sclerosis: increasing clinician capacity. International journal of MS care. 2017;19 (6):283-289.
9. Fisher E, Hasselberg M, Conwell Y, et al. Telementoring primary care clinicians to improve geriatric mental health care. Population health management. 2017;20(5):342-347.
10. Gordon MJ. Self-assessment programs and their implications for health professions training. Academic medicine : journal of the Association of American Medical Colleges. Oct 1992;67(10):672-9. doi:10.1097/00001888- 199210000-00012
11. Approved initiatives for Maintenance of Certification for the ABMS (2006).
12. Arora S, Kalishman SG, Thornton KA, et al. Project ECHO: A Telementoring Network Model for Continuing Professional Development. The Journal of continuing education in the health professions. Fall 2017;37(4):239-244. doi:10.1097/ ceh.0000000000000172
13. Komaromy M, Duhigg D, Metcalf A, et al. Project ECHO (Extension for Community Healthcare Outcomes): A new model for educating primary care providers about treatment of substance use disorders. Substance abuse. 2016;3 7(1):20-4. doi:10.1080/08897077.2015.1129388
14. Shelley BM, Katzman JG, Comerci GD, Jr., et al. ECHO Pain Curriculum: Balancing Mandated Continuing Education With the Needs of Rural Health Care Practitioners. The Journal of continuing education in the health professions. Summer 2017;37(3):190-194. doi:10.1097/ceh.000 0000000000165
15. Zhou C, Crawford A, Serhal E, Kurdyak P, Sockalingam S. The Impact of Project ECHO on Participant and Patient Outcomes: A Systematic Review. Academic medicine : journal of the Association of American Medical Colleges. Oct 2016;91(10) :1439-1461. doi:10.1097/acm.0000000000001328
16. Arora S, Geppert CM, Kalishman S, et al. Academic health center management of chronic diseases through knowledge networks: Project ECHO. Academic medicine : journal of the Association of American Medical Colleges. Feb 2007;82 (2):154-60. doi:10.1097/ACM.0b013e31802d8f68
17. A. B. Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ: PrenticeHall; 1986.
18. Mind in Society: The Development of Higher Psychological Processes. MA: Harvard University Press; 1978.
19. Situated Learning: Legitimate Peripheral Participation. . Cambridge university press; 1991.
20. Control. BASETEo. W.H. Freeman and Co; 1997.
21. Coach DSaDBM. Increasing Student Mathematics Self-Efficacy Through Teacher Training. Journal of Advanced Academics. 2007;18(2):44.
22. Mazurek MO, Brown R, Curran A, Sohl K. ECHO Autism. Clinical pediatrics. Mar 2017;56(3):247-256. doi:10.1177/0009922816648288
23. Arora S, Kalishman S, Thornton K, et al. Expanding access to hepatitis C virus treatment–Extension for Community Healthcare Outcomes (ECHO) project: disruptive innovation in specialty care. Hepatology (Baltimore, Md). Sep 2010;52(3) :1124-33. doi:10.1002/hep.23802
24. Masi C, Hamlish T, Davis A, et al. Using an established telehealth model to train urban primary care providers on hypertension management. Journal of clinical hypertension (Greenwich, Conn). Jan 2012;14(1):45-50. doi:10.1111/j.1751-7176.20 11.00559.x
25. Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. Journal of advanced nursing. 2006;53(2):205-212.
26. Mengual-Andrés S, Roig-Vila R, Mira JB. Delphi study for the design and validation of a questionnaire about digital competences in higher education. International Journal of Educational Technology in Higher Education. 2016;13:1-11.
27. Blasco Mira JE, López Padrón A, Mengual Andrés S. Validación mediante el metodo Delphi de un cuestionario para conocer las experiencias e interés hacia las actividades acuáticas con especial atención al Winsurf. 2010;
28. Campbell DT, & Fiske, D. W. Convergent and discriminant validation by the multitrait-multimethod matrix.Psychological Bulletin. 1959;5 6(2):25. doi:https://doi.org/10.1037/h0046016
29. Landeta O, Calvete E. Adaptación y validación de la escala multidimensional de apoyo social percibido. Ansiedad estrés. 2002:173-182.
30. Lawshe CH. A QUANTITATIVE APPROACH TO CONTENT VALIDITY. Personnel Psychology. 1975; 28:563-575.
31. Arora S, Thornton K, Bradford A. Access to care: Management of hepatitis C viral infection in remote locations†. Clinical Liver Disease. 2012; 1(3):87-90. doi:10.1002/cld.46
32. Arora S, Brakey HR, Jones JL, Hood N, Fuentes JE, Cirolia L. Project ECHO for Cancer Care: a Scoping Review of Provider Outcome Evaluations. Journal of Cancer Education. 2023/10/01 2023;38 (5):1509-1521. doi:10.1007/s13187-023-02292-x
33. Tian J, Atkinson NL, Portnoy B, Lowitt NR. The Development of a Theory-Based Instrument to Evaluate the Effectiveness of Continuing Medical Education. Academic Medicine. 2010;85(9):1518-1525. doi:10.1097/ACM.0b013e3181eac3fb
34. Ramos MM, Sebastian RA, Murphy M, Oreskovich K, Condon TP. Adolescent substance use: Assessing the knowledge, attitudes, and practices of a school-based health center workforce. Substance abuse. Apr-Jun 2017;38 (2):230-236. doi:10.1080/08897077.2017.1287149
35. Löfmark A, Mårtensson G. Validation of the tool assessment of clinical education (AssCE): A study using Delphi method and clinical experts. Nurse Education Today. 2017/03/01/ 2017;50:82-86. doi:https://doi.org/10.1016/j.nedt.2016.12.009
36. Hong QN, Pluye P, Fàbregues S, et al. Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study. Journal of Clinical Epidemiology. 2019/07/01/ 2019;111:49-59.e1. doi:https://doi.org/10.1016/j.jclinepi.2019.03.008
37. Gandek B, Ware J, Aaronson N, et al. Tests of data quality, scaling assumptions, and reliability of the SF-36 in eleven countries: results from the IQOLA Project. International Quality of Life Assessment. Journal of clinical epidemiology. 1998;51 11:1149-58.
38. Ware J, Gandek B. Methods for testing data quality, scaling assumptions, and reliability: the IQOLA Project approach. International Quality of Life Assessment. Journal of clinical epidemiology. 1998;51 11:945-52.
39. McHorney CA, Ware JE, Jr., Raczek AE. The MOS 36-Item Short-Form Health Survey (SF-36): II. Psychometric and clinical tests of validity in measuring physical and mental health constructs. Medical care. Mar 1993;31(3):247-63. doi:10.1097 /00005650-199303000-00006
40. Sockalingam S, Rajaratnam T, Gambin A, Soklaridis S, Serhal E, Crawford A. Interprofessional continuing professional development programs can foster lifelong learning in healthcare professionals: experiences from the Project ECHO model. BMC Medical Education. 2022/06/06 2022; 22(1):432. doi:10.1186/s12909-022-03500-w
41. Hojat M, Veloski JJ, Gonnella JS. Measurement and Correlates of Physicians’ Lifelong Learning. Academic Medicine. 2009;84(8):1066-1074. doi:10 .1097/ACM.0b013e3181acf25f
42. Ware J, Brook R, Davies A, et al. Conceptualization and Measurement of Health for Adults in the Health Insurance Study: Vol. I, Model of Health and Methodology. 1979:
43. Humphrey-Murto S, Varpio L, Wood TJ, et al. The Use of the Delphi and Other Consensus Group Methods in Medical Education Research: A Review. Academic Medicine. 2017;92(10):1491-1498. doi:1 0.1097/acm.0000000000001812
44. Nguyen Thanh Huong LNK. Designing and validation of a living standards questionnaire in Vietnam’s elderly community. Medical Practice Journal. 2010;