Validation and reliability of the self-efficacy scale to assess the competence of pediatricians in ECHO-Pediatrics course

Le Hong Nhung¹, Nguyen Phuong Lan¹, Piter Martinez Benitez², Pham Van Hoc², Pham Duc Han², Matthew F Bouchonville³*

1. Vietnam National Children’s Hospital, Hanoi, Vietnam
2. General Hospital Hong Ngoc, Hanoi, Vietnam
3. University of New Mexico Health Sciences Center, Albuquerque, New Mexico, US

OPEN ACCESS

PUBLISHED: 31 October 2024

CITATION: Nhung, L.H., Lan, N.P., et al., 2024. “Validation and reliability of the self-efficacy scale to assess the competence of pediatricians in ECHO-Pediatrics course”. Medical Research Archives, [online] 12(10).
https://doi.org/10.18103/mra.v12i10.5771

COPYRIGHT: © 2024 European Society of Medicine. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

DOI https://doi.org/10.18103/mra.v12i10.5771

ISSN 2375-1924

Abstract

Keywords

• Self-efficacy
• Pediatrics
• ECHO-Pediatrics
• Competence assessment
• Reliability

Introduction

Extension for Community Healthcare Outcomes (ECHO) is a platform for academic health centers to expand healthcare worker capacity to underserved populations to meet the need of medical healthcare service¹. This program uses the model of “hub and spoke” through a video conferencing platform to connect experts at Vietnam National Children’s Hospital (VNCH, the hub) with the healthcare workers at different provincial hospital (spokes)². The ECHO model has been developed and replicated in many different countries to address health problems and treat common and complex medical conditions, such as HIV/AIDS, substance use disorders, palliative care, rheumatism, chronic diseases and other diseases²–⁹. During the implementation process, the ECHO model is regarded as a continuing medical education to improve the professional capacity and professional collaboration of health workers in patient care.

The first self-assessment of competency is a key concept that has shaped medical training since the 1990s, demonstrating the desire of healthcare workers to make self-determination and selfregulation based on the realities faced in patient care. This desire depends on the readiness of healthcare workers to self-assess their knowledge, skill and practice based on patient outcomes¹⁰.

Self-assessment of competency in continuing medical education has been fully approved by the American Medical Association (AMA) in its program of retention of practice certificates¹¹. To evaluate clinician competence, confidence and professional collaboration in clinical practice in the workplace are significant goals in studies assessing the effectiveness of the ECHO model according to Moore’s method¹²–¹⁵. Results from scale development inform future studies related to the ECHO model with respect to validity and reliability.

The first ECHO program, developed by Dr. Arora at the University of New Mexico for hepatitis C, used the self-assessment of competency scale named “self-efficacy,” laying the foundation for the construction of the scale later on in the authors’ studies on different topics¹⁶. The theoretical basis of the self-efficacy scale included the combination of social cognitive theories of Bandura¹⁷, Vygotsky’s situation-based learning theory¹⁸ and the practice community of Lave¹⁹. The term of “self-efficacy” refers to “self-confidence or the perception that one has the ability to organize and take the actions necessary to succeed when given a task.” (Bandura, 1997)²⁰. Self-efficacy is based on three sources of information: an individual’s previous experience, observational experiences in practice, and verbal persuasion on the psychological status of the learner²¹.

Later, many ECHO programs were developed, but all were based on these three theories. A number of studies have shown that synergies in learning, coaching and mentoring by experts, among colleagues, have formed “learning loops”²². Selfefficacy was assessed at three stages: before training, after training and 6 months after completing the training²³. Later studies improved the concept of “self-assessment of competence” developed for the ECHO hypertension program²⁴.

When studying ECHO programs from the perspective of continuing medical education (CME), researchers found that the scale of “selfefficacy” or “self-assessment of competence” had also been developed and applied in different aspects in the implementation process. These included assessing participants’ satisfaction with implementing best practices in the workplace, known as the scale of “job satisfaction” or “professional satisfaction.”

In the current study, we aimed to introduce the theoretical framework for developing a set of scales to facilitate self-assessment of professional capacity among healthcare workers participating in project ECHO. Additionally, we proposed a viable method for creating a self-efficacy scale applicable to the continuing medical training program. We hypothesized that the scale of efficacy would be applied to the research and evaluation procedure at VNCH.

Methods

STUDY SETTING
The study was conducted at the VNCH and its satellite hospitals from January to December 2020. Recruitment of participants occurred from July to September 2020. The study proposal was approved by the research ethics committees at the Hanoi University of Public Health (261/2020/ YTCCHD3) and the VNCH (883/BVNTW-VNCSKKTE). The VNCH was one of the leading pediatric hospitals in Vietnam, located in the North of Vietnam. The hospital played the role as a center of pediatric healthcare directives and provided the Continuing Medical Education (CME) programs of pediatrics for the province-level hospitals. The administrative function of the hospital included updating professional referrals and treatment guidelines for the province-level hospitals, organizing CME courses to update knowledge, and supporting expertise resources in difficult cases at the provincial level. The Project ECHO for pediatricians was launched in January 2020 to improve the capacity of physicians at province-level hospitals, following which the program recruited participants from July to September 2020 and organized online courses after the recruitment period. The anatomy of each session comprised two parts: the theoretical part (didactics) and the practical part (case study discussion). The online course included 8 to 10 sessions depending on the specific course, in which each session lasted two hours with 30 minutes for theoretical presentation and the remaining 90 minutes for discussing a patient case. A typical online course was conducted twice per week for 3-5 weeks. In this study, we only focused on learners who participated in the ECHO Immunology course in 2020. The hub was VNCH and the spokes were learners from 18 province-level hospitals specializing on pediatric faculty to participate in the ECHO course. The learners were sponsored by the national grant to pay fees during the period of studying.

STUDY DESIGN AND SAMPLING
Applying a mixed method of qualitative and quantitative research, the study was performed through a three-round Delphi iterative consultation process with experts²⁵. This technique is widely utilized in the research context²⁶ and its validity for questionnaire development has already been described²⁷. Furthermore, the final questionnaire was tested in the field, then validated following the Campbell & Fiske method with constructive validity and inconsistent reliability²⁸.

PARTICIPANTS
The panel of expert was made up of six researchers from six units including Hanoi Medical University, Hanoi University of Public Health, and the Agency of Science, Technology and Training, at Ministry of Health. Eligible participants were academic training managers or professors in the areas of both teaching and research relating to the study topic. Intentional sampling²⁹ was used for those panelists who complied with the aforementioned criteria, the panel being contacted by e-mail. The procedure below began to implement after their confirmation of participation.

PROCEDURE
The construction of the scale “self-assessment of the physician’s professional competence” took place in three distinct stages: Qualitative study with Delphi method (included preliminary stage and exploratory stage):

• Preliminary stage: Set up the focus group discussion by research team to find a consensus on the content of the initial scale.

• Exploratory stage: Invited six experts on the CME implementation as panelists to comment on the revised scale through the roundtable discussion

  • The research team called or met in person, wrote emails to discuss with the experts about the purpose of collecting the expert’s opinion, and instructed the experts to focus on the content of items in the scale (Appendix 1)

  • Collected comments from six experts and continued to discuss with the research team, revised the items of scale according to the expert’s comments in each round. The submission was done in 3 rounds.

• Collected the expert’s opinion based on Delphi method following the issues:
  ▪ Experts commented on whether the items were appropriate to measure the “selfassessment of competency” before and after participating the virtual training following the ECHO model.
  ▪ Experts made any possible suggestions with adding or removing items or rewording the items on the scale.
  ▪ Experts evaluated the instruction of scale.
  ▪ Experts evaluated the format of the scale.
  Inclusion Criteria:

  • Items with a consensus rate (surface validity) above 0.78³⁰.
    Exclusion Criteria:

  • Items with a consensus rate lower than 0.5.

  • Items with a consensus rate from 0.5–0.78 continued to be evaluated in the following round after revision based on the experts’ suggestion.

Final stage: Quantitative study to test the adjusted scale in the field with Campbell & Fiske method

• Chose the convenience sample with 38 learners who joined in the ECHO course at VNCH.

• Invited learners to participate in the study with the answers to the questions of the scale, collected feedbacks from learners who were evaluated by the data.

• Calculated the internal consistency reliability Cronbach’s Alpha, the convergent value and the discriminant value based on the learner’s feedback.

STUDY TOOLS AND DATA COLLECTION
The initial scale after being translated into Vietnamese then being revised based on focus group’s discussions that included 28 items naming “self-assessment of the physician’s professional competence”, Appendix 1, divided into 4 domains including:

Section 1: self-assessment of competence in knowledge and professional practice before training with six items;
Section 2: self-assessment of competence in knowledge and professional practice after training with six items;
Section 3: self-assessment of professional satisfaction in clinical practice before training with eight items;
Section 4: self-assessment of professional satisfaction in clinical practice after training with eight items.

The final adjusted scale (Table 6) was developed after three rounds of expert voting and was adjusted based on their comments. The experts were requested to rate from 1 to 4 on a Likert scale with level 1: Very unsuitable; level 2: Not suitable; level 3: Suitable; level 4: Very suitable.

DATA COLLECTION AND ANALYSIS

• Data collection on the expert opinions was carried out three times, counting on the concensus rate (face validity) and the median value in each rounds to select sub-items in the scale. All data were analyzed in 2020.

• The revised scale with 28 items was then sent to the experts for rating from 1 to 4 to find the degree of approval. With the point 3 and 4 obtained, collecting the experts’ opinions continue until the face validity reached above 0.78, considered high approval.

• The final adjusted scale with high approval would be tested for reliability using Crobach’s Alpha internal consistency index and validated following the Campbell & Fiske method.

---

Results

PRELIMINARY STAGE RESULT:
Based on the literature review and the focus group discussion, we removed unsuitable items/subitems, leaving the initial scale with 28 items (Appendix 1).

The revised scale was sent to six experts for judgement process.

Exploratory stage result on the assessment of six experts through roundtable discussions

Table 1: the first roundtable discussion result

Section I, II: Scale evaluate the self-assessment of competency in knowledge, professional practice following Likert 5 BEFORE and AFTER training
(Learner tick the point from 1 to 5 under each question 1:Very poor; 2:Poor; 3: Average; 4: Good; 5: Excellent)

Sentence 1: Have the ability to detect symptoms of patients who need to be screened                                                                                                                      3.0 (2.75; 3.0) — 83.3%

Sentence 2: Have the ability to detect patients suitable for treatment                                                                                                                                                    3.0 (2.0; 3.0) — 66.7%

Sentence 3: Have the ability to assess the extent of damage done to relatable organs in patients                                                                                                    2.5 (1.75; 3.00) — 50%

Sentence 4: Have the ability to treat the patients and manage one’s side effects                                                                                                                               3.0 (2.0; 3.0) — 66.7%

Sentence 5: Have the ability to educate and motivate the patients                                                                                                                                                       2.5 (2.0; 3.0) — 50%

Sentence 6: Have the ability as a consultant in my clinic as well as locally regarding medical problems                                                                                       2.5 (2.0; 3.0) — 50%

---

Section III, IV: Scale evaluate the self-efficacy of job satisfaction following Likert 5 BEFORE and AFTER training
(Learner tick the point from 1 to 5 under each question 1:Very poor; 2:Poor; 3: Average; 4: Good; 5: Excellent)

Sentence 1: I feel isolated when working professionally at my workplace                                                                                                                                     2.5 (2.0; 3.0) — 50%

Sentence 2: I could easily form close relationships with my coworkers                                                                                                                                      2.5 (1.75–3.0) — 50%

Sentence 3: I could easily access a doctor when I needed their professional feedback or their help                                                                                      3.0 (2.0; 3.0) — 66.7%

Sentence 4: I could easily access all sources and documents for my career development (improvement in my knowledge)                                              2.0 (1.0; 2.25) — 16.7%

Sentence 5: When I needed the clinical doctors’ help or support, I was able to contact the experts in due time.                                                                3.0 (2.0; 3.0) — 66.7%

Sentence 6: I was able to have the opportunity to share my clinical experiences with my co-workers at regular intervals                                              3.0 (2.0; 3.0) — 66.7%

Sentence 7: I’m generally satisfied with my job                                                                                                                                                                                 3.0 (2.0; 3.0) — 66.7%

Sentence 8: I’m confident that I could improve the quality of medical service delivery in my facility                                                                                    3.0 (2.75; 3.0) — 83.3%

Table 2. the second roundtable discussion result

Section I, II: Scale evaluate the self-assessment of competency in knowledge, professional practice following Likert 5 BEFORE and AFTER training
(Learner tick the point from 1 to 5 under each question 1: Very poor; 2: Poor; 3: Average; 4: Good; 5: Excellent)

Sentence 1: Ability to detect symptoms of patients who need to be screened.
3.00 (3.00; 3.00) — 100%

Sentence 2: Ability to analyze and synthesize clinical and subclinical manifestation to make an appropriate diagnosis.
3.00 (2.75; 4.00) — 83.3%

Sentence 3: Ability to apply standardized treatment protocol and control side effects when prescribing.
3.50 (2.75; 4.00) — 83.3%

Sentence 4: Ability to handle emergency situations according to standard protocols.
3.00 (2.75; 3.25) — 83.3%

Sentence 5: Ability to advise and educate patients on self-care and disease prevention.
3.50 (2.75; 4.00) — 83.3%

Sentence 6: Ability to manage complications of the disease.
3.00 (2.00; 3.00) — 66.7%

---

Section III, IV: Scale evaluate the self-efficacy of job satisfaction following Likert 5 BEFORE and AFTER training
(Learner tick the point from 1 to 5 under each question 1: Very poor; 2: Poor; 3: Average; 4: Good; 5: Excellent)

Sentence 1: Ability to resolve independently professional issues at the workplace.
3.00 (2.75; 3.25) — 83.3%

Sentence 2: Ability to exchange clinical experiences among colleagues.
3.00 (2.75; 3.00) — 83.3%

Sentence 3: Ability to get experts’ support to solve illness cases at the workplace.
3.00 (2.75; 3.25) — 83.3%

Sentence 4: Ability to finish the assigned work according to one’s expertise in the workplace.
3.00 (2.75; 4.00) — 83.3%

Sentence 5: Ability to improve the quality of the treatment and examination services according to one’s professional department in the workplace.
3.00 (2.75; 4.00) — 83.3%

---

In the second round, two additional items were added and continued to be reviewed for approval by the experts. The result of the second round included 22 items with six couples of items regarding confidence in professional knowledge and practical expertise before and after ECHO training regarding before and after ECHO. The remaining three couples of items regarding one’s ability to coordinate professionally in clinical practices.

Table 3: the third roundtable discussion

---

Section I: Scale evaluate the self-assessment of competency in knowledge, professional practice following Likert 5 BEFORE and AFTER training

(Learners tick the given score from 1 to 5 beneath each question. 1: Very poor; 2: Poor; 3: Average; 4: Good; 5: Excellent. (Students can skip this section if they deem it unsuitable))

Sentence 1: Ability to examine and detect clinical symptoms
3.0 (3.0; 3.0) — 100%

Sentence 2: Ability to analyze and synthesize the clinical and paraclinical characteristics to present the most appropriate diagnosis
3.00 (2.75; 4.00) — 83.3%

Sentence 3: Ability to apply treatments per standard protocols and control the side effects when prescribing
3.5 (2.75; 4.0) — 83.3%

Sentence 4: Ability to handle emergencies according to standard protocols
3.0 (2.75; 3.25) — 83.3%

Sentence 5: Ability to advise and educate patients on self-care and disease prevention following standard protocols
3.5 (3.0; 4.0) — 100%

Sentence 6: Ability to manage the risks involved, detect and smartly solve the complications of the illness
3.0 (2.75; 3.25) — 83.3%

---

Section II: Scale evaluate the self-efficacy of job satisfaction following Likert 5 BEFORE and AFTER training

(Students tick the given score from 1 to 5 beneath each question. 1: Very poor; 2: Poor; 3: Average; 4: Good; 5: Excellent (Students can skip this section if they deem it unsuitable))

Sentence 1: Ability to resolve independently the professional issues at the workplace
3.00 (2.75; 4.00) — 100%

Sentence 2: Ability to exchange experiences in examination skills, diagnosis skills, and treatment skills among co-workers at the workplace
3.00 (3.00; 3.25) — 100%

Sentence 3: Ability to get senior experts’ support to solve illness cases at the workplace
3.00 (2.75; 3.25) — 83.3%

Sentence 4: Ability to finish the assigned work according to one’s expertise in the workplace
3.00 (2.75; 4.00) — 83.3%

Sentence 5: Ability to improve the quality of the treatment and examination services according to one’s professional department in the workplace
3.0 (2.75; 4.00) — 83.3%

---

As the result of the third round, sentence 6 in section I was revised:
“Ability to manage the risks involved, detect and smartly solve the complications of the illness.”

Table 4: The final adjusted scale

---

Section I: Scale evaluate the self-assessment of competency in knowledge, professional practice following Likert 5.

(Students tick the given score from 1 to 5 beneath each question.
1: Very poor; 2: Poor; 3: Average; 4: Good; 5: Excellent
(Students can skip this section if they deem it unsuitable))

Sentence 1: Ability to examine and detect clinical symptoms.

Sentence 2: Ability to analyze and synthesize the clinical and paraclinical characteristics to present the most appropriate diagnosis.

Sentence 3: Ability to apply treatments per standard protocols and control the side effects when prescribing.

Sentence 4: Ability to handle emergencies according to standard protocol(s).

Sentence 5: Ability to manage the risks involved, detect and solve the complications of the illness.

Sentence 6: Ability to advise and educate patients on self-care and disease prevention following standard protocol(s).

---

Section II: Scale evaluate the self-efficacy of job satisfaction following Likert 5

(Students tick the given score from 1 to 5 beneath each question.
1: Very poor; 2: Poor; 3: Average; 4: Good; 5: Excellent
(Students can skip this section if they deem it unsuitable))

Sentence 1: Ability to independently resolves professional issues at the workplace.

Sentence 2: Ability to exchange experiences in examination skills, diagnosis skills, and treatment skills among co-workers at the workplace.

Sentence 3: Ability to get senior experts’ support to solve illness cases at the workplace.

Sentence 4: Ability to finish the assigned work according to one’s expertise in the workplace.

Sentence 5: Ability to improve the quality of the treatment and examination services according to one’s professional department in the workplace.

---

Finally, the adjusted scale of self-assessment of competency measured the change of a physician knowledge and professional practice before and after ECHO model training, which consisted of four categories:

Section I: “Scale evaluate the self-assessment of competency in knowledge, professional practice following Likert 5 BEFORE training” consisted of six items.

Section II: “Scale evaluate the self-assessment of competency in knowledge, professional practice following Likert 5 AFTER training” consisted of six items.

Section III: “Scale evaluate the self-efficacy of job satisfaction following Likert 5 BEFORE training” consisted of five items.

Section IV: “Scale evaluate the self-efficacy of job satisfaction following Likert 5 AFTER training” consisted of five items.

According to Lawshe’s suggestion, consensus rate at 0.78 and above were considered satisfactory³⁰. The research team had already examined the final scale based on the three rounds of experts’ judgements.

---

Final Stage: Quantitative study to test the adjusted scale in the field with Campbell & Fiske method.

---

Table 5: Characteristics of item statistics

Self-assessment of competency in knowledge, professional practice before training
6 items — 0% missing — Mean: 2.18–2.37 — SD: 0.68–0.75

Self-assessment of competency in knowledge, professional practice after training
6 items — 0% missing — Mean: 3.55–3.68 — SD: 0.47–0.60

Self-efficacy of job satisfaction before training
5 items — 0% missing — Mean: 2.37–2.55 — SD: 0.82–0.47

Self-efficacy of job satisfaction after training
5 items — 0% missing — Mean: 3.45–3.63 — SD: 0.49–0.56

---

The mean score and standard deviations of the items in each domain before training ranged from 2.18±0.68 to 2.55±0.47;
The mean score and standard deviation of the items in each domain after training ranged from 3.45±0.49 to 3.68±0.6.

---

The construct validity including convergent and discriminant validity

---

Table 6: results of multi-point matrix correlation

Self-assessment of competency in knowledge, professional practice before training
0.68–0.94 | 0.25–0.75

Self-assessment of competency in knowledge, professional practice after training
0.42–0.75 | 0.23–0.73

Self-efficacy of job satisfaction before training
0.55–0.88 | 0.23–0.73

Self-efficacy of job satisfaction after training
0.33–0.82 | 0.25–0.75

The construct validity was evaluated by calculating the item’s convergent with the value fluctuate from 0.33–0.94 and discriminant validity from 0.23–0.75.

---

Internal consistency reliability

---

Table 7: Cronbach Alpha reliability

Self-assessment of competency in knowledge, professional practice before training
Correlations: —
Alpha: 0.829

Self-assessment of competency in knowledge, professional practice after training
Correlations: 0.744 with before-training competency, 1 with itself, 0.72 with job satisfaction before training
Alpha: 0.818

Self-efficacy of job satisfaction before training
Correlations: 0.789 with competency before training, —, 1 with itself
Alpha: 0.840

Self-efficacy of job satisfaction after training
Correlations: 0.764 with competency before training, 0.84 with competency after training, 0.813 with job satisfaction before training, 1 with itself
Alpha: 0.875

---

Internal consistency reliability was tested by Cronbach’s Alpha coefficient for each domain in the range of 0.72–0.87.

---

Discussion

Sanjeev Arora is well-known for his work on the Project ECHO model, which focuses on expanding access to specialty care in underserved areas through telemedicine and collaborative learning. While his primary work has been on the ECHO model, which emphasizes knowledge sharing and mentorship, self-efficacy has been a key concept in evaluating the outcomes of such programs. In particular, self-efficacy scales have been used to assess the impact of the ECHO model on healthcare providers’ confidence and abilities. Although Arora himself may not have explicitly developed or published studies solely focused on self-efficacy scales, his research and the evaluations of the ECHO model often incorporate self-efficacy as a measure of success³¹,³².

Self-efficacy, a concept developed by psychologist Albert Bandura, refers to an individual’s belief in their ability to successfully

execute tasks or achieve goals. In the context of Continuing Medical Education (CME), applying a self-efficacy scale can help assess and improve healthcare professionals’ confidence in their abilities related to their practice³³,³⁴.

The theoretical foundation of applying the “self-efficacy” assessment method in the ECHO model has been built on 3 learning theories: Bandura’s social cognition¹⁷, Lave’s situation-based learning theory¹⁹, and Vygotsky’s learning community theory¹⁹. In particular, Bandura’s Social Cognitive Theory is key, which describes three key factors that increase a person’s confidence to change their behavior to help them actively assess their own abilities through participation in the learning process that the ECHO model uses. It is called the “All teach all learn” method, exchanging experiences through cases that are accumulated gradually and regularly.

The “self efficacy” scale using in the ECHOPediatrics program in Vietnam was developed from the research related to the ECHO-Hepatitis C model by S Arora³⁵. In addition to self-assess the capacity improvement of healthcare providers, there has been a new point of the scale that evaluate the healthcare provider’s ability to meet the job satisfaction at workplace after being educated and how they concretize the ability to coordinate professionally with the team at their workplace.

This research used the Delphi method to adjust the “self efficacy” scale using in the ECHOPediatrics with face validity. This method was first introduced in a RAND report published in 1953, which outlined its use for obtaining expert opinions and reaching consensus on complex issues, reflecting the method’s aim to provide a structured process for predicting future events and gaining insights through expert judgment. The Delphi method can indirectly address face validity by refining instruments, questionnaires, or assessment tools through expert feedback³⁵.

Face validity is a form of validity that is concerned with whether an instrument appears to be measuring what it is supposed to measure, based on subjective judgments… leading to more accurate and relevant insights³⁶. In this study, in the exploratory stage, the face validity (consensus rate) attained at 0.78 and above were considered satisfactory³⁰. The researcher had already carefully examined the final scale based on the three rounds of experts’ judgements.

In addition to calculating the face validity according to the Delphi method, this study calculated the construct validity including convergent validity and discriminant validity according to the Campell Fiske method. Convergent validity refers to the degree to which two measures that are supposed to be related are actually related. In other words, if two different methods are used to measure the same construct, they should show high correlations with each other. Discriminant validity (or divergent validity) refers to the degree to which a measure is not related to measures of different constructs. It is essentially the opposite of convergent validity. If a measure is supposed to assess a specific construct, it should not be strongly related to measures of other, distinct constructs²⁸.

The final scale (22 items) after being validated with face validity was distributed by the research team to all learners participating in the online training program following the ECHO-Immunology course at the VNCH. None of the learners were disqualified. The survey was distributed to learners in an online session at the end of the course. The final adjusted scale provided instructions on the objective of the scale, how to respond to the questions of the scale, and ensuring anonymity. The items included demographic characteristics, the name of the program, and the learner’s qualifications for those who participated in the ECHO-Immunology course.

The quality and completeness of the feedback from³⁸ learners response to the scale were assessed. The mean and standard deviation were calculated based on the points in each item category. They were collected per measuring less

The percentage of missing data throughout the scale indicated that the items were confusing or that the scale layout was problematic³⁷. Spearman’s correlation was used in this study with a non-normal distribution.

---

THE CONSTRUCT VALIDITY INCLUDING CONVERGENT AND DISCRIMINANT VALIDITY

The construct validity was evaluated by calculating the item’s convergent and discriminant validity. The correlation of each item with its own total score was considered to be satisfactory if it valued > 0.30³⁸. Theoretically, the convergence value in the same item was higher than the correlation value in other items. The discriminant validity of the item assumed that in the tool with more than one domain, the correlations between items in the same domain were expected to be significantly higher than those in other domains. The scale success rate was calculated, as suggested by McHorney et al.³⁹, as the percentage of items in each domain that met the criteria for convergent validity and discriminant validity. Following the matrix model Campbell & Fiske, the result was interpreted at the diagonal values²⁶,³⁸.

---

INTERNAL CONSISTENCY RELIABILITY

Internal consistency reliability was tested by Cronbach’s Alpha coefficient for each domain and for the whole scale with an acceptable value of Cronbach Alpha ≥ 0.70, in the range of 0.72–0.87⁴⁰,⁴¹. However, a Cronbach Alpha value ≥ 0.60 was considered to be accepted in the newly developed scale⁴². Cronbach’s Alpha was also checked when individual items were deleted. Items that reduced the Cronbach’s Alpha value in the domain were considered for exclusion.

---

Limitation

The number of experts invited to participate in the quality assessment of the scale were less than previous studies according to the Delphi method⁴³. However, they were the leading experts whose academic qualifications were PhDs and Professors in the divisions of training management at the Department of Science Technology and Research – Ministry of Health, Vietnam Universities of Medicine and Public Health. Therefore, the value of judgements achieved were scientific and useful. A similar study in Vietnam invited five experts to participate in the evaluation of the questionnaire⁴⁴.

Additionally, to calculate the construct validity, there is a similar study in Vietnam using exploratory factor analysis (EFA), which is a widely used statistical analysis method. EFA helps researchers reduce the number of sub-items of a scale by removing sub-items with low relevant values and providing the optimal component structure of the scale, so it is appropriate to apply to a scale with a large number of sub-items⁴⁴.

---

Conclusion

The scale to assess the physician professional capacity improvement has been adjusted accordingly to each ECHO program in the world since 2007. In our program, we developed “the self-assessment of competency” using the Delphi method to evaluate the surface validity (consensus rate), combining with testing the scale in the field in order to calculate the construction validity and reliability following the Campbell & Fiske method. Therefore, this scale was valid and highly reliable that would be widely applied in evaluating the effectiveness of CME courses following the ECHO model in Vietnam.

We recommend that the scale continue to be developed depending on the disease state and customised to each organization’s capacity. In our study, this scale was built for general use in the ECHO courses with different specialties in VNCH, and the medical terminology in the domains were adjusted to fit an ECHO program for CME purpose.

Financial Disclosure Statement

The authors declare that the hospital managers and healthcare providers who participated in this study have no financial interest. There are no funder or grantor support for the research.

Acknowledgements

The views expressed in this article are solely those of the authors and do not represent the official positions of the organizations the authors are affiliated with.

Ethics approval and consent to participate

The study proposal was approved by research ethics committees at the Hanoi University of Public Health.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

1. Fathy L.J., Purohit A., Chappell A., Taplin C., Martinez M., Fisher E., Hasselberg M., et al. Telemonitoring for the management of patients with chronic diseases: Findings of a Technical Expert Panel. Journal of general internal medicine. Nov 2016; 31(11): 1597-602. doi:10.1007/s11606-016-3725-9.
2. Arora S., Thornton K., Murata G., et al. Outcomes of treatment for hepatitis C virus infection by primary care providers. The New England Journal of medicine. Jun 9 2011; 364(23):2199-207. doi:10.1056/NEJMoa1001970.
3. Salvador D.J., Bhat S.R., Jacobson V.C., et al. Pallidum (Tn) monitoring with an ECHO intervention to expand access to care. BMJ global health. 2019; 4(1): e001059. doi:10.1136/bmjgh-2018-001059.