Effects of Virtual Education of Nutrition Programs in the Health System Evolution Plan on the Knowledge and Performance of Primary Health Workers (Behvarzes) in Mashhad, Iran Based on the Kirkpatrick Model in 2020

Introduction

In recent years, virtual education has emerged as one of the important applications of new information and communication technologies in the world and extensive activities have begun in this aspect. Due to the rapid changes taking shape in the environment, the implementation of virtual systems to provide new services and technologies in the field of teaching and learning has become a basic need. Virtual education is only a complement to physical educational environments and cannot replace them (1).

Given that the implementation of training courses requires a lot of time, a thorough and scientific evaluation is necessary to identify its strengths and weaknesses, help find ways to modify and improve them, and also reveal the extent to which its goals are achieved (2). There are different models for the evaluation of educational effectiveness, but the most common and general model used for this purpose is the four-level Kirkpatrick Model (reaction, learning, behavior, and results) (3).

Shayan and Nowroozi Rad (1398) conducted a study to determine the effectiveness of codified training programs for Taleghani Hospital staff in Tehran, Iran based on the Kirkpatrick Model in 2018. Their descriptive survey research was performed on 48 participants, and their results showed that the conducted training course was effective based on the four levels of the Kirkpatrick Model (4). Moreover, Nazari et al. in their study in 2018 examined the effectiveness of virtual training programs that aimed to improve the performance of Human Resources Deputy staff of NAJA (Law Enforcement Force of Islamic Republic of Iran). Their study was descriptive-analytical and the sample size was calculated at 137 participants based on Cochran's formula. Results of the above-mentioned study indicated that virtual training had a significant positive effect on the performance of NAJA staff. Moreover, they found that virtual training had the greatest effect on the knowledge of Human Resources Deputy staff (1).

Most of the previous studies have examined the effect of education on knowledge and less attention has been paid to the effect of virtual education on attitude and performance. Therefore, this study aimed to determine the effects of virtual education of nutrition programs in the Health System Evolution Plan on the knowledge and performance of the primary health workers (Behvarzes) in Health Center No. 1 of Mashhad, Iran based on the Kirkpatrick Model.

Methods

This analytical cross-sectional study was performed on 94 primary health workers in Health Center No. 1 in Mashhad. The sample size was calculated with 45 participants based on the following formula and the samples were selected using a random number table.

The training course was held online through the virtual staff training system according to the staff training criteria based on the approved lesson plan. The inclusion criteria were willingness to participate in the study (choosing this course based on a sense of need and interest in the assessment of their individual needs), on-time attendance in the course exam, and not leaving the exam. On the other hand, the exclusion criterion was non-completion of the questionnaire which referred to participants who had registered for the course but were not able to participate in the course exam.

The participants completed the researcher-made questionnaire which was based on the four-level Kirkpatrick Model (reaction, learning, behavior, and results) during the course. The questionnaire was scored based on a Likert scale ranging from one (very poor) to five (very good). The first level of the questionnaire (reaction) consisted of two parts; the first part included 10 items about the quality of the course content. The second part had nine items regarding the presentation of the course. The second level questionnaire (learning) consisted of 40 four-option multiple-choice questions. Each participant had to answer 20 items that were randomly assigned to them in the virtual education system of the staff. The third (behavior) and fourth (results) levels of the questionnaire consisted of 12 and 6 items, respectively, and were scored based on a scale ranging from one (never) to five (always). It should be mentioned that these two levels of the questionnaire were completed one month after the course.

To determine the content validity ratio, 10 specialists and experts, including six Master's degree holders, one Ph.D. holder, one Ph.D. student, and two specialist physicians were invited to examine the items of the study tool to divide them into "necessary", "useful but unnecessary", and "unnecessary" categories. The content validity ratios of the reaction questionnaire regarding training course content and presentation of the course were 0.82 and 0.95, respectively. Moreover, the content validity ratios of the learning questionnaire (training course questions), behavior questionnaire, and results questionnaire were 0.78, 0.91, and 0.80, respectively. Furthermore, the content validity index (CVI) values of the reaction questionnaire in terms of course content and course presentation were 0.96 and 0.98, respectively. Besides, the CVI values of the learning questionnaire (training course questions), behavior questionnaire, and results questionnaire were 0.98, 0.99, and 0.90, respectively.

To determine the reliability of the tool, the reaction questionnaire (quality of course content and course presentation) was completed by 20 participants who completed it once more after 10 days. The paired t-test was used to evaluate the reliability of the questionnaire. Given the p-value at the margin of error less than 5%, the mean scores of the questionnaire at the two stages did not have a statistically significant difference. Hence, it could be said that the questionnaire is reliable (P=0.149 for quality of course content) and (P=0.056 for course presentation).

The reliability of the learning questionnaire (training course questions) was evaluated using the split-half reliability method. Accordingly, the test questions were divided into two halves and the score of the first and second halves of the items were calculated and resulted in a correlation coefficient of r=0.572 and a reliability coefficient of 0.727. The reliability of the behavior (acquired skills of the subjects) and results (job performance of the subjects) questionnaires were calculated at 0.81 and 0.83 by Cronbach's alpha, respectively.

The collected data were analyzed in SPSS software (version 21). After entering the data, analysis was performed using the software output. Through descriptive statistics, the samples were described by constructing frequency distribution tables and required graphs as well as calculating numerical indicators, such as mean, standard deviation, and percentages.

Results

Based on the results, 71.2% of the primary health workers participating in the course were female. Moreover, it was found that most of the participants had a diploma (66.7%), were within the age range of 35-40 years (29%), and had 5-10 years of work experience (33%). The mean scores of the quality of the course content and course presentation are summarized in Table 1. According to Table 1, the participants were satisfied with the quality of the course content and the way the course was held. The mean score obtained from the test is presented in Figure 1. 

As can be seen in Figure 1, the mean score obtained from the test was 85.9±10.3. After scoring the items on the acquired skill questionnaire from one (never) to five (always), the mean score of each participant was calculated. Table 2 summarizes the mean score of the obtained skills by the participants.

As can be seen in Table 2, the mean score of the acquired skill questionnaire was at the level of “always”. After scoring the items on the job performance questionnaire from one (never) to five (always), the mean score of each participant was calculated. Figure 2 summarizes the mean score of the job performance questionnaire. 

As can be seen in Figure 2, the mean of the job performance questionnaire was at the level of always.

Discussion

Based on the results, the mean score of the course content quality questionnaire was 4.61±0.45 which can indicate the satisfaction of the participants in this regard. Moreover, the mean score of the course presentation questionnaire was 4.5±0.6 which can indicate that the presentation of the course was at a very good level. This means that according to the first level of the Kirkpatrick Model (reaction) the primary health workers were satisfied with how the virtual course was held.

Gholami and Norouzi performed a study (2015) to investigate and compare the satisfaction of the employees of the National Oil Company with virtual and face-to-face training courses. They found that the employees were more satisfied with virtual training courses, compared to face-to-face training courses. Moreover, they studied the three dimensions of education satisfaction (content, presentation, and feedback) and found that satisfaction with the course content, method of presentation, and feedbacks of virtual education courses were more than face-to-face training courses (5). Arian Mehr (2015) conducted a study to evaluate the effectiveness of in-service training courses for high school teachers in Yasouj, Iran based on the Kirkpatrick Model in the 2014-15 academic year. He found that the content of training courses had a positive effect on the participants (6). The results of the above-mentioned studies were in line with those of the present study.

In the present study, the mean score of the learning questionnaire was 85.9±10.3. Shayan and Norouzi Rad in their research titled “Determining the effectiveness of written training programs for the staff of Taleghani Hospital in Tehran Kirk Patrick pattern in 2018” found that the training course was very good in terms of the reaction of participants and there was a significant change in their levels of learning. Therefore, according to the Kirkpatrick Model, the training course was effective (4).

Furthermore, the results of a study performed by Jamaledini etal in 2016 revealed that the basic crisis management training course was effective at two levels of reaction and learning according to the Kirkpatrick Model. In his study, there was a significant difference between the pre-test and post-test mean scores of the participants. Besides, the participants evaluated the quality of the training course and the whole course as desirable (7).

The mean score of acquired skills by participants was 4.75±0.34 indicating that the degree of skills acquired by individuals was at the level of “always” (based on the Likert scale). Moreover, the mean score of the job performance of participants was 4.55±0.54 which also means that the level of their job performance was at the level of “always” (based on the Likert scale). Nazari et al. conducted a study (2018) to investigate the effect of virtual training courses on improving the performance of Deputy Human Resources staff of NAJA. They found that the virtual training had a positive and significant effect on the performance; accordingly, virtual training improved their job and professional performance. Based on their findings, virtual training was most effective on the knowledge of the Deputy of Human Resources staff (1). Furthermore, Kermani performed a research (2018) titled “Evaluating the Effectiveness of In-Service Training of Staff at Tarbiat Modarres University in Tehran from the Employer's Point of View Based on the Four-level Model of Kirk Patrick”. He found that the courses were effective based on the four levels of the Kirkpatrick Model (8) which is consistent with the results of the present study.

One of the limitations of this research was its cross-sectional nature and also its performance on a specific statistical population which reduces its generalizability.

A training program can only be valuable if it provides reliable and credible evidence of the impact of training on the improvement of the behavior and performance of the participants. Examination of the effectiveness of training courses helps the quality of course presentation. Accordingly, the results of quantitative and qualitative assessments of programs and plans can be used to increase the effectiveness of the courses and develop virtual training programs. Findings of this study indicated the effect of in-service education of nutrition programs in Health System Evolution Plan on the knowledge and performance of primary health workers in Mashhad based on the four levels of the Kirkpatrick Model (reaction, learning, behavior, and results).

It is suggested to identify the obstacles and problems that lead to the reluctance of employees to attend virtual training courses and try to reduce and eliminate them. This will help encourage more employees to participate and enrich the courses and virtual training programs (through the use of new methods in virtual training). Finally, considering that in previous studies the Kirkpatrick Model has been mostly used to examine the impact of face-to-face training courses, it is suggested for future research to use this model for the investigation of virtual education.

Ethical considerations 

Ethical issues including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc. have been completely observed by the authors. The ethics committee of Mashhad University of Medical Sciences approved this research, ethics code IR.MUMS.MEDICAL.REC.1398.822.