Mon, Jun 2, 2025
[Archive]
Volume 16, Issue 51 (2023)
JMED 2023, 16(51): 38-48 |
Back to browse issues page
Ethics code: IR.VUMS.REC.1400.029
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Ebrahimpour S, Rezaei E, Namazi S, Mohammadi M. Design, implementation, and evaluation of a diabetes educational game for the pharmacy students: A parallel-group study. JMED 2023; 16 (51) :38-48
URL: http://edujournal.zums.ac.ir/article-1-1737-en.html
URL: http://edujournal.zums.ac.ir/article-1-1737-en.html
1- Department of Clinical Pharmacy, School of Pharmacy, Alborz University of Medical Sciences, Karaj, Iran.
2- Medical Education Department, Smart University of Medical Sciences, Tehran, Iran.
3- Department of Clinical Pharmacy, Faculty of Pharmacy, Tehran University of Medical Sciences, Tehran, Iran.
4- Department of Clinical Pharmacy, School of Pharmacy, Alborz University of Medical Sciences, Karaj, Iran. ,m.mohammadi@abzums.ac.ir
2- Medical Education Department, Smart University of Medical Sciences, Tehran, Iran.
3- Department of Clinical Pharmacy, Faculty of Pharmacy, Tehran University of Medical Sciences, Tehran, Iran.
4- Department of Clinical Pharmacy, School of Pharmacy, Alborz University of Medical Sciences, Karaj, Iran. ,
Full-Text [PDF 509 kb]
(477 Downloads)
| Abstract (HTML) (1519 Views)
Figure 1. An overview of the study phases.
Phase one- Preparation and validation of educational material and game scenario
In the first step, educational content for diabetes was prepared by a three-member panel of clinical pharmacists with sufficient knowledge and experience in diabetes management. This content was based on the latest edition of the reference book “Applied therapeutics: the clinical use of drugs” (14), supplemented with essential highlights of the 2020 American Diabetes Association guideline (15). Three sets of educational materials were then prepared based on this content. The first included the theoretical aspects of the diagnosis and management of diabetes that were required by the Iranian pharmacy curriculum. This material was used to educate all students during a four-hour webinar, regardless of whether they would participate in the study. The second and third materials were problem-based content used to prepare the multimedia and game files and were intended to enhance the students’ problem-solving capabilities. Because the game was designed with real cases, it was inherently different from the material used for the webinar. If the study had only compared the game to a control group, any observed effect on student’s learning and retention would have been due to the game’s case-based nature rather than the game itself. To avoid this, a parallel-group, the “multimedia” arm, with a case-based structure was included in the study. By doing so, any effect of the game on student’s learning and retention could be attributed to the “nature of the game” rather than the case-based element alone.
The multimedia file included educational material and clinical cases on diabetes and was created by Articulate Storyline v.3.12. The game scenario consisted of 25 questions (stages) in four areas; diagnosis of diabetes, oral hypoglycemic agents, management of hypoglycemia, and insulin therapy. Afterward, the storyboard and wireframe of the game were created by an educational technology specialist.
The initial version of the game, GlucoGame, was developed by the iSpring Suite for both Windows and Android platforms. The clinical pharmacist panel played it and provided necessary feedback to the game developer. Three education technology experts evaluated the content validity of the game. The final version included 25 stages in the form of puzzles, drag-and-drop, matching, and multiple-choice questions. In each stage, an audio guide was placed that explained the related scientific basics, and the students could listen to it before playing that stage. The players could proceed to the next step only if they did the game correctly. In case of a wrong answer, it was possible to change it. The game environment is shown in Figure 2.
The second series of analyses included using the modified version of the MEEGA+ systematic model to evaluate the quality of game-based and multimedia-based learning. The questionnaires were completed by 50 students from the multimedia group (23 males and 27 females) and 49 students from the game group (14 males and 35 females).
Twenty-eight out of 35 (80.00%) female respondents to the game questionnaire stated that they had “never played” or “rarely played” other digital games. Five of the 14 males (35.71%) reported this rate of digital game playing. Eleven students in the game group stated that they play other digital games daily or several times a week; of them, 8 (72.72%) preferred learning through GlucoGame over other learning methods, such as multimedia or lectures. On the other hand, 34.21% (13 out of 38) of students with lower frequencies of digital game playing preferred Gluco Game over other learning methods.
Appendix 1 presents an overview of the participants’ opinions on various game dimensions and multimedia quality. Common questions of the game and multimedia questionnaires are presented sequentially to facilitate comparisons. As Appendix 1 displays, nearly all dimensions of the game quality were acceptable to the students.
Around 40 (81.63%) respondents to the game and 44 (88.00%) respondents to the multimedia questionnaires believed that the playing the game or multimedia assured them that they could learn from it. A total of 39 out of 49 respondents (79.59%) to the game questionnaire preferred the game over the usual teaching classes, and two-thirds (33 out of 49, 67.34%) would recommend it to other students. Only 24.48 % (12 out of 49) of the respondents believed that using the game alone was enough to learn the course, while the corresponding value was 66.00% (33 out of 50) for the multimedia content.
Discussion
This study aimed to design and implement an innovative game on the education of diabetes and evaluate its effect on the learning and retention of pharmacy students.
In the current study, the observed change in the students’ scores did not reach statistical significance. This finding is consistent with those of Sando and Feng, who evaluated the effect of an online game of drug information on the students’ knowledge scores (11). Although the students’ level of engagement with the game was high and acceptable, the percentages of correctly answered questions did not increase significantly. In another study, Auman (2011) used a simulation game to increase the students’ engagement in a psychology course (16). The result showed that the change in the post-test score compared to the pretest was not significantly different between the simulation and non-simulation groups. In contrast, in the study by Eukel et al., the mean post-test score was 81%, which was significantly higher than the mean pretest score of 56% (P<0.01) (12). The lack of significant change in the knowledge scores of our participants may be explained by several reasons. Firstly, the parallel group also used the multimedia file that reviewed all the aspects of diabetes diagnosis and management on a case-based method. As shown in Appendix 1, the multimedia file also received an acceptable score from the students regarding quality. Secondly, our study could not control students cheating. Although the students’ IP addresses were controlled to find the same ones, this method could not identify all cheating cases. Thirdly, one of the measures taken to limit material sharing between the study arms was to give the total score of the final exam to all students, regardless of their actual scores. It may have reduced students’ competitiveness. Previous studies showed conflicting results on the role of competitiveness in serious games. Cagiltay et al. (2015) reported that the competitive environment of a serious game improved students’ motivation and post-test scores (17). Conversely, Chen et al. (2018) revealed that students in the non-competition arm learned better than in the competition arm (18). The competitive element of a game may discourage some students (19). Therefore, we decided to remove the competitive element of the game to provide the less confident students with the opportunity for trial and error.
It is important to note that an increase in the students’ scores has not been a consistent measure for evaluating the efficacy of educational games (20). Instead, the games have generally been used to motivate the safe practice of students in an attractive environment, and it does not necessarily lead to an increase in the students’ learning.
This study showed no significant increase in retention scores by playing the game. There are at least two possible explanations for this finding. First, the time interval between playing the game and assessing the knowledge retention (the final exam) was short. Although the ideal time interval for examining knowledge retention is unclear, longer intervals have been used in similar studies. For example, Hu et al. (2021) developed a serious game for teaching COVID-19 lesson to medical students in China (21). They used a 5-week interval between playing the game and the final test and found that game users scored significantly higher than those who only participated in an online lecture. Similarly, in the studies that evaluated the effect of non-game learning methods on knowledge retention, longer intervals of three months to more than one year have been considered (22, 23). The short interval in our study was due to the time-consuming process of game development and validation phases, which made the short interval time inevitable. The second explanation, as mentioned above, was the inability to rule out all cases of students cheating in the final exam.
The second objective of the study was to evaluate the quality of the game by using a modified version of the MEEGA+ questionnaire. As shown in Appendix 1, the game was acceptable to the students in almost all dimensions. Of note, the median score of respondents to the question “This game is an adequate teaching method for this course” was zero. This score was not surprising because the game consisted of only 25 stages (questions) on diabetes, and it was not intended to cover the entire topic of diabetes. As discussed previously, the primary purpose of educational games is to practice the previous knowledge in a safe environment, not to use them as a complete educational method. This is reflected by the finding that about two-thirds of the participants would recommend the game to other students, probably complementing the traditional lecture-based teaching method. Given that more than 70% of the respondents “strongly agreed” or “agreed” with several dimensions of the game’s quality, it can be suggested that its overall quality was acceptable.
Interestingly, compared with other participants, regular gamers were more likely to prefer learning through GlucoGame over other teaching methods (72.72% of regular gamers vs. 34.21% of those who play less frequently). In fact, the same teaching technique cannot be used for all students, and each student may prefer a particular method of education based on their attitudes and interests. Our finding suggests that game-based learning may be helpful as a complementary teaching method for students who play other digital games regularly.
Another important finding of the study was that the aesthetic dimensions of the game were endorsed by students who regularly played other digital games. This finding is important because these students are familiar with the design and interface of digital games, which further supports the appropriate and attractive design of the game environment.
The findings in the current study are subject to at least three limitations. Firstly, the game group was compared with a multimedia group that used case-based education material. Therefore, the current investigation was limited by the lack of a control group who had only participated in lecture-based learning. A future study investigating the effect of the game- and multimedia-based learning compared with a control group would be helpful. Secondly, in future investigations, it might be possible to use a longer interval to evaluate retention. Thirdly, the majority of the students in the game group stated that they were not regular game players, which may have affected the study results.
Conclusion
An innovative game, GlucoGame, was developed and used by fourth-year pharmacy students. Compared with the multimedia file, although the use of the game did not increase the students’ knowledge scores, its quality was confirmed by the majority of the students.
Ethical considerations
The study design was reviewed and approved by the Ethics Committee of the Smart (Virtual) University of Medical Sciences (IR.VUMS.REC.1400.029). At the beginning of the study, the objectives of the research were explained to all students and they were assured that their information would remain confidential.
Acknowledgement
This study was funded by Smart (Virtual) University of Medical Sciences (grant number: 170-91).
Disclosure
None
Author Contributions
SE, MM, and SN were involved in idea generation. ER prepared the game and all authors contributed to its evaluation. MM and SE conducted data analysis and prepared the initial draft. All authors read and approved the manuscript draft.
Availability of data and materials
The datasets used in the study are available upon reasonable request.
Appendix 1.
Results of the quality assessment of game and multimedia based on the modified MEEGA+ questionnaire (n=99)
Please see Appendix 1 in the PDF file.
Full-Text: (833 Views)
Abstract
Background & Objective: The objectives of this study were to design, implement, and evaluate an educational game on the diagnosis and management of diabetes for pharmacy students.
Materials & Methods: Educational content for diabetes was prepared and used to design a multimedia file and game scenario. The game was validated and consisted of 25 stages of diagnosing and managing diabetes. Participants were 8th-semester pharmacy students who were divided into two groups. Groups A and B had 10-day access to the multimedia and game files, respectively. Pretest and post-test were held to evaluate students’ learning, and the final exam was used to assess retention. Moreover, user experience was evaluated by a modified and validated version of the Model for the Evaluation of Educational Games questionnaire.
Results: A total of 154 students participated in the study. The mean ranks were 69.25 and 60.99 (P=0.808) in the pretest; 64.80 and 67.63 (P=0.551) in the post-test; and 67.44 and 62.19 (P=0.490) in the final exam for groups A and B, respectively. Ninety-nine participants completed the quality questionnaire of the study. The students confirmed the quality of the game in nearly all quality dimensions. About 80 percent of the game users preferred it over usual teaching methods.
Conclusion: A high-quality educational game for diabetes was designed and implemented. Although the use of the game was not associated with increased learning or retention scores, pharmacy students preferred it over traditional teaching methods.
Background & Objective: The objectives of this study were to design, implement, and evaluate an educational game on the diagnosis and management of diabetes for pharmacy students.
Materials & Methods: Educational content for diabetes was prepared and used to design a multimedia file and game scenario. The game was validated and consisted of 25 stages of diagnosing and managing diabetes. Participants were 8th-semester pharmacy students who were divided into two groups. Groups A and B had 10-day access to the multimedia and game files, respectively. Pretest and post-test were held to evaluate students’ learning, and the final exam was used to assess retention. Moreover, user experience was evaluated by a modified and validated version of the Model for the Evaluation of Educational Games questionnaire.
Results: A total of 154 students participated in the study. The mean ranks were 69.25 and 60.99 (P=0.808) in the pretest; 64.80 and 67.63 (P=0.551) in the post-test; and 67.44 and 62.19 (P=0.490) in the final exam for groups A and B, respectively. Ninety-nine participants completed the quality questionnaire of the study. The students confirmed the quality of the game in nearly all quality dimensions. About 80 percent of the game users preferred it over usual teaching methods.
Conclusion: A high-quality educational game for diabetes was designed and implemented. Although the use of the game was not associated with increased learning or retention scores, pharmacy students preferred it over traditional teaching methods.
Introduction
Pharmacists are the most accessible healthcare providers and are appropriately placed to contribute to the diagnosis, management, and follow-up of diabetic patients (1). The role of pharmacists in the healthcare system has changed dramatically from drug dispensers to pharmacotherapy consultants. Therefore, the methods of pharmacy education should be changed following this role modification. The classic lecture-based teaching does not meet the current needs, and active learning is required to prepare pharmacy students for higher-level services (2).
Several learning methods have been investigated to involve the students in an active learning process. Active learning is defined as any educational method that involves students in the learning course (3). These include but are not limited to the flipped classroom, team-based learning, case-based learning, and audio response systems (4, 5). As emphasized in Knowles’ theory, adult learning is problem-oriented rather than content-based. This method allows them to apply their knowledge practically (6).
Educational games are one of the promising techniques of active learning in which the players learn new skills, simulate real situations, and develop their social and emotional abilities. The purpose of these games is not entertainment but to enhance the ability of the players to use their knowledge in an integrated fashion. It provides an opportunity for the safe practice of real-life scenarios without any concern. During the game, the students actively participate in the learning process, practice critical thinking, and learn from their mistakes (4). The Academic Affairs Committee also endorsed the use of serious games by faculty members and students for learning and professional development (7).
Diabetes games have been used to educate patients and healthcare providers, including pharmacy students. The outcomes measured by these studies mainly included the learning and confidence level of students while providing care for patients (8-10). In a study by Sando and Feng (2018) at the University of Florida, the use of an online spaced education game improved students’ engagement (11). Eukel et al. (2017) presented a diabetes-themed escape room designed for pharmacy students at Washington State University. Their study evaluated the game’s effectiveness in improving students’ diabetes knowledge and found a significant improvement in pre- and post-game knowledge scores (12).
The current study investigated the design and evaluation of an innovative game, GlucoGame, for educating regarding diabetes to pharmacy students at Tehran University of Medical Sciences (TUMS), Tehran, Iran. The main questions addressed in this research are: (a) to explore the effect of using GlucoGame on learning and retention in pharmacy students, and (b) to evaluate the quality of the game by a modified version of the Model for the Evaluation of Educational Games (MEEGA+) systematic model (13).
Pharmacists are the most accessible healthcare providers and are appropriately placed to contribute to the diagnosis, management, and follow-up of diabetic patients (1). The role of pharmacists in the healthcare system has changed dramatically from drug dispensers to pharmacotherapy consultants. Therefore, the methods of pharmacy education should be changed following this role modification. The classic lecture-based teaching does not meet the current needs, and active learning is required to prepare pharmacy students for higher-level services (2).
Several learning methods have been investigated to involve the students in an active learning process. Active learning is defined as any educational method that involves students in the learning course (3). These include but are not limited to the flipped classroom, team-based learning, case-based learning, and audio response systems (4, 5). As emphasized in Knowles’ theory, adult learning is problem-oriented rather than content-based. This method allows them to apply their knowledge practically (6).
Educational games are one of the promising techniques of active learning in which the players learn new skills, simulate real situations, and develop their social and emotional abilities. The purpose of these games is not entertainment but to enhance the ability of the players to use their knowledge in an integrated fashion. It provides an opportunity for the safe practice of real-life scenarios without any concern. During the game, the students actively participate in the learning process, practice critical thinking, and learn from their mistakes (4). The Academic Affairs Committee also endorsed the use of serious games by faculty members and students for learning and professional development (7).
Diabetes games have been used to educate patients and healthcare providers, including pharmacy students. The outcomes measured by these studies mainly included the learning and confidence level of students while providing care for patients (8-10). In a study by Sando and Feng (2018) at the University of Florida, the use of an online spaced education game improved students’ engagement (11). Eukel et al. (2017) presented a diabetes-themed escape room designed for pharmacy students at Washington State University. Their study evaluated the game’s effectiveness in improving students’ diabetes knowledge and found a significant improvement in pre- and post-game knowledge scores (12).
The current study investigated the design and evaluation of an innovative game, GlucoGame, for educating regarding diabetes to pharmacy students at Tehran University of Medical Sciences (TUMS), Tehran, Iran. The main questions addressed in this research are: (a) to explore the effect of using GlucoGame on learning and retention in pharmacy students, and (b) to evaluate the quality of the game by a modified version of the Model for the Evaluation of Educational Games (MEEGA+) systematic model (13).
Materials & Methods
Design and settings(s)
After reviewing the literature, a focus group consisting of clinical pharmacists and educational technology experts was formed. The group members discussed the educational needs, current gaps, game design, and the study structure. A parallel-group design was used to evaluate the game’s impact on learning and retention.
Participants and sampling
The Doctor of Pharmacy is a 6-year (12 semesters) program in Iran, and our study was conducted on fourth-year pharmacy students at the TUMS Faculty of Pharmacy. “Diabetes” is one of the topics of the 8th semester of the pharmacy curriculum at TUMS.
A total of 154 students were recruited for the study, which included 77 participants in each of the study arms. Twelve students of the game group and one student of the multimedia group did not complete the study and were excluded from the analysis.
Tools/instruments & Data collection methods
The study consisted of three phases. During phase one, the educational content and the game scenario were prepared, and the game was developed. Students used the game in phase two, and finally, in phase three, the effect of the game on learning and retention was assessed. Figure 1 shows an overview of the study phases.
Design and settings(s)
After reviewing the literature, a focus group consisting of clinical pharmacists and educational technology experts was formed. The group members discussed the educational needs, current gaps, game design, and the study structure. A parallel-group design was used to evaluate the game’s impact on learning and retention.
Participants and sampling
The Doctor of Pharmacy is a 6-year (12 semesters) program in Iran, and our study was conducted on fourth-year pharmacy students at the TUMS Faculty of Pharmacy. “Diabetes” is one of the topics of the 8th semester of the pharmacy curriculum at TUMS.
A total of 154 students were recruited for the study, which included 77 participants in each of the study arms. Twelve students of the game group and one student of the multimedia group did not complete the study and were excluded from the analysis.
Tools/instruments & Data collection methods
The study consisted of three phases. During phase one, the educational content and the game scenario were prepared, and the game was developed. Students used the game in phase two, and finally, in phase three, the effect of the game on learning and retention was assessed. Figure 1 shows an overview of the study phases.
Phase one- Preparation and validation of educational material and game scenario
In the first step, educational content for diabetes was prepared by a three-member panel of clinical pharmacists with sufficient knowledge and experience in diabetes management. This content was based on the latest edition of the reference book “Applied therapeutics: the clinical use of drugs” (14), supplemented with essential highlights of the 2020 American Diabetes Association guideline (15). Three sets of educational materials were then prepared based on this content. The first included the theoretical aspects of the diagnosis and management of diabetes that were required by the Iranian pharmacy curriculum. This material was used to educate all students during a four-hour webinar, regardless of whether they would participate in the study. The second and third materials were problem-based content used to prepare the multimedia and game files and were intended to enhance the students’ problem-solving capabilities. Because the game was designed with real cases, it was inherently different from the material used for the webinar. If the study had only compared the game to a control group, any observed effect on student’s learning and retention would have been due to the game’s case-based nature rather than the game itself. To avoid this, a parallel-group, the “multimedia” arm, with a case-based structure was included in the study. By doing so, any effect of the game on student’s learning and retention could be attributed to the “nature of the game” rather than the case-based element alone.
The multimedia file included educational material and clinical cases on diabetes and was created by Articulate Storyline v.3.12. The game scenario consisted of 25 questions (stages) in four areas; diagnosis of diabetes, oral hypoglycemic agents, management of hypoglycemia, and insulin therapy. Afterward, the storyboard and wireframe of the game were created by an educational technology specialist.
The initial version of the game, GlucoGame, was developed by the iSpring Suite for both Windows and Android platforms. The clinical pharmacist panel played it and provided necessary feedback to the game developer. Three education technology experts evaluated the content validity of the game. The final version included 25 stages in the form of puzzles, drag-and-drop, matching, and multiple-choice questions. In each stage, an audio guide was placed that explained the related scientific basics, and the students could listen to it before playing that stage. The players could proceed to the next step only if they did the game correctly. In case of a wrong answer, it was possible to change it. The game environment is shown in Figure 2.
Figure 2. The user interface of GlucoGame.
Phase two- Implementation of the game
All 8th-semester students participated in a four-hour lecture-based webinar in two sessions. After that, students who wished to participate in the study were allocated randomly to groups A or B based on their ID numbers. They were excluded if they did not want to continue the study or did not complete the pretest and post-test.
For group A, the multimedia file, and for group B, the educational game file was uploaded to the Navid Learning Management System of TUMS. Both groups had access to their uploaded files for ten days.
Two incentives were considered to limit material sharing between the study groups. First, the students were informed that all participants would have access to all materials at the end of the study. Second, all participants would be given the total score on the diabetes topic in the final exam, irrespective of their actual scores.
Phase three- Evaluation and outcome measurement
Participants were required to take a pretest after the webinar. A post-test was given after the 10-day access to the multimedia or game files to evaluate the students’ learning. Both the pretest and post-test consisted of 15 multiple-choice questions with similar taxonomy on the diagnosis and management of diabetes.
To evaluate the retention, participants in both groups were asked to take the final exam without studying. The final exam (out of 10 scores) was designed as multiple-choice questions with a taxonomy similar to the pretest and post-test. It was designed based on the game and multimedia content and was given one week after the 10-day period. The pretest, post-test, and final exams were all held online.
The MEEGA+ questionnaire, a model developed by Petri et al., measures the game quality in terms of two broad areas of player experience and usability, each consisting of several dimensions (13). This instrument is available in some non-English languages. To assess the game quality, the questionnaire was translated into Persian. Because the players’ interaction was not possible in GlucoGame, some of the MEEGA+ items were removed, and a modified questionnaire was provided to the participants. Eight experts evaluated the adopted version, and its content validity was confirmed by a content validity ratio of 0.93 and a content validity index of 0.90. The internal consistency of the questionnaire was verified by a Cronbach’s alpha equal to 0.88.
The quality of the multimedia file was also evaluated in terms of user experience and usability. To facilitate a comparison between the game and multimedia files, questions similar to the modified MEEGA+ questionnaire were used with appropriate modifications. Moreover, some dimensions, such as “fun” and “operability” did not apply to the multimedia content. These questions were omitted from the multimedia questionnaire.
The age and gender of respondents to both questionnaires were recorded. In addition, respondents to the modified MEEGA+ questionnaire were asked about the frequency of playing digital and non-digital games on a 5-point Likert scale: “never”; “rarely: from time to time”; “monthly: at least once a month”; “weekly: at least once a week”; and “every day”.
Data analysis
Data analysis was conducted by IBM SPSS Statistics (version 26). Qualitative variables were reported as frequencies and percentages. Data collected for each quality factor was reported in terms of central tendency (median) and frequency distribution (bar graphs). The normal distribution of quantitative variables was examined by using the Shapiro-Wilk test. Due to non-normal distribution of quantitative variables, the Wilcoxon Signed-Rank Test was used to compare the pretest and post-test results within each group. In addition, the Mann-Whitney U test was used to compare the results of the pretest, post-test, and final exam between the study groups. A P-value less than 0.05 was considered statistically significant.
Results
In the first set of analyses, the effect of game-based and multimedia-based learning on the students’ learning and retention was examined.
As indicated in Table 1, the two study groups showed no significant differences in terms of pretest, post-test, and final exam scores. A within-group analysis was conducted by comparing pretest and post-test scores in each study arm. In the multimedia group, the use of multimedia did not result in a significant change in students’ scores (Z=-1.962, P=0.050). Similarly, the use of the game was not associated with a significant change in students’ scores (Z=-1.603, P=0.109).
All 8th-semester students participated in a four-hour lecture-based webinar in two sessions. After that, students who wished to participate in the study were allocated randomly to groups A or B based on their ID numbers. They were excluded if they did not want to continue the study or did not complete the pretest and post-test.
For group A, the multimedia file, and for group B, the educational game file was uploaded to the Navid Learning Management System of TUMS. Both groups had access to their uploaded files for ten days.
Two incentives were considered to limit material sharing between the study groups. First, the students were informed that all participants would have access to all materials at the end of the study. Second, all participants would be given the total score on the diabetes topic in the final exam, irrespective of their actual scores.
Phase three- Evaluation and outcome measurement
Participants were required to take a pretest after the webinar. A post-test was given after the 10-day access to the multimedia or game files to evaluate the students’ learning. Both the pretest and post-test consisted of 15 multiple-choice questions with similar taxonomy on the diagnosis and management of diabetes.
To evaluate the retention, participants in both groups were asked to take the final exam without studying. The final exam (out of 10 scores) was designed as multiple-choice questions with a taxonomy similar to the pretest and post-test. It was designed based on the game and multimedia content and was given one week after the 10-day period. The pretest, post-test, and final exams were all held online.
The MEEGA+ questionnaire, a model developed by Petri et al., measures the game quality in terms of two broad areas of player experience and usability, each consisting of several dimensions (13). This instrument is available in some non-English languages. To assess the game quality, the questionnaire was translated into Persian. Because the players’ interaction was not possible in GlucoGame, some of the MEEGA+ items were removed, and a modified questionnaire was provided to the participants. Eight experts evaluated the adopted version, and its content validity was confirmed by a content validity ratio of 0.93 and a content validity index of 0.90. The internal consistency of the questionnaire was verified by a Cronbach’s alpha equal to 0.88.
The quality of the multimedia file was also evaluated in terms of user experience and usability. To facilitate a comparison between the game and multimedia files, questions similar to the modified MEEGA+ questionnaire were used with appropriate modifications. Moreover, some dimensions, such as “fun” and “operability” did not apply to the multimedia content. These questions were omitted from the multimedia questionnaire.
The age and gender of respondents to both questionnaires were recorded. In addition, respondents to the modified MEEGA+ questionnaire were asked about the frequency of playing digital and non-digital games on a 5-point Likert scale: “never”; “rarely: from time to time”; “monthly: at least once a month”; “weekly: at least once a week”; and “every day”.
Data analysis
Data analysis was conducted by IBM SPSS Statistics (version 26). Qualitative variables were reported as frequencies and percentages. Data collected for each quality factor was reported in terms of central tendency (median) and frequency distribution (bar graphs). The normal distribution of quantitative variables was examined by using the Shapiro-Wilk test. Due to non-normal distribution of quantitative variables, the Wilcoxon Signed-Rank Test was used to compare the pretest and post-test results within each group. In addition, the Mann-Whitney U test was used to compare the results of the pretest, post-test, and final exam between the study groups. A P-value less than 0.05 was considered statistically significant.
Results
In the first set of analyses, the effect of game-based and multimedia-based learning on the students’ learning and retention was examined.
As indicated in Table 1, the two study groups showed no significant differences in terms of pretest, post-test, and final exam scores. A within-group analysis was conducted by comparing pretest and post-test scores in each study arm. In the multimedia group, the use of multimedia did not result in a significant change in students’ scores (Z=-1.962, P=0.050). Similarly, the use of the game was not associated with a significant change in students’ scores (Z=-1.603, P=0.109).
Table 1. Comparison of the study groups in terms of pretest, post-test, and final exam scores
| Multimedia group | Game group | P-value | |||
| median (range) | mean rank | median (range) | mean rank | ||
| Pretest | 12.00 (6.00-15.00) | 69.25 | 11.00 (5.00-15.00) | 60.99 | 0.808 |
| Post-test | 11.0 (8.00-15.00) | 64.80 | 12.00 (2.00-15.00) | 67.63 | 0.551 |
| Final exam | 5.00 (1.00-8.00) | 67.44 | 5.00 (1.00-10.00) | 62.19 | 0.490 |
| p-value† | 0.050 | 0.109 | |||
| †This P-value is the result of comparing the pretest and post-test scores within each of the study groups. | |||||
The second series of analyses included using the modified version of the MEEGA+ systematic model to evaluate the quality of game-based and multimedia-based learning. The questionnaires were completed by 50 students from the multimedia group (23 males and 27 females) and 49 students from the game group (14 males and 35 females).
Twenty-eight out of 35 (80.00%) female respondents to the game questionnaire stated that they had “never played” or “rarely played” other digital games. Five of the 14 males (35.71%) reported this rate of digital game playing. Eleven students in the game group stated that they play other digital games daily or several times a week; of them, 8 (72.72%) preferred learning through GlucoGame over other learning methods, such as multimedia or lectures. On the other hand, 34.21% (13 out of 38) of students with lower frequencies of digital game playing preferred Gluco Game over other learning methods.
Appendix 1 presents an overview of the participants’ opinions on various game dimensions and multimedia quality. Common questions of the game and multimedia questionnaires are presented sequentially to facilitate comparisons. As Appendix 1 displays, nearly all dimensions of the game quality were acceptable to the students.
Around 40 (81.63%) respondents to the game and 44 (88.00%) respondents to the multimedia questionnaires believed that the playing the game or multimedia assured them that they could learn from it. A total of 39 out of 49 respondents (79.59%) to the game questionnaire preferred the game over the usual teaching classes, and two-thirds (33 out of 49, 67.34%) would recommend it to other students. Only 24.48 % (12 out of 49) of the respondents believed that using the game alone was enough to learn the course, while the corresponding value was 66.00% (33 out of 50) for the multimedia content.
Discussion
This study aimed to design and implement an innovative game on the education of diabetes and evaluate its effect on the learning and retention of pharmacy students.
In the current study, the observed change in the students’ scores did not reach statistical significance. This finding is consistent with those of Sando and Feng, who evaluated the effect of an online game of drug information on the students’ knowledge scores (11). Although the students’ level of engagement with the game was high and acceptable, the percentages of correctly answered questions did not increase significantly. In another study, Auman (2011) used a simulation game to increase the students’ engagement in a psychology course (16). The result showed that the change in the post-test score compared to the pretest was not significantly different between the simulation and non-simulation groups. In contrast, in the study by Eukel et al., the mean post-test score was 81%, which was significantly higher than the mean pretest score of 56% (P<0.01) (12). The lack of significant change in the knowledge scores of our participants may be explained by several reasons. Firstly, the parallel group also used the multimedia file that reviewed all the aspects of diabetes diagnosis and management on a case-based method. As shown in Appendix 1, the multimedia file also received an acceptable score from the students regarding quality. Secondly, our study could not control students cheating. Although the students’ IP addresses were controlled to find the same ones, this method could not identify all cheating cases. Thirdly, one of the measures taken to limit material sharing between the study arms was to give the total score of the final exam to all students, regardless of their actual scores. It may have reduced students’ competitiveness. Previous studies showed conflicting results on the role of competitiveness in serious games. Cagiltay et al. (2015) reported that the competitive environment of a serious game improved students’ motivation and post-test scores (17). Conversely, Chen et al. (2018) revealed that students in the non-competition arm learned better than in the competition arm (18). The competitive element of a game may discourage some students (19). Therefore, we decided to remove the competitive element of the game to provide the less confident students with the opportunity for trial and error.
It is important to note that an increase in the students’ scores has not been a consistent measure for evaluating the efficacy of educational games (20). Instead, the games have generally been used to motivate the safe practice of students in an attractive environment, and it does not necessarily lead to an increase in the students’ learning.
This study showed no significant increase in retention scores by playing the game. There are at least two possible explanations for this finding. First, the time interval between playing the game and assessing the knowledge retention (the final exam) was short. Although the ideal time interval for examining knowledge retention is unclear, longer intervals have been used in similar studies. For example, Hu et al. (2021) developed a serious game for teaching COVID-19 lesson to medical students in China (21). They used a 5-week interval between playing the game and the final test and found that game users scored significantly higher than those who only participated in an online lecture. Similarly, in the studies that evaluated the effect of non-game learning methods on knowledge retention, longer intervals of three months to more than one year have been considered (22, 23). The short interval in our study was due to the time-consuming process of game development and validation phases, which made the short interval time inevitable. The second explanation, as mentioned above, was the inability to rule out all cases of students cheating in the final exam.
The second objective of the study was to evaluate the quality of the game by using a modified version of the MEEGA+ questionnaire. As shown in Appendix 1, the game was acceptable to the students in almost all dimensions. Of note, the median score of respondents to the question “This game is an adequate teaching method for this course” was zero. This score was not surprising because the game consisted of only 25 stages (questions) on diabetes, and it was not intended to cover the entire topic of diabetes. As discussed previously, the primary purpose of educational games is to practice the previous knowledge in a safe environment, not to use them as a complete educational method. This is reflected by the finding that about two-thirds of the participants would recommend the game to other students, probably complementing the traditional lecture-based teaching method. Given that more than 70% of the respondents “strongly agreed” or “agreed” with several dimensions of the game’s quality, it can be suggested that its overall quality was acceptable.
Interestingly, compared with other participants, regular gamers were more likely to prefer learning through GlucoGame over other teaching methods (72.72% of regular gamers vs. 34.21% of those who play less frequently). In fact, the same teaching technique cannot be used for all students, and each student may prefer a particular method of education based on their attitudes and interests. Our finding suggests that game-based learning may be helpful as a complementary teaching method for students who play other digital games regularly.
Another important finding of the study was that the aesthetic dimensions of the game were endorsed by students who regularly played other digital games. This finding is important because these students are familiar with the design and interface of digital games, which further supports the appropriate and attractive design of the game environment.
The findings in the current study are subject to at least three limitations. Firstly, the game group was compared with a multimedia group that used case-based education material. Therefore, the current investigation was limited by the lack of a control group who had only participated in lecture-based learning. A future study investigating the effect of the game- and multimedia-based learning compared with a control group would be helpful. Secondly, in future investigations, it might be possible to use a longer interval to evaluate retention. Thirdly, the majority of the students in the game group stated that they were not regular game players, which may have affected the study results.
Conclusion
An innovative game, GlucoGame, was developed and used by fourth-year pharmacy students. Compared with the multimedia file, although the use of the game did not increase the students’ knowledge scores, its quality was confirmed by the majority of the students.
Ethical considerations
The study design was reviewed and approved by the Ethics Committee of the Smart (Virtual) University of Medical Sciences (IR.VUMS.REC.1400.029). At the beginning of the study, the objectives of the research were explained to all students and they were assured that their information would remain confidential.
Acknowledgement
This study was funded by Smart (Virtual) University of Medical Sciences (grant number: 170-91).
Disclosure
None
Author Contributions
SE, MM, and SN were involved in idea generation. ER prepared the game and all authors contributed to its evaluation. MM and SE conducted data analysis and prepared the initial draft. All authors read and approved the manuscript draft.
Availability of data and materials
The datasets used in the study are available upon reasonable request.
Appendix 1.
Results of the quality assessment of game and multimedia based on the modified MEEGA+ questionnaire (n=99)
Please see Appendix 1 in the PDF file.
Article Type : Orginal Research |
Subject:
Medical Education
Received: 2022/10/10 | Accepted: 2023/05/10 | Published: 2023/08/15
Received: 2022/10/10 | Accepted: 2023/05/10 | Published: 2023/08/15
References
1. David Hughes J, wibowo Y, Sunderland B, Hoti K. The role of the pharmacist in the management of type 2 diabetes: current insights and future directions. Integrated Pharmacy Research and Practice. 2017;6:15-27. [DOI]
2. Kiersma ME, Plake KS, Newton GD, Mason HL. Factors affecting prepharmacy students' perceptions of the professional role of pharmacists. American Journal of Pharmaceutical Education. 2010;74(9):161. [DOI]
3. Crisol-Moya E, Romero-López MA, Caurcel-Cara MJ. Active Methodologies in Higher Education: Perception and Opinion as Evaluated by Professors and Their Students in the Teaching-Learning Process. Frontiers in Psychology. 2020;11(1703). [DOI]
4. Aburahma M, Mohamed H. Educational Games as a Teaching Tool in Pharmacy Curriculum. American journal of Pharmaceutical Education. 2015;79 (4):59. [DOI]
5. Beigzadeh A, Haghani A. Active learning methods: a way of tackling large classroom setting. Strides in Development of Medical Education. 2016;13(1):107-13.
6. Akl EA, Sackett KM, Erdley WS, et al. Educational games for health professionals. Cochrane Database Systematic Review. 2013(1):Cd006411. [DOI]
7. Cain J, Piascik P. Are Serious Games a Good Strategy for Pharmacy Education? American Journal of Pharmaceutical Education. 2015;79(4):47. [DOI]
8. Wongwiwatthananukit S, Zeszotarski P, Thai A, et al. A training program for pharmacy students on providing diabetes care. American journal of Pharmaceutical Education. 2013;77(7):153. [DOI]
9. Shrader S, Kavanagh K, Thompson A. A Diabetes Self-Management Education Class Taught by Pharmacy Students. American Journal of Pharmaceutical Education. 2012;76:13. [DOI]
10. Morello CM, Neighbors M, Luu L, Kobayashi S, Mutrux B, Best BM. Impact of a First-Year Student Pharmacist Diabetes Self-Care Education Program. American Journal of Pharmaceutical Education. 2013;77(10):215. [DOI]
11. Sando KR, Feng X. Use of an Online Spaced-Education Game to Study Top 200 Drugs in a Skills Laboratory Course. American Journal of Pharmaceutical Education. 2018;82(7):6324. [DOI]
12. Eukel HN, Frenzel JE, Cernusca D. Educational Gaming for Pharmacy Students - Design and Evaluation of a Diabetes-themed Escape Room. American Journal of Pharmaceutical Education. 2017;81(7):6265. [DOI]
13. Petri G, Gresse von Wangenheim C, Borgatto AF. MEEGA+, Systematic Model to Evaluate Educational Games. In: Lee N, editor. Encyclopedia of Computer Graphics and Games. Cham: Springer International Publishing; 2017;1-7.
14. Jennifer D. Goldman, Dhiren K. Patel, Schnee D. Diabetes Mellitus. In: Caroline S. Zeind, Carvalho MG, editors. Applied therapeutics: the clinical use of drugs. 11th ed: Wolters Kluwer; 2018; 1071-147.
15. American Diabetes Association. Standards of Medical Care in Diabetes-2019 Abridged for Primary Care Providers. Clinical Diabetes. 2019;37(1):11-34. . [DOI]
16. Auman C. Using Simulation Games to Increase Student and Instructor Engagement. College Teaching. 2011;59:154-61. [DOI]
17. Cagiltay N, Ozcelik E, Ozcelik N. The effect of competition on learning in games. Computers & Education. 2015;87:35-41. [DOI]
18. Chen C-H, Liu J-H, Shou W-C. How Competition in a Game-based Science Learning Environment Influences Students' Learning Achievement, Flow Experience, and Learning Behavioral Patterns. Journal of Educational Technology & Society. 2018;21(2):164-76.
19. Bangalee V, Oosthuizen F, Perumal-Pillay VA, Suleman F, Walters F. Pharmacy students experience with PharmacyPhlash - a pilot educational board game. Currents in Pharmacy Teaching and Learning. 2021;13(3):292-301. [DOI]
20. Mandana I, Kaveh E, Razie Jamshidi A, Leila K. Designing PharmQuiz Educational Game Application to Facilitate Learning of Pharmaceutical Information for Pharmacy Students. Journal of Pharmaceutical Care. 2021;9(1). [DOI]
21. Hu H, Xiao Y, Li H. The Effectiveness of a Serious Game Versus Online Lectures for Improving Medical Students' Coronavirus Disease 2019 Knowledge. Games for Health Journal. 2021;10(2):139-44. [DOI]
22. Yo EC, Witjaksono AN, Fitriani DY, Werdhani RA, Parikesit D. Evaluating knowledge retention and perceived benefits of medical webinar for professional development among Indonesian physicians. Korean Journal of Medical Education. 2021;33(4):381-91. [DOI]
23. Pande S, Pande S, Parate V, Pande S, Sukhsohale N. Evaluation of retention of knowledge and skills imparted to first-year medical students through basic life support training. Advances in Physiology Education. 2014;38(1):42-5. [DOI]
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
