Sun, Jun 8, 2025
[Archive]
Volume 16, Issue 52 (2024)
JMED 2024, 16(52): 27-36 |
Back to browse issues page
Ethics code: (IEC/ HMPCMCE/ 114/faculty 13, 7/08/2020)
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Mehta T, Patel S, Kumar A, Trivedi B, Ganjiwale J, Chaudhary M et al . Development, implementation, and evaluation of structured Performa in the initial assessment of sick children at the emergency department: A third level Kirkpatrick’s evaluation model. JMED 2024; 16 (52) :27-36
URL: http://edujournal.zums.ac.ir/article-1-1899-en.html
URL: http://edujournal.zums.ac.ir/article-1-1899-en.html
Tanvi Mehta1 
, Shradha Patel2 , Amit Kumar2 , Bhadra Trivedi2 , Jaishree Ganjiwale3 , Manoj Chaudhary4 , Krutika Tandon *5
, Shradha Patel2 , Amit Kumar2 , Bhadra Trivedi2 , Jaishree Ganjiwale3 , Manoj Chaudhary4 , Krutika Tandon *5
1- Department of Pediatrics, SBKS Medical Institute and Research Center, Sumandeep Vidyapeeth, Pipariya, Waghodiya, Vadodara, Gujarat, India.
2- Department of Pediatrics, Pramukhswami Medical College, Bhaikaka University, Karamsad, Gujarat, India.
3- Department of Community Medicine, Pramukhswami Medical College, Bhaikaka University, Karamsad, Gujarat, India.
4- Department of Pediatrics, Birat Medical College Teaching Hospital, Kathmandu University, Buddhiganga-02, Morang, Nepal.
5- Department of Pediatrics, Pramukhswami Medical College, Bhaikaka University, Karamsad, Gujarat, India. ,tandonkrutika72@gmail.com
2- Department of Pediatrics, Pramukhswami Medical College, Bhaikaka University, Karamsad, Gujarat, India.
3- Department of Community Medicine, Pramukhswami Medical College, Bhaikaka University, Karamsad, Gujarat, India.
4- Department of Pediatrics, Birat Medical College Teaching Hospital, Kathmandu University, Buddhiganga-02, Morang, Nepal.
5- Department of Pediatrics, Pramukhswami Medical College, Bhaikaka University, Karamsad, Gujarat, India. ,
Full-Text [PDF 819 kb]
(395 Downloads)
| Abstract (HTML) (1328 Views)
Full-Text: (272 Views)
Abstract
Background & Objective: At emergency department (ED) at different point of time different teams are involved in management of patients as per their posting/duty roster. So, a robust system should be in place to avoid major mishaps. The current study was an attempt to improve pediatric care at our Emergency Department through third-level Kirkpatrick’s evaluation model after training of pediatric resident doctors to use newly developed structured proforma while doing initial assessment of sick children at ED.
Materials & Methods: This Quasi-experimental study included all departmental pediatric resident doctors and as per sample size calculation, 36 (pre-intervention) and 36 (post-intervention) computer-generated random selection records were reviewed from the total of serially arranged admissions of pediatric patients. The intervention was the implementation of a Structured proforma, and training of all departmental residents. All raters scored the records of patients on 47 & 51 items of documentation pre-and post-intervention periods respectively on a scale 0-2, 0 meaning ‘no mention’, 1-incomplete/improper mention and 2-complete/proper mention. Mean, standard deviation (SD) of scores were calculated item-wise, raters-wise and overall. Bland Altman analysis was done to find agreement in scoring among raters both in pre-and post-intervention.
Results: The mean (SD) and percentage of mean score were 32.93 (4.50) and 35.03% before intervention whereas 89.64 (4.35) and 87.88% post-intervention. This indicated 53.5% improvement post-intervention. Bland Altman analysis found good agreement post-intervention.
Conclusion: The introduction of Educational tool along with the training of pediatric resident doctors to implement it, has improved documentation process significantly.
Background & Objective: At emergency department (ED) at different point of time different teams are involved in management of patients as per their posting/duty roster. So, a robust system should be in place to avoid major mishaps. The current study was an attempt to improve pediatric care at our Emergency Department through third-level Kirkpatrick’s evaluation model after training of pediatric resident doctors to use newly developed structured proforma while doing initial assessment of sick children at ED.
Materials & Methods: This Quasi-experimental study included all departmental pediatric resident doctors and as per sample size calculation, 36 (pre-intervention) and 36 (post-intervention) computer-generated random selection records were reviewed from the total of serially arranged admissions of pediatric patients. The intervention was the implementation of a Structured proforma, and training of all departmental residents. All raters scored the records of patients on 47 & 51 items of documentation pre-and post-intervention periods respectively on a scale 0-2, 0 meaning ‘no mention’, 1-incomplete/improper mention and 2-complete/proper mention. Mean, standard deviation (SD) of scores were calculated item-wise, raters-wise and overall. Bland Altman analysis was done to find agreement in scoring among raters both in pre-and post-intervention.
Results: The mean (SD) and percentage of mean score were 32.93 (4.50) and 35.03% before intervention whereas 89.64 (4.35) and 87.88% post-intervention. This indicated 53.5% improvement post-intervention. Bland Altman analysis found good agreement post-intervention.
Conclusion: The introduction of Educational tool along with the training of pediatric resident doctors to implement it, has improved documentation process significantly.
Introduction
Healthcare systems have incorporated quality improvement methods to reduce errors, and costs, and improve access, safety, and outcomes of health care (1). However, gaps exist in the quality of care provided to children and the delivery of acute pediatric care is complex (2). At the tertiary care center in the emergency department (ED) in each point of time one team is there to look after the emergencies but teams do change frequently as per their posting and duty roster. If the proper system is not in place, then major mishaps are likely to happen. Many times, even documentation does not find proper and adds more problems especially if a vital piece of information or interpretation is missing while another team takes over which may put the pediatric patient at risk further. Though physicians give a lot of time to documentation there is hardly any research to examine its role. Recently, Moy AJ conducted a scoping review to know the approaches to documentation burden measurement and their characteristics. It was concluded the need for further research to operationalize the concept of documentation burden and to find best practices for measurement including standardization for its use (3). In one recent study, it was demonstrated that complex processes of work like documentation should be from the combination of data rather than a single data point (4). And so, it is an institutional focus rather than an individual focus.
Ours is a university teaching hospital that uses various quality indicators to screen, monitor, evaluate, and improve the quality of patient care as well as their outcome via various teaching-training programs. The Kirkpatrick Model is a globally recognized method of evaluating the results of training and learning programs (5). Earlier one study showed the successful use of the Kirkpatrick model in faculty development programs (6). There are also a few studies that were done on undergraduate students and resident doctors (7-11). The Kirkpatrick Model assesses both formal and informal training methods and rates them against four levels of criteria: reaction, learning, behavior, and results (5). The majority of the studies assessed Kirkpatrick’s level 1 and level 2 outcomes meaning satisfaction and knowledge/skill improvement, but the quality of published research remains poor beyond that and there is inadequate reporting of interventions like educational theory, curricula, pedagogy and requirement of resources as well as outcomes evaluation (Kirkpatrick levels). So, difficulties persisted in replicating it frequently or at a large scale (12).
Most quality indicators-related studies were from developed countries including few focused on the assessment of the quality of care delivered to pediatric patients in ED also. However, limited work has been done in this field in India. The first step of any quality improvement method is to know the baseline status and accordingly focus further. One common gap observed at our place was improper and inadequate documentation and lack of uniformity while attending pediatric patients at ED. To streamline the process of documentation in our place, we developed an educational tool in the form of Structured Proforma after reviewing the large numbers of items or indicators (13-17) which was implemented and evaluated subsequently to assess the degree of improvement.
Materials & Methods
Design and setting(s)
The study was conducted based on a quasi-experimental -one-group time series design at the Department of Pediatrics, Pramukhswami Medical College, Karamsad, Gujarat, India.
Participants and sampling
All 14 pediatric resident doctors were participants of the study whereas file records of 36-36 in pre- and post-intervention periods were chosen as per computer-generated random numbers. October to December 2019 was the pre-intervention phase whereas October to December 2020 was the post-intervention phase. Figure 1 shows the flow diagram of the study.
Due to the Covid-19 lockdown situation and pending query submission at the ethical committee, written approval was delayed. But on partial relaxation of lockdown during the Covid-19 situation, all departmental Pediatric Residents were trained in two batches by faculty guide offline. It included power point lecture on “How to evaluate critically ill children”, pre-sensitization with the status of documentation of pilot pre-intervention scoring maintaining the confidentiality and also how to use new structured performa while attending emergency calls. All were assured that there wouldn’t be any personal or academic harm to them irrespective of the post-intervention scoring or any mistake. Each session lasted for 75 minutes. It also emphasized the utmost benefit of overall improvement in patient management by doing adequate and proper documentation. All readily gave consent to participate in this study.
Tools/Instruments
The education tool used in this study was a newly designed structured performa for the initial evaluation of sick children at the ED. It was developed by subject experts of our department who are qualified pediatric intensivists. It included general information as well as many vital items or indicators after reviewing various modules like pediatric advanced life support (PALS), pediatric fundamental critical care support (PFCCS), basic pediatric intensive care course (BPICC), Indian academy of pediatrics, advanced life support, basic life support (IAP ALS BLS) or essential pediatric intensive care (EPIC) (13-17). structured proforma (Educational tool) was finalized after face validation and content validation by three Pediatric intensivists and one PICU fellow with final reliability was 0.85 according to Cronbach's alpha when 10 records checked as a pilot project. The tool was designed in such a way that postgraduate students must encircle or tick from provided options only. It was less time-consuming than conventional writing and had the least chance of missing any vital information. Apart from general information about a patient, there were main three columns: Evaluate, Identify, and Intervention. In each patient’s structured proforma, the resident is required to address each row, and under Identify column, they need to choose one option (encircle/tick) and based on that, the required intervention is to be mentioned under Intervene column. The pilot screening of 10 records suggested a difference of around a score of 6 among experts and the final version was developed adding 4 items/indicators with a total of 51 with significantly improved content validity.
As per the calculated sample size, 36 (pre-intervention) and 36 (post-intervention) computer-generated random selection records were reviewed from a total of serially arranged admissions of pediatric patients (1 month to 18 years of age) at ED whose call was attended by Pediatrics resident doctors. In the current study, the operational definition of accuracy and completeness of documentation was based on the scoring. Score 0 - no mention or an unattended portion, Score 1 - incomplete or inappropriate mention, and Score 2 - proper and complete mention. ACCURACY means each item was written what it should be as per Expert or Rater. COMPLETENESS means the number of items that have got a score of 2 out of the total items or indicators. There was a total of 47 items to score from 0-2 in the pre-intervention phase as per the first version of structured performa whereas the final version had a total of 51 items. So, the maximum score can be 94 in the first version and 102 in the final version. After 2 months of implementation of structured performa we evaluated records again, scored and entered them in Microsoft Excel files item-wise and patient-wise separately by all Raters which was finally compiled in a single Excel file and submitted to a statistician for analysis with blinding of names of Raters.
Data analysis
For the calculation of sample size WINPEPI software was used. The baseline proportion of completeness was 35% and the end-point proportion of completeness we wanted to achieve was 75% with 5% level of significance and 80% power. The calculated sample size was 72 (36 in the pre-intervention phase and 36 in the post-intervention phase) and records were selected by computer-generated random number. In both pre- and post-intervention phases, item-wise frequency, and percentages of score 0, score 1, and score 2 were derived. Also, Item-wise mean (SD) and overall mean (SD) were derived for both phases of the study. Rater-wise mean was calculated in both pre-and post-intervention phases and compared with each other.
We considered the difference in mean inter-rater scoring up to 2% as an acceptable value with acceptable confidence limits of 10%. So, in pre-intervention, the highest total score of 47 items was 94 hence upto 1.88 units (2% of 94) taken as an acceptable difference in mean scoring between the raters and upto 9.4 units (10% of 94) was considered as acceptable confidence limits of difference of scoring between the raters. Similarly in post-intervention, 51 items were included hence the highest total score was 102. So, the acceptable difference in mean scoring among the raters was upto 2 units (2% of 102), and upto 10.2 units (10% of 102) was considered as acceptable confidence limits of difference of scoring between the raters. The agreement between raters was good when the difference in mean scoring was less than 2% of the total score with confidence limits of ≤ 10% of the total score. Bland Altman’s analysis was done to find out the degree of agreement between the raters both in the pre-and post-intervention phases.
Results
Item-wise and Rater–wise comparisons of scoring of both pre- to post-intervention phases are shown in Table 1 and Table 2 respectively.
Table 2. Rater-wise comparison of mean scoring and percentage of mean (pre- to post-intervention)
In our study except for Rater 5 in the preintervention group, all other raters had a good inter-rater agreement in both the phase. The inference of inter-rater agreement in the phase is shown in Table 3.
Discussion
The most striking outcome of the current study was a tremendous improvement in the overall mean (SD) score to 89.64 (4.35) from the baseline of 32.93 (4.51) of an initial assessment of sick children evaluation notes at ED. The use of the third level of the Kirkpatrick evaluation model changed the practices of documentation by introducing education tools and by training pediatric residents. The Kirkpatrick model assesses both formal and informal training methods and rates them against four levels of criteria: reaction, learning, behavior, and results (5) It is a globally recognized method of evaluating the results of training and learning programs.
Earlier in 1999, for triage of patients in the Emergency Department, Emergency Severity Index (ESI) was introduced but pediatric criteria were added to it in 2000 (18,19). As residents play a major role in ensuring quality care for patients, introducing quality measures suitable for them in practice is desirable and in 2016 Schumacher et al (20) identified and developed Resident-sensitive quality measures (RSQMs) for the use in the Pediatric Emergency Department setting. However, we used parameters that were important documentation point of view. In the current study, in the pre-intervention phase, the items which were not mentioned the in majority of files were weight, appearance of child, breathing, color, airways, work of breathing, chest rise, saturation of oxygen(SPO2), cardiac rhythm, central vs peripheral pulses, extremities, capillary refill time(CRT), pupils (Brainstem function), oculocephalic movements, skin bruise/bleeds, allergies, medications- if any ongoing, past medical history, last meal taken, imaging studies in diagnostic tests and severity/life-threatening problem or severity of respiratory problem or type of respiratory problem or type of circulatory problem. But in the post-intervention phase the majority of the items improved significantly after training and implementation of structured proforma except height, SPO2 with 8L of oxygen with a non-rebreathing mask, oculocephalic movements, and blood sugar. The probable reason may be that the patient may not need oxygen and residents might have confusion that what to write and also in some situations they felt that blood sugar is not required. So, such items have scope for modification in future versions of structured proforma.
Kalet et al proposed the adoption of educationally sensitive patient outcomes (ESPOs) for training resident doctors. They described this as “patient outcomes that are sensitive to provider education, can be measured, and are in the pathway linking medical education interventions to patient outcomes” (21). They suggested two measures: patient activation (PA), a strong component of healthful behaviour change and clinical microsystem activation (CMSA), a major influence on patient safety and healthcare quality. However, in the present study this may be the limitation as patients ‘outcome of patients was not assessed along with the documentation improvement process but in a future study, it can be planned and done.
Adler et al did their work using Simulation techniques where they conducted a two-phase randomized control trial to develop and evaluate a simulation-based Pediatric Emergency Medicine Curriculum (22). In the validation phase (2006–2007), he randomized 69 residents from two EM residencies to either an intervention group that received the curriculum one month before the first assessment of all participants or a waitlist control group that received the identical curriculum three months later. A final assessment of all residents followed one month after that. Two raters evaluated all residents. Primary outcome measures were percentages of items completed correctly. They assessed rater agreement using intraclass correlation (ICC) and compared group performance using mixed-model analysis of variance, but they observed that a one-day, simulation-based pediatric EM curriculum produced limited results. In the current study, inter-rater agreement was used, and it was found that the introduction of structured proforma after training residents improved the desired result, and both in the pre-and post-intervention phases were fair to a good agreement among most raters. The main limitation of our study was that it evaluated only the process of documentation among the pediatric residents after training and not the outcome of patients based on it. Also, those few items need to be modified or removed which were not addressed even in the post-intervention phase during documentation.
Conclusion
Teaching and training Pediatrics residents while implementing the newly developed educational tool in the Emergency department has improved the documentation process significantly and there was fair to good agreement between the raters it is suggested to use at other places also and can be incorporated in digital format also in future.
Ethical considerations
The Institutional Ethics Committee of H M Patel Center of Medical Care and education approved this project. (IEC/ HMPCMCE/ 114/faculty 13, 7/08/2020) All students were adequately informed about the aims and methodology of the study before their participation. Written informed consent was obtained from participants. They were assured that no academic harm to them, their data would be kept confidential, and only general information and statistics would be released.
Acknowledgments
We thank all Resident doctors of Pediatrics who contributed to this study.
Disclosure
None to declare
Author contribution
T.N.M, B.T, A.K, S.P, M.K.C, K.R.T: Performed the study intervention and collected data. K.R.T: Designed and supervised the procedure. The data were analyzed by J.G who participated in planning the study and revising the manuscript. The manuscript was read and approved by all authors.
Data availability statement
Upon a reasonable request, the corresponding author can provide the data set that was analyzed during this study.
Ours is a university teaching hospital that uses various quality indicators to screen, monitor, evaluate, and improve the quality of patient care as well as their outcome via various teaching-training programs. The Kirkpatrick Model is a globally recognized method of evaluating the results of training and learning programs (5). Earlier one study showed the successful use of the Kirkpatrick model in faculty development programs (6). There are also a few studies that were done on undergraduate students and resident doctors (7-11). The Kirkpatrick Model assesses both formal and informal training methods and rates them against four levels of criteria: reaction, learning, behavior, and results (5). The majority of the studies assessed Kirkpatrick’s level 1 and level 2 outcomes meaning satisfaction and knowledge/skill improvement, but the quality of published research remains poor beyond that and there is inadequate reporting of interventions like educational theory, curricula, pedagogy and requirement of resources as well as outcomes evaluation (Kirkpatrick levels). So, difficulties persisted in replicating it frequently or at a large scale (12).
Most quality indicators-related studies were from developed countries including few focused on the assessment of the quality of care delivered to pediatric patients in ED also. However, limited work has been done in this field in India. The first step of any quality improvement method is to know the baseline status and accordingly focus further. One common gap observed at our place was improper and inadequate documentation and lack of uniformity while attending pediatric patients at ED. To streamline the process of documentation in our place, we developed an educational tool in the form of Structured Proforma after reviewing the large numbers of items or indicators (13-17) which was implemented and evaluated subsequently to assess the degree of improvement.
Materials & Methods
Design and setting(s)
The study was conducted based on a quasi-experimental -one-group time series design at the Department of Pediatrics, Pramukhswami Medical College, Karamsad, Gujarat, India.
Participants and sampling
All 14 pediatric resident doctors were participants of the study whereas file records of 36-36 in pre- and post-intervention periods were chosen as per computer-generated random numbers. October to December 2019 was the pre-intervention phase whereas October to December 2020 was the post-intervention phase. Figure 1 shows the flow diagram of the study.
Due to the Covid-19 lockdown situation and pending query submission at the ethical committee, written approval was delayed. But on partial relaxation of lockdown during the Covid-19 situation, all departmental Pediatric Residents were trained in two batches by faculty guide offline. It included power point lecture on “How to evaluate critically ill children”, pre-sensitization with the status of documentation of pilot pre-intervention scoring maintaining the confidentiality and also how to use new structured performa while attending emergency calls. All were assured that there wouldn’t be any personal or academic harm to them irrespective of the post-intervention scoring or any mistake. Each session lasted for 75 minutes. It also emphasized the utmost benefit of overall improvement in patient management by doing adequate and proper documentation. All readily gave consent to participate in this study.
Tools/Instruments
The education tool used in this study was a newly designed structured performa for the initial evaluation of sick children at the ED. It was developed by subject experts of our department who are qualified pediatric intensivists. It included general information as well as many vital items or indicators after reviewing various modules like pediatric advanced life support (PALS), pediatric fundamental critical care support (PFCCS), basic pediatric intensive care course (BPICC), Indian academy of pediatrics, advanced life support, basic life support (IAP ALS BLS) or essential pediatric intensive care (EPIC) (13-17). structured proforma (Educational tool) was finalized after face validation and content validation by three Pediatric intensivists and one PICU fellow with final reliability was 0.85 according to Cronbach's alpha when 10 records checked as a pilot project. The tool was designed in such a way that postgraduate students must encircle or tick from provided options only. It was less time-consuming than conventional writing and had the least chance of missing any vital information. Apart from general information about a patient, there were main three columns: Evaluate, Identify, and Intervention. In each patient’s structured proforma, the resident is required to address each row, and under Identify column, they need to choose one option (encircle/tick) and based on that, the required intervention is to be mentioned under Intervene column. The pilot screening of 10 records suggested a difference of around a score of 6 among experts and the final version was developed adding 4 items/indicators with a total of 51 with significantly improved content validity.
Figure 1. Flowchart: Development and implementation stage
Data collection methodsAs per the calculated sample size, 36 (pre-intervention) and 36 (post-intervention) computer-generated random selection records were reviewed from a total of serially arranged admissions of pediatric patients (1 month to 18 years of age) at ED whose call was attended by Pediatrics resident doctors. In the current study, the operational definition of accuracy and completeness of documentation was based on the scoring. Score 0 - no mention or an unattended portion, Score 1 - incomplete or inappropriate mention, and Score 2 - proper and complete mention. ACCURACY means each item was written what it should be as per Expert or Rater. COMPLETENESS means the number of items that have got a score of 2 out of the total items or indicators. There was a total of 47 items to score from 0-2 in the pre-intervention phase as per the first version of structured performa whereas the final version had a total of 51 items. So, the maximum score can be 94 in the first version and 102 in the final version. After 2 months of implementation of structured performa we evaluated records again, scored and entered them in Microsoft Excel files item-wise and patient-wise separately by all Raters which was finally compiled in a single Excel file and submitted to a statistician for analysis with blinding of names of Raters.
Data analysis
For the calculation of sample size WINPEPI software was used. The baseline proportion of completeness was 35% and the end-point proportion of completeness we wanted to achieve was 75% with 5% level of significance and 80% power. The calculated sample size was 72 (36 in the pre-intervention phase and 36 in the post-intervention phase) and records were selected by computer-generated random number. In both pre- and post-intervention phases, item-wise frequency, and percentages of score 0, score 1, and score 2 were derived. Also, Item-wise mean (SD) and overall mean (SD) were derived for both phases of the study. Rater-wise mean was calculated in both pre-and post-intervention phases and compared with each other.
We considered the difference in mean inter-rater scoring up to 2% as an acceptable value with acceptable confidence limits of 10%. So, in pre-intervention, the highest total score of 47 items was 94 hence upto 1.88 units (2% of 94) taken as an acceptable difference in mean scoring between the raters and upto 9.4 units (10% of 94) was considered as acceptable confidence limits of difference of scoring between the raters. Similarly in post-intervention, 51 items were included hence the highest total score was 102. So, the acceptable difference in mean scoring among the raters was upto 2 units (2% of 102), and upto 10.2 units (10% of 102) was considered as acceptable confidence limits of difference of scoring between the raters. The agreement between raters was good when the difference in mean scoring was less than 2% of the total score with confidence limits of ≤ 10% of the total score. Bland Altman’s analysis was done to find out the degree of agreement between the raters both in the pre-and post-intervention phases.
Results
Item-wise and Rater–wise comparisons of scoring of both pre- to post-intervention phases are shown in Table 1 and Table 2 respectively.
Table 1. Item-wise comparison of the percentage of scoring (pre- to post-intervention)
| Item description | Preintervention (%) | Mean (SD) |
Postintervention (%) | Mean (SD) | ||||
| Score 0 | Score 1 | Score 2 | Score 0 | Score 1 | Score 2 | |||
| Name | 2.4 | 40.5 | 57.1 | 1.55(0.550) | 0 | 0 | 100 | 2.00 (0.000) |
| Father’s name | - | - | - | - | 11.1 | 16.7 | 72.2 | 1.61 (0.688) |
| Surname | - | - | - | - | 0 | 0 | 100 | 2.00 (0.000) |
| Age | 9.5 | 0 | 90.5 | 1.81(0.594) | 0 | 0 | 100 | 2.00 (0.000) |
| Sex | 9.5 | 0 | 90.5 | 1.81 (0.594) | 8.3 | 0 | 91.7 | 1.83 (0 .561) |
| Weight | 76.2 | 0 | 23.8 | 0.48 (0.862) | 5.6 | 0 | 94.4 | 1.89 (0 .465) |
| Height | - | - | - | - | 75 | 0 | 25 | 0 .50 (0.878) |
| Hospital number | 9.5 | 0 | 90.5 | 1.81(0.594) | 0 | 0 | 100 | 2.00 (0.000) |
| Appearance | 97.6 | 2.4 | 0 | 0 .02(0.154) | 2.8 | 0 | 97.2 | 1.94 (0.333) |
| Breathing | 95.2 | 4.8 | 0 | 0 .05(0.216) | 2.8 | 0 | 97.2 | 1.94 (0.333) |
| Color | 100.0 | 0 | 0 | 0.00(0.000) | 0 | 0 | 100 | 2.00 (0.000) |
| Airways | 100.0 | 0 | 0 | 0.00(0.000) | 2.8 | 0 | 97.2 | 1.94 (0.333) |
| Resp. rate | 2.4 | 0 | 97.6 | 1.95(0.309) | 16.7 | 0 | 83.3 | 1.67 (0.756) |
| Interpretation of RR | - | - | - | - | 2.8 | 0 | 97.2 | 1.94 (0.333) |
| Work of breathing | 92.9 | 0 | 7.1 | 0.14(0.521) | 8.3 | 0 | 91.7 | 1.83 (0 .561) |
| Chest Rise | 100.0 | 0 | 0 | 0.00(0.000) | 2.8 | 0 | 97.2 | 1.94 (0.333) |
| Air entry | 2.4 | 0 | 97.6 | 1.95(0.309) | 2.8 | 0 | 97.2 | 1.94 (0.333) |
| Adventitious sounds | 2.4 | 59.5 | 38.1 | 1.36(0.533) | 2.8 | 5.6 | 91.7 | 1.89(0.398) |
| SPO2 without oxygen | 2.4 | 0 | 97.6 | 1.95(0.309) | 19.4 | 0 | 80.6 | 1.61 (0.803) |
| SPO2 with O2 8L with NRM | 92.9 | 0 | 7.1 | 0.14(0.521) | 61.1 | 2.8 | 36.1 | 0.75 (0.967) |
| Heart Rate | 2.4 | 0 | 97.6 | 1.95(0.309) | 0 | 41.7 | 58.3 | 1.58 (0.500) |
| Rhythm | 97.6 | 2.4 | 0 | 0.02(0.154) | 0 | 0 | 100 | 2.00 (0.000) |
| Pulse: Central vs Peripheral | 85.7 | 2.4 | 11.9 | 0.26(0.665) | 0 | 0 | 100 | 2.00 (0.000) |
| Extremities | 92.9 | 0 | 7.1 | 0.14(0.521) | 0 | 0 | 100 | 2.00 (0.000) |
| CRT | 78.6 | 0 | 21.4 | 0 .43(0.831) | 0 | 2.8 | 97.2 | 1.97(0.167) |
| BP (mmHg) | 35.7 | 0 | 64.3 | 1.29(0.970) | 11.1 | 36.1 | 52.8 | 1.42 (0.692) |
| GCS/AVPU (Cortical dysfunction) | 19.0 | 78.6 | 2.4 | 0.83(0.437) | 2.8 | 0 | 97.2 | 1.94 (0.333) |
| Pupils | 92.9 | 0 | 7.1 | 0.14(0.521) | 0 | 0 | 100 | 2.00 (0.000) |
| Oculocephalic movements | 100 | 0 | 0 | 0 .00(0.000) | 55.6 | 0 | 44.4 | 0.89 (1.008) |
| RBS (mg/dl) | 83.3 | 0 | 16.7 | 0.33(0.754) | 13.9 | 44.4 | 41.7 | 1.28 (0.701) |
| Temperature | 0 | 38.1 | 61.9 | 1.62(0.492) | 0 | 41.7 | 58.3 | 1.58 (0.500) |
| Skin bruise/bleeds | 100 | 0 | 0 | 0 .00(0 .000) | 16.7 | 0 | 83.3 | 1.67 (0.756) |
| Symptoms | 0 | 0 | 100 | 2.00 (0.000) | 2.8 | 2.8 | 94.4 | 1.92 (0.368) |
| Allergies | 100 | 0 | 0 | 0.00 (0.000) | 16.7 | 0 | 83.3 | 1.67 (0.756) |
| Medications- if any ongoing | 92.9 | 2.4 | 4.8 | 0.12 (0.453) | 5.6 | 0 | 94.4 | 1.89 (0.465) |
| Past Medical History | 90.5 | 0 | 9.5 | 0.19 (0 .594) | 5.6 | 5.6 | 88.9 | 1.83 (0.507) |
| Last meal taken | 100 | 0 | 0 | 0.00 (0.000) | 25.0 | 8.3 | 66.7 | 1.42 (0.874) |
| Event preceding to present status | 64.3 | 4.8 | 31.0 | 0.67 (0.928) | 19.4 | 8.3 | 72.2 | 1.53 (0.810) |
| Hematological | 66.7 | 26.2 | 7.1 | 0.40 (0.627) | 2.8 | 2.8 | 94.4 | 1.92 (0.368) |
| Microbiological | 85.7 | 14.3 | 0 | 0.14 (0.354) | 11.1 | 0 | 88.9 | 1.78 (0.637) |
| Biochemical | 81.0 | 19.0 | 0 | 0.19(0.397) | 8.3 | 0 | 91.7 | 1.83 (0.561) |
| Imaging studies | 97.6 | 2.4 | 0 | 0.02 (0.154) | 19.4 | 2.8 | 77.8 | 1.58 (0.806) |
| Severity | 95.2 | 2.4 | 2.4 | 0.07 (0.342) | 2.8 | 0 | 97.2 | 1.94 (0.333) |
| Severity of respiratory problem | 100 | 0 | 0 | 0.00 (0.000) | 0 | 0 | 100 | 2.00 (0.000) |
| Type of respiratory problem | 100 | 0 | 0 | 0.00 (0.000) | 13.9 | 0 | 86.1 | 1.72 (0.701) |
| Type of circulatory problem | 100 | 0 | 0 | 0.00 (0.000) | 2.8 | 0 | 97.2 | 1.94 (0.333) |
| Others | 100 | 0 | 0 | 0.00 (0.000) | 5.6 | 2.8 | 91.7 | 1.86 (0.487) |
| Name of resident | 0 | 0 | 100 | 2.00 (0.000) | 11.1 | 5.6 | 83.3 | 1.72 (0.659) |
| Sign of resident | 16.7 | 9.5 | 73.8 | 1.57 (0 .770) | 5.6 | 0 | 94.4 | 1.89 (0.465) |
| Date of sign | 0 | 21.4 | 78.6 | 1.79 (0.415) | 0 | 0 | 100 | 2.00 (0.000) |
| Time of sign | 7.1 | 14.3 | 78.6 | 1.71 (0.596) | 19.4 | 0 | 80.6 | 1.61 (0.803) |
| Total | 32.93 (4.507) | 89.64 (4.350) | ||||||
Table 2. Rater-wise comparison of mean scoring and percentage of mean (pre- to post-intervention)
| Pre-intervention mean (SD) | Percentage of pre-intervention mean (SD) | Post-intervention means (SD) | Percentage of post- intervention mean (SD) | Percentage of Improvement (pre- to post-intervention) | p-value* | |
| Rater 1 | 30.30(4.181) | 32.23% | 88.90(5.004) | 87.16% | 54.93% | <0.0001 |
| Rater 2 | 29.31(4.409) | 31.18% | 90.66(4.798) | 88.88% | 57.70% | <0.0001 |
| Rater 3 | 31.05(3.300) | 33.03% | 89.50(5.176) | 87.74% | 54.71% | <0.0001 |
| Rater 4 | 32.26(6.061) | 34.32% | 90.28(4.676) | 88.50% | 54.18% | <0.0001 |
| Rater 5 | 36.05(4.310) | 38.35% | -- | -- | -- | - |
| Average | 32.93(4.507) | 33.82% | 89.64(4.35) | 87.07% | 53.25% | <0.0001 |
| *Paired t-test | ||||||
In pre-intervention phase, the items which were not mentioned in the majority of files were Weight in General information; Appearance, breathing, color, airways, work of breathing, chest rise, a saturation of oxygen(SPO2) with oxygen, rhythm, central vs peripheral pulses, extremities, capillary refill time(CRT), pupils (Brainstem function), oculocephalic movements, skin bruise/bleeds in initial and primary assessment; Allergies, medications- if any ongoing, past medical history, last meal taken in Secondary assessment; imaging studies in Diagnostic tests and Severity/life-threatening problem or severity of the respiratory problem or type of respiratory problem or type of circulatory problem in Overall assessment. In the post-intervention phase majority of the items improved after the implementation of structured proforma except Height, SPO2 with 8L of oxygen with a non-rebreathing mask, oculocephalic movements and RBS. Figure 2 shows the degree of agreement by Bland-Altman plots between the Raters in the pre-intervention phase whereas Figure 3 depicted the same for the post-intervention phase.
Figure 2. Bland altman pre-intervention inter-rater comparison
Figure 3. Bland altman plots - post-intervention inter-rater comparison
Figure 3. Bland altman plots - post-intervention inter-rater comparison
In our study except for Rater 5 in the preintervention group, all other raters had a good inter-rater agreement in both the phase. The inference of inter-rater agreement in the phase is shown in Table 3.
Table 3. Interpretation of inter-rater agreement pre-and post-intervention
| Raters | Pre -Intervention | Post- intervention | ||||||
| Mean Difference | Any trend? Bias in rating? |
No. of values beyond CL* | Agreement | Mean Difference | Any trend? Bias in rating? |
No. of values beyond CL* | Agreement | |
| 1 & 2 | 1.0 | No. No | 1 | Fair | 0.8 | No, no | 2 | Fair |
| 1 & 3 | 0.8 | No, no | 2 | Fair | 0.6 | No, no | 2 | Fair |
| 1 & 4 | 0.3 | No, no | 1 | Fair | 0.6 | No, no | 0 | Good |
| 1 & 5 | 5.7 | No, no | --- | Poor | --- | --- | --- | --- |
| 2 & 3 | 1.7 | No, no | 1 | Fair | 0.2 | No, no | 2 | Fair |
| 2 & 4 | 1.2 | No, no | 2 | Fair | 0.2 | No, no | 1 | Fair |
| 2 & 5 | 6.7 | No, no | --- | Poor | --- | --- | --- | --- |
| 3 to 4 | 0.5 | No, no | 1 | Fair | 0.1 | No, no | 0 | Good |
| 3 & 5 | 5.0 | No, no | --- | Poor | --- | --- | --- | --- |
| 4 & 5 | 5.5 | No, no | --- | Poor | --- | --- | --- | --- |
Discussion
The most striking outcome of the current study was a tremendous improvement in the overall mean (SD) score to 89.64 (4.35) from the baseline of 32.93 (4.51) of an initial assessment of sick children evaluation notes at ED. The use of the third level of the Kirkpatrick evaluation model changed the practices of documentation by introducing education tools and by training pediatric residents. The Kirkpatrick model assesses both formal and informal training methods and rates them against four levels of criteria: reaction, learning, behavior, and results (5) It is a globally recognized method of evaluating the results of training and learning programs.
Earlier in 1999, for triage of patients in the Emergency Department, Emergency Severity Index (ESI) was introduced but pediatric criteria were added to it in 2000 (18,19). As residents play a major role in ensuring quality care for patients, introducing quality measures suitable for them in practice is desirable and in 2016 Schumacher et al (20) identified and developed Resident-sensitive quality measures (RSQMs) for the use in the Pediatric Emergency Department setting. However, we used parameters that were important documentation point of view. In the current study, in the pre-intervention phase, the items which were not mentioned the in majority of files were weight, appearance of child, breathing, color, airways, work of breathing, chest rise, saturation of oxygen(SPO2), cardiac rhythm, central vs peripheral pulses, extremities, capillary refill time(CRT), pupils (Brainstem function), oculocephalic movements, skin bruise/bleeds, allergies, medications- if any ongoing, past medical history, last meal taken, imaging studies in diagnostic tests and severity/life-threatening problem or severity of respiratory problem or type of respiratory problem or type of circulatory problem. But in the post-intervention phase the majority of the items improved significantly after training and implementation of structured proforma except height, SPO2 with 8L of oxygen with a non-rebreathing mask, oculocephalic movements, and blood sugar. The probable reason may be that the patient may not need oxygen and residents might have confusion that what to write and also in some situations they felt that blood sugar is not required. So, such items have scope for modification in future versions of structured proforma.
Kalet et al proposed the adoption of educationally sensitive patient outcomes (ESPOs) for training resident doctors. They described this as “patient outcomes that are sensitive to provider education, can be measured, and are in the pathway linking medical education interventions to patient outcomes” (21). They suggested two measures: patient activation (PA), a strong component of healthful behaviour change and clinical microsystem activation (CMSA), a major influence on patient safety and healthcare quality. However, in the present study this may be the limitation as patients ‘outcome of patients was not assessed along with the documentation improvement process but in a future study, it can be planned and done.
Adler et al did their work using Simulation techniques where they conducted a two-phase randomized control trial to develop and evaluate a simulation-based Pediatric Emergency Medicine Curriculum (22). In the validation phase (2006–2007), he randomized 69 residents from two EM residencies to either an intervention group that received the curriculum one month before the first assessment of all participants or a waitlist control group that received the identical curriculum three months later. A final assessment of all residents followed one month after that. Two raters evaluated all residents. Primary outcome measures were percentages of items completed correctly. They assessed rater agreement using intraclass correlation (ICC) and compared group performance using mixed-model analysis of variance, but they observed that a one-day, simulation-based pediatric EM curriculum produced limited results. In the current study, inter-rater agreement was used, and it was found that the introduction of structured proforma after training residents improved the desired result, and both in the pre-and post-intervention phases were fair to a good agreement among most raters. The main limitation of our study was that it evaluated only the process of documentation among the pediatric residents after training and not the outcome of patients based on it. Also, those few items need to be modified or removed which were not addressed even in the post-intervention phase during documentation.
Conclusion
Teaching and training Pediatrics residents while implementing the newly developed educational tool in the Emergency department has improved the documentation process significantly and there was fair to good agreement between the raters it is suggested to use at other places also and can be incorporated in digital format also in future.
Ethical considerations
The Institutional Ethics Committee of H M Patel Center of Medical Care and education approved this project. (IEC/ HMPCMCE/ 114/faculty 13, 7/08/2020) All students were adequately informed about the aims and methodology of the study before their participation. Written informed consent was obtained from participants. They were assured that no academic harm to them, their data would be kept confidential, and only general information and statistics would be released.
Acknowledgments
We thank all Resident doctors of Pediatrics who contributed to this study.
Disclosure
None to declare
Author contribution
T.N.M, B.T, A.K, S.P, M.K.C, K.R.T: Performed the study intervention and collected data. K.R.T: Designed and supervised the procedure. The data were analyzed by J.G who participated in planning the study and revising the manuscript. The manuscript was read and approved by all authors.
Data availability statement
Upon a reasonable request, the corresponding author can provide the data set that was analyzed during this study.
Article Type : Orginal Research |
Subject:
Medical Education
Received: 2023/03/31 | Accepted: 2023/08/8 | Published: 2024/03/16
Received: 2023/03/31 | Accepted: 2023/08/8 | Published: 2024/03/16
References
1. Macias CG. Quality improvement in pediatric emergency medicine. Academic Pediatrics. 2013 Nov 1; 13(6): S61-8. [DOI]
2. Borland ML, Shepherd M. Quality in paediatric emergency medicine: Measurement and reporting. Journal of Pediatrics and Child health. 2016 Feb; 52(2): 131-6. [DOI]
3. Moy AJ, Schwartz JM, Chen R et al. Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review. Journal of the American Medical Informatics Association. 2021 May; 28(5): 998-1008. [DOI]
4. Fouquet SD, Fitzmaurice L, Chan YR et al. Doctors documenting: an ethnographic and informatics approach to understanding attending physician documentation in the pediatric emergency department. Journal of the American Medical Informatics Association. 2021 Feb 15; 28(2): 239-48. [DOI]
5. Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick's four levels of training evaluation. Association for Talent Development; 2016 Oct 1. [ISBN: 9781607280088] [Article]
6. Alhassan AI. Implementing faculty development programs in medical education utilizing kirkpatrick’s model. Advances in Medical Education and Practice. 2022 Jan 1: 945-54. [DOI] [PubMed]
7. Holt N, Crowe K, Lynagh D et al. Is there a need for formal undergraduate patient handover training and could an educational workshop effectively provide this? A proof-of-concept study in a Scottish Medical School. BMJ Open. 2020 Feb 1; 10(2): e034468. [Article] [DOI]
8. Rouse M, Jones M, Zogleman B et al. Resident integration with inpatient clinical documentation improvement: a quality improvement project. BMJ Open Quality. 2022 Jun 1; 11(2): e001300. [DOI]
9. Nyman SJ, Vogel ME, Heller GM et al. Development and evaluation of a health behavior change clinic in primary care: an interdisciplinary partnership. Journal of Clinical Psychology in Medical Settings. 2023 Mar 4: 1-5. [DOI]
10. Mizumoto J, Son D, Izumiya M et al. Experience of residents learning about social determinants of health and an assessment tool: Mixed‐methods research. Journal of General and Family Medicine. 2022 Sep; 23(5): 319-26. [DOI]
11. Tan XH, Foo MA, Lim SL et al. Teaching and assessing communication skills in the postgraduate medical setting: a systematic scoping review. BMC Medical Education. 2021 Dec; 21: 1-9. [DOI]
12. Gordon M, Hill E, Stojan JN et al. Educational interventions to improve handover in health care: an updated systematic review. Academic Medicine. 2018 Aug; 93(8): 1234. [DOI] [PubMed]
13. Rodenbarger AL, Maa T, Chase M et al. Effectiveness of pediatric fundamentals of critical care support (PFCCS) in enhancing resident self-efficacy and skills. Academic Pediatrics. 2016 Aug 1;16(6): e54. [Article] [DOI]
14. Mejίa R, Fields A, Greenwald BM et al. Pediatric fundamental critical care support (PFCCS). 2nd ed. Mount Prospect, IL (USA): Society of Critical Care Medicine, 2011: 1-17 [ISBN: 978-1-620750-09-4] [Google Scholar]
15. Khilnani P, Uttam R, Chugh K. Manual of basic pediatric intensive care course.3rd ed. New Delhi: IAP Intensive Care Chapter, 2015: 1-8. [Article]
16. Jayashree M, Kulgod V, Sharma AK. IAP ALS handbook. New Delhi: Iap, national publication house, 2018: 10-27.
17. Karunakara BP, Shrivastav K, Buche V et al. Manual of essential pediatric intensive care course. New Delhi: IAP Intensive Care Chapter, 2017: 1-13.
18. Travers DA, Waller AE, Katznelson J et al. Reliability, and validity of the emergency severity index for pediatric triage. Academic Emergency Medicine. 2009 Sep; 16(9): 843-9. [Article] [DOI]
19. Mistry B, De Ramirez SS, Kelen G et al. Accuracy and reliability of emergency department triage using the emergency severity index: an international multicenter assessment. Annals of Emergency Medicine. 2018 May 1; 71(5): 581-7. [DOI]
20. Schumacher DJ, Holmboe ES,Van Der Vleuten C et al. Developing resident-sensitive quality measures: A model from pediatric emergency medicine. Academic Medicine. 2018 Jul 1; 93(7): 1071-8. [DOI]
21. Kalet AL, Gillespie CC, Schwartz MD Bruno JH et al. New measures to establish the evidence base for medical education: identifying educationally sensitive patient outcomes. Academic Medicine. 2010 May 1; 85(5): 844-51. [DOI]
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
