Challenges In Implementing Direct Observation of Procedural Skills as Assessment Tool in Postgraduate Training of Neurosurgery

Background: DOPS is a brief, verified technique of assessment that involves a supervisor observing a trainee do a procedural skill and providing feedback to the trainee.

Objective: To explore challenges in implementing Direct Observation of Procedural Skills as a workplace-based assessment tool in postgraduate training of Neurosurgery.

Subject and Methods: To investigate challenges in the implementation of DOPS, a qualitative exploratory approach was used. via purposive sampling technique 11 interviews were carried out, 5 from Postgraduate Neurosurgery resident, 3 faculty members and 3 people from administration including, Medical Director, Hospital Director and Dean of the college. All of the interviews were taped and afterwards transcribed using the Inqscribe transcription programme.

Results: Thematic analysis of the interview was done. common words and phrases were identified and grouped. fifty different codes were identified and labelled. two themes were identified including challenges in successful implementation of DOPS (Table 1) and suggestions for achieving this purpose. (Table II)

Conclusion: Lack of awareness, training, and time constraints are major obstacles to implementing the DOPS programme, although faculty development through training workshops and making of DOPS protocols are ideas for a successful implementation.

DOPS is a brief, verified technique of assessment that involves a supervisor observing a trainee do a procedural skill and providing feedback to the trainee. Providing feedback to the trainee is the most significant aspect of the DOPS [1]. The lack of desired awareness regarding DOPS among the resident of Neurosurgery in Pakistan made us to conduct this study. 

Until recently, the conventional apprenticeship model with the "see one, do one, teach one" teaching technique was widely used in the majority of postgraduate medical institutes [2]. However, DOPS is widely used in undergraduate and postgraduate student as a formative assessment [3]. The faculty members thought DOPS was a valid kind of assessment, although they had worries regarding its viability due to time restrictions. DOPS was a little more stressful for the students, but they loved the idea of getting feedback(4). Some students praised DOPS for improving their attitude toward studying, despite the fact that some of them did not fully comprehend the system's complexities [5]. In Neurosurgery, Waqar SH investigated students' perceptions of Direct Observation of Procedural Skills as a Workplace Based Assessment technique. However, it was necessary to learn about assessor attitudes and administrative problems in implementing DOPS as tan assessment instrument [6, 7].

To keep pace with the advancements in assessment techniques across the globe, we want to introduce DOPS as a WBA technique to gain understanding into the difficulties encountered while using it as an assessment instrument [8]. This will likely aid us in determining the best methods for implementing this valuable assessment procedure [9, 10].  At Hayatabad Medical Complex, a qualitative study was designed to investigate the problems of integrating Direct Observation of Procedural Skills in postgraduate Neurosurgery training. The findings were expected to help in identifying the challenges faced in implementation of DOPS along with their possible solutions.

To investigate challenges in the implementation of DOPS, a qualitative exploratory approach was used. via purposive sampling technique 11 interviews were carried out, 5 from Postgraduate Neurosurgery resident, 3 faculty members and 3 people from administration including, Medical Director, Hospital Director and Dean of the college. Neurosurgery residency is a five-year program initial two year spent in General surgery, and the next three year in Neurosurgery. so, we included 3rd, 4th and final year resident. 

Residents and faculty members who have not participated in the DOPS programme were excluded. The study was approved by Ethical Review Board. Every participant gave their informed consent, ensuring confidentiality and anonymity. All of the interviews were taped and afterwards transcribed using the Inqscribe transcription programme. Thematic analysis was used to analyze the data, which involved open and axial coding to separate categories and concepts that led to themes. The data was associated and thematically analyzed using NVivo 10, a computer-assisted qualitative data analysis software. Findings were presented in tabulated form.

A total of eleven interviews were carried out. Three faculty members, five postgraduate trainees, and three people from administration department including Dean of Medical College, Medical Director, and Hospital Director at Hayatabad Medical Complex, Peshawar. Thematic analysis of the interview was done. common words and phrases were identified and grouped. fifty different codes were identified and labelled. two s themes were identified including challenges in successful implementation of DOPS (Table 1) and suggestions for achieving this purpose (Table 2).

Table 1: Theme: Challenges 

Table 2:Theme: Suggestions

Direct observation of practical skills (DOPS) is a method of evaluating trainee proficiency in practical processes. Different assessors should carry out the evaluations, which should include a wide range of operations. While the trainee is engaged in the activity, the observer is present [2]. Before the assessment, the procedure being observed should last no more than 10–15 minutes. With the trainee present, the assessor will spend 5-10 minutes offering rapid feedback and completing the evaluation form [11].

Surgical education has evolved dramatically during the last few decades. Many modern strategies for accurately assessing the surgical abilities of neurosurgery residents have been developed. For its successful deployment, it is critical to be aware of its full potential [12, 13]. The majority of the participants in this study believed that a suitable type of postgraduate training monitoring is critical. However, many of them were unsure whether DOPS was a good instrument for workplace assessments. The validity and reliability of assessment procedures for evaluating procedural abilities is constantly a source of debate [14, 15].

Abolghasem Amini et al. showed that residents’ performances were almost good DOPS examinations, according to the majority of instructors and residents, can be useful in facilitating students' learning and assisting them in passing their final exam [11, 14]. DOPS, according to some residents, helped them relax throughout the final exam. David Valentine et al, recently found evidence of validity and reliability in favour of DOPS in assessing spinal endoscopic procedure and indicated that it may be used to track competency growth [15]. Our study's 05 participants all agreed that DOPS performance is more indicative of procedural skill competency than competence in another procedural skill [16, 17].

Lack of awareness about the importance of DOPS as a newer assessment technique with feedback option and educational impact, lack of prior training of assessors and residents to make them familiar with the procedure, difficulty in finding time for DOPS due to a heavy workload faced by all stakeholders, lack of realistic DOPS protocols, and some administrative issues were among the major challenges faced by participants [18, 19]. Suggestions for efficient DOPS adoption included raising awareness among teachers and residents and training them on how to use the tool more effectively.

The study's scope was limited by the fact that it only looked at the difficulties encountered in implementing DOPS as an assessment tool in postgraduate training at one institute. To understand the actual situation on the ground, it is necessary to delve into these difficulties on a much larger scale.

Direct Observation of Procedural Skills was a novel technique of assessing Neurosurgery residents, according to the participants in this study. The main problems in utilizing DOPS as an evaluation tool were a lack of understanding about its utility among professors and residents, a lack of training for assessors and residents, and time limits due to intense workload in hospitals. The organization of awareness / training workshops, as well as the development of actual DOPS procedures by the monitoring body, were deemed key milestones in the effective application of DOPS. It was also suggested that training institutes offer faculty development programs.