Evaluating the effectiveness of ‘MCQ development workshop using 3 cognitive model framework —a pre-post study

11 A workshop on MCQ development using cognitive model framework was 12 conducted for health educators from Aga Khan University (AKU) and other 13 academic institutions. The aim was to develop the skill of preparing MCQs for 14 assessing higher cognitive levels. A pre-post study was conducted, participant 15 satisfaction was evaluated and pre–post test scores were used to assess learning 16 capability of the workshop participants. Out of the 19 who attended the workshop, 17 16 participated in the pre-and post-tests and were included in the study through 18 convenience sampling. The total duration of the study was six months. There was a 19 significant difference in the overall pre-post test scores of the participants with a 20 mean difference of -4.176 ± 4.83 (p value < 0.05). A significant difference was 21 observed in the mean pre-post test scores of junior faculty (-6.350 ± 4.5829; P 22 value = 0.02). The mean pre-test scores of junior faculty were significantly lower 23 4.950 ± 2.83 as compared to the senior faculty 10.417 ± 1.56 (P value= 0.001). 24 Active participation in faculty development workshops may lead to enhancing 25 skills for preparing one-best MCQs based on international guidelines.


Introduction
Assessment of health professionals' education heavily relies on multiple choice questions (MCQ).Peer review and review by educational experts ensures quality of one best MCQs.However, during the MCQ review it was observed that most of the questions developed by the faculty assess the knowledge at recall level and fail to follow the guidelines laid by the National Board of Medical Examiners (NBME). 1 MCQs are considered an appropriate tool to not only measure a candidate's ability to recall and comprehend but can also be designed to measure higher order thinking skills such as application and analysis. 2Well-constructed MCQs test reasoning at higher levels and, therefore, can discriminate between high and low-achievers. 3,4 he construction of MCQs to assess higher levels of comprehension and analysis is a challenging task which requires both sufficient knowledge and training.
A multiple choice question comprises a stem which may perhaps be a clinical scenario.It is followed by a number of options which are related to the stem, and only one option may be correct (one correct type MCQ) or the best (one-best type of MCQ).To be awarded marks, the students 'task is to identify the most appropriate option. 5 with flaws lead to errors in interpretation of test scores and may undesirably affect the pass rate of the students.To ensure reliability and validity of tests, items constructed must be free of errors. 6Flawed items have an overall impact on the performance of students.The structural flawlessness of MCQs can be improved by the level of experience. 6

P r o v i s i o n a l l y A c c e p t e d f o r P u b l i c a t i o n
Maximum errors are attributed to the MCQs' structure, followed by inaccuracies in the stem and vignette of the MCQs. 7It is reported that items developed by those faculty members who have not undergone faculty development training lack quality.This is an evidence to the value of faculty development in generating quality items. 8as been observed that the questions being used in examinations held at institutional levels are not following international guidelines for developing one best MCQs, which, however, may be significantly improved by conducting formal workshop on MCQ development.Internationally accepted guidelines and a multidisciplinary review lead to improved quality of items. 9objectives of this study were to explore the perceptions of workshop participants regarding achievement of workshop objectives, content covered, and facilitation skills, and to determine if participation in the workshop helped them gain new knowledge and skills.The instructional approach of the workshop was based on flip class methodology.

It
Prior to the face-to-face session of the workshop, the chapter on One Best type MCQ construction from the NBME guidelines was shared with all the registered participants.Questions to stimulate prior knowledge and check basic understanding of the concept was assessed through the pre-workshop readings which were posted on an online discussion forum created on Padlet.The purpose of this discussion forum was to assess the participants' knowledge regarding basic rules on constructing One-Best MCQs and to engage students outside the workshop.
To determine if the participants acquired new knowledge, a pre-test was held just was a pre-post study conducted at the Aga Khan University (AKU) in which faculty from AKU (Medical College, School of Nursing and Allied Health), as well as other academic institutions offering undergraduate and postgraduate health science education, participated in a workshop on how to construct effective onebest MCQs.Both senior (assistant professor and above)and junior faculty (below assistant professor) attended the workshop.This evaluation is based on Kirkpatrick's model to analyse and evaluate the impact of educational training at level 1and 2. 10 Gagnes nine events of instruction were used to design the workshop: seek attention, inform learners of objectives, recall prior learning, present learning P r o v i s i o n a l l y A c c e p t e d f o r P u b l i c a t i o n material, provide support to learner, perform individually or in a group and provide feedback, measure performance and promote retention and transfer. 11 before starting the three-hour workshop and a post-test was held on completion of the workshop.In the pre/post-test the participants were asked to categorise MCQs according to the Blooms level of learning being assessed by the MCQ and identify flaws based on general rules for one best answer items as given in the NBME guidelines.Total number of participants in the workshop were 19; convenience sampling was done and data of only 15 participants was included in the quantitative analysis.All candidates who sat the pre-and post-test and participated in the entire workshop were included in the study.Participants who did not attend the entire workshop and did not complete both the pre-and post-test were excluded from the study.Data to measure acquisition of new knowledge was collected by using pre/post-test questionnaire and perceptions of the workshop participants regarding achievement of workshop objectives, content covered and facilitation was acquired by the workshop evaluation form.P r o v i s i o n a l l yA c c e p t e d f o r P u b l i c a t i o n no significant difference in post-test scores of junior (11.3 ± 5.3) vs senior faculty (11.08± 3.78) (p value > 0.05).No significant difference was noted in the mean post-test scores of junior faculty and senior faculty.Conclusion Faculty development workshop improved one best MCQs writing skills based on NBME guidelines.A significant difference was observed in the mean pre-post-test scores of junior faculty.Such workshops are conducted to improve the MCQ writing skills of faculty members.Disclaimer: None to declare