Validation of an assessment tool for professionalism in medical students: A mixed method study

Objective: To develop and validate a tool for the assessment of professionalism in medical students during clerkship. Method: The mixed-method study was conducted from February to September 2018 at the Islamic International Medical College, Riphah University, Islamabad, Pakistan. A preliminary focus group discussion was followed by two rounds of Delphi technique to validate micro-scenarios for the relevance of each item of the instrument alongside its content validity. For content and construct validation of the tool, content validity, cognitive validity, cognitive pretesting, confirmatory factor analysis and reliability assessment was done. Piloting was done on medical students doing clerkship. Data was analysed using SPSS 21 and AMOS 21. Results: A 35-item preliminary tool from micro-scenarios generated after focus group discussion was developed for the first Delphi round. For the second round, a 29-item tool was resent to the experts for validation. After the second round, the content validity scale was 0.94. Removal of 1 item produced a 28-item tool. Pre-cognitive testing of the 28 items resulted in rephrasing of 4, modification of 2, and deletion of 1 and a tool with 27 items. Piloting was carried out and after CFA resulted in a reduced 21 item tool. Cronbach’s alpha value was 0.96. A good to excellent model fit was produced after performing confirmatory factor analysis. Conclusion: A tool comprising of 21 items scored on a 5-point Likert scale that had acceptable construct validity and good content validity was developed. It can be used to evaluate the level of professionalism of local medical students doing clerkship.


Introduction
Professionalism in medical is a "commitment to carrying out professional responsibilities, adherence to ethical principles and sensitivity to diverse patient populations". 1he American Board of Internal Medicine (ABIM) framework has identified six areas of professionalism --respect, accountability, excellence, altruism, integrity and duty -that a graduating medical doctor must display. 2 The cognitive and non-cognitive base of professionalism is provided during preclinical and clinical years through teaching and role modelling, respectively.Compared to the cognitive base of professionalism, assessment of noncognitive base during clerkship is more challenging due to a complex blend of ethical commitment and essential attributes.3 Periodic assessment of professionalism through a valid and reliable instrument may help implement effective teaching methods that aids the growth of professional behaviour in medical students.4 According to the theory of planned behaviour, development of professionalism in undergraduate students can be predicted.5 Therefore, a feedback tool based on the behavioural elements, such as attitude, subjective norm and perceived behavioural control, may be used by the faculty to assess, specific professional behaviours in different workplace situations, and guide students in improving professionalism.6 Assessment lies at the core of lasting knowledge.To ensure the development of the desirable internalised professionalism attributes and behaviours, its formative and summative assessment must be incorporated in the curriculum, especially during clerkship.4 The faculty can be helped through the introduction of an assessment tool developed in the local context as part of a 360 degree feedback loop during their clinical rotation. 7dical professionalism is a culture sensitive construct as a social contract is formed between medical doctors and the public demonstrated in explicit behaviours by the evolving future doctors.7 The ABIM framework is most commonly used in the Western world for assessing the professional behaviour.Learner's Attitude on Medical Professionalism Scale (LAMPS) has been developed in the Arabian context using ABIM framework.8 Similarly, in Pakistan, which is a Muslim country, the provision of a feedback tool based on local norms may be helpful to assess a particular professional behaviour of the students in the local context reflecting its culture, beliefs and religion.9 At present, various tools, such as Conscientiousness Index (CI), LAMPS, mini-clinical evaluation exercise (Mini-CEX) and professionalism minievaluation exercise (P-MEX) are being used to assess professionalism without reporting the psychometrics properties.10 Societies vary in their expectations of doctors, and, hence, a tool needs to be developed on the basis of cultural values of the region reflecting regional diversity, such as displaying respect, completion of task and dressing up properly.The current study was planned to develop and validate a new tool to measure the professionalism of medical students during the clerkship module.

Materials and Methods
The mixed-method study was conducted from February to September 2018 at the Islamic International Medical College, Riphah University, Islamabad, Pakistan.
The Professionalism in Medical Students (PIMS) tool was micro-scenarios designed in 3 phases after approval from the institutional ethics review board.
Phase-1: Comprised content, domain specification and item generation.Guided by the recommendations of the Association for Medical Education in Europe (AMEE) Guide No 87, a literature review was carried out, and 6 questions were finalised guided by the ABIM framework. 2,11Focus group discussion (FGD) was conducted with nine qualified medical educationists having more than 3 years' experience. 12This phase comprised of micro-scenario were developed on the basis of quotes of the participants depicting possible professional situations for a preliminary tool in the local context.

Phase-2:
Comprised of content validation.A 35-item preliminary tool was finalised from the micro-scenarios.This content was validated by a panel of experts through two Delphi rounds.Clinical teachers having more than 5 years' teaching experience were involved.The experts were assistant professors and above, both males and female.They were from hospitals of Rawalpindi, Pakistan.For round 1, 40 experts were sent emails with a covering letter, consent form and the tool in Google Forms.The experts were asked to rate each item on a scale ranging from 5='very important' to 1='not important' .They were also encouraged to suggest any additional component to be included.Data was analysed using SPSS 21.
For round 2, a tool with reduced items and a summary of all the responses were sent back to the same participants.They were asked to categorise each item based on relevance and essentiality.For relevance of items to the construct, a 5-point Likert scale was used, raninging from 5='highly relevant' to 1='not relevant' .For essentiality of the item, a 3-point Likert scale was used, ranging from 3=essential to 1='non-essential' .For Delphi, a period of 3-4 weeks was given to the participants to respond.Upon reaching consensus, the Delphi phase was concluded. 13r computing the content validity index (CVI) of the items (I-CVI), data was transferred to Excel sheet.The collective result for the content validity scale (S-CVI/Ave) was determined.Using Lawshe formula, content validity ratio (CVR) was computed to assess the essentiality of items. 14tems with CVR extending from 0.60 to 0.80 were modified, and those with score <0.6 were deleted.
Pre-cognitive testing was done to determine the response process validity of the tool.It was carried out on a 10member panel before piloting.

Phase-3:
Comprised piloting of the tool, evaluation of the reliability and construct validity.A 12-member faculty team of assistant professors and senior registrars from the Ear-Nose-Throat (ENT), Eye, Medicine, Paediatrics, Surgery and Obstetrics and Gynaecology (OBGYN) departments conducted the piloting of the tool on students doing clerkship from fourth and final year of their studies.A 5 point Likert scale was employed where observed behaviour was marked from 5='excellent; to 1='very poor' .There were 27 items against 06 constructs. 15AMOS 21 was used to conduct confirmatory factor analysis (CFA).

Results
In the first phase, 46 micro-scenario item statements were developed as a result of FGD.This led to a 35-item tool.
In phase 2, of the 40 experts approached, 24(60%) responded for Delphi round 1.There was ≥80% agreement with a median score of ≥4 which resulted in the removal of 6(17%) items.A 29-item tool was finalized for Delphi round 2 for which 18(75%) experts responded.Items having I-CVI ≥0.78 were eliminated, I-CVI 0.78-0.90were modified, and I-CVI 0.90 were incorporated.The thematic analysis of the comments from the experts resulted in the modification of 5(17.24%)items and elimination of 1(3.44%).A 28-item preliminary tool for precognitive testing was finalised.
The pre-cognitive testing in phase 3 resulted in the deletion of 1(3.6%) item, and a 27-item remained for piloting (Table-1).
Cronbach's alpha for internal consistency of the tool was 0.96 despite multifaceted elements.Item for the domains of Respect were 4, Altruism had 3, Honor and Integrity had 4 items, accountability had 4, duty had 3, and excellence had 4 items respectively.The Cronbach's alpha was 0.933 for respect, 0.74 for altruism, 0.80 for honour and integrity, 0.85 for accountability, 0.81 for duty and 0.87 for excellence.The inter-item correlation ranged from 0.741 to 0.933 which ranged from satisfactory to excellent.
CFA measurement model (Figure) showed that all the item loadings exhibited acceptable factor loadings (>50).CFA parameters ranged from acceptable to excellent (Table -2).

Discussion
The current study outlined the method of construction and validation of a tool for measuring the development of professionalism in a local setting, focussing on the areas of professionalism specified in the ABIM framework. 2ccording to the key findings, the identified areas of professionalism were comparable to those recognised by ABIM and other instruments with diverse backgrounds.This suggests that these are universal characteristics, but these items were valued as per the local contextual requirement by the participants.Effective display of a respectful attitude is extremely necessary and is desirable when dealing with peers, patients and seniors.Concerning duty, it was agreed that it should be reflected in the actions of students, such as assisting junior staff with the patients and finishing assignments on time.During high-pressure situations, the students should exhibit the element of 'honour and integrity' characterised as one area in the present tool.In the ABIM framework, they are considered separate domains. 2 This might be because the concept might be considered as having similar meaning in the local context since during the FGD, the participants' views frequently overlapped.
Regarding excellence, it was established that students should wear proper attire.A study concluded that proper dressing of the physicians improves patient satisfaction. 16octors must behave in a dignified manner as they are not only praised for their skills but also for mannerisms.The doctor must show him/herself to be attentive, courteous neither aggressive nor insensitive as these are humanistic qualities.These are the attributes that are also reflected in the Hippocratic Oath and are important in establishing a patient centred professional care. 17Accountability is the desire to improve and hold oneself responsible for one's actions.
Compared to a prior tool developmental study on professionalism, the statements of the present tool were developed more methodically. 18That study evaluated the observations of different professional groups about professional behaviours and described 29 circumstantial statements on elements such as respect, accountability, honesty and integrity, empathy, self-directed learning, and skill communication. 18In the current study, FGD was carried out to construct micro-scenarios on areas of ABIM framework 2 of professionalism and validated through modified Delphi rounds.
The present tool can be compared to a tool developed earlier. 8That tool was designed to assess professionalism in an Arabian context whereas this tool is for Pakistani context.
The current study can be compared to a professionalism tool development study in Taiwan which had 34 items. 19he current tool has 21 items.For the earlier study, the expert panel was not specified and the methodology for piloting was not clear. 19The current study had a 9-member panel for FGD.Furthermore, the technique of pre-cognitive testing was used to evaluate the tool.A preliminary study was also performed on 231 medical students.It is similarly comparable to the 36-item Penn State Professionalism Questionnaire, which has four parallel forms for medical students, residents, basic sciencs faculty, and clinical sciences faculty. 20The current tool is meant for clerkship undergraduate medical students.
For a tool, determining the content validity is essential, which if not acceptable may not be insightful of the intended group.ICV-I, S-CVI and I-CVI/Avg was calculated which was 0.94.This feature of the study is in agreement with a study that assessed the characteristics of communication in professionalism. 21ny of the tool development studies do not report CFA for construct validity. 10The suitability of a hypothetical design is evaluated on the basis of whether or not it meets data statistics, providing theoretical justification.These indices are squared distribution with degree of freedom (x2/df), comparative fit index (CFI), Tucker-Lewis index (TLI), root mean square residual (SRMR) and standardised mean root square residual (SRMR). 22The basic value for fit indices, the CFI and TLI, is 0.90 and is compared with the proposed null model. 23The current tool had CFI and TLI indices 0.921 and 0.905, respectively, signifying acceptable fit.Root mean square error of approximation (RMSEA) is absolute fit and measures overall goodness of fit model.Values ≤0.05 are good, values ranging 0.05-0.08 are acceptable and value between 0.08 and 0.01 are marginal.However, it was 0.089 in the present study, so statistically the tool cannot be rejected as the CFI and the TLI were within the range. 23e current study identified that all the constructs expected to be exhibited by the medical students for the 2180 Open Access J Pak Med Assoc attainment of a professional behaviour befitting a doctor were correlated.This indicates that while all the domains of professionalism may be an important means of exhibiting professionalism, all affect each other.The direct correlation emphasises the importance of all the domains of professionalism.
The tool is meant for the faculty to measure the development of professionalism in medical undergraduate students who are about to exit the medical institutions.One of the strengths of the current study is that the microscenario statements were developed in the local context, bearing in mind the fact that the opinion of the practising faculty can differ in how the undergraduate students should exhibit professionalism.Through feedback at the undergraduate level, this will serve to standardise the professional quality of the students' conduct and attitude toward patient care.
When the different items of this tool were examined, the micro-scenario statements were developed to assess clinical competence and acceptable professional behaviour contextually.If the participants perceived the construct in the tool as something positive and felt that it was recommended by their surroundings, they were more likely to practice it.Comparing the outcomes of the present research to those of Maat et al. reveals a good model fit.However, the earlier study was conducted on high school students with a sample size of 1887. 24r assessing the reliability of a scale, the most widely used marker is Cronbach's alpha Even though its use and analysis can be subject to lapses.According to recommendations, it is important to assess the matrix of correlations of the individual items and to look for inter-item correlations. 25A maximum value of 0.90 can be accepted. 25Most of the items displayed good alpha reliabilities in the current study.
The current study has limitations, like the sample size was too small to be generalised and was taken from a single university.
After the tool's first implementation, it is possible to do follow-up research to see how it affected the students' professionalism.

Conclusion
The constructed tool contained 21 items.The tool's content validity, reliability and construct validity ranged from satisfactory to good.It can be used to evaluate medical student professionalism in the local context.Using this instrument in the clinical setting, it is envisaged that the professionalism of students in the local context can be evaluated to improve patient care.

Figure :
Figure: Sequential equation model for professionalism in medical students, showing factor loadings, factor co-relations and good for fit indices (parsimonious, absolute, and incremental fit) for 6-factor model containing 21 items.RP: Respect, Alt: Alturism, HI: Honour and Integrity, AB: Accountability, D: Duty, E: Excellence, x2/d: Squared distribution with degree of freedom, CFI: Comparative fit index, TLI: Tucker-Lewis index, GFI: Goodness of fit index, RMSEA: Root mean square error of approximation.
PIMS: Professionalism in medical students, n1: Number of participants in round 1, n2: Number of students in round2, M: Mean, %A: Percentage of agreement, I-CVI: Content validity index of item, CVR: Content validity ratio.