Utilizing cognitive interview in the item refinement of the Blended Teaching Assessment Tool (BTAT) for Health Professions Education

Abstract

Background: A high-quality measurement tool is essential to accurately assess the innovative teaching strategies in health professions education. The Blended Teaching Assessment Tool (BTAT) aims to evaluate quality blended teaching or instructional delivery in Philippine health science programs. However, there is a lack of studies examining students' cognitive processes to support the validity of questionnaires.

Methodology: Cognitive interviewing (CI) was employed to determine whether students interpreted and responded to the items correctly. Content analysis was done using Tourangeau's Cognitive framework. Four CIs were conducted by an expert moderator and note-taker with a total of 8 health science students (2 groups with 3 members, and 2 one-on-one interview) for around 1 to 2 hours via Zoom following a retroactive approach with verbal and spontaneous probing, guided by a semi-structured interview questionnaire.

Results: Various issues related to comprehension, retrieval, judgment, and response were identified, leading to significant revisions of the tool from 82 items across 8 dimensions to 53 items across 5 dimensions. The challenges included unfamiliar terminology, ambiguous phrasing, complex statements, inconsistencies and irrelevance to students' real-life experiences. These findings emphasize the importance of students' feedback in enhancing the validity and reliability of assessment tools.

Conclusion: The Cognitive Interview identified crucial issues in comprehension, retrieval, judgment, and response, making it essential for developing the Blended Teaching Assessment Tool and ensuring valid responses on the quality of blended teaching and learning delivery.