Equivalence of entrustable professional activities and context-dependent item sets as summative assessments in undergraduate physical therapy programs

Abstract

Background: Summative assessment of student performance should provide information on achievement of program outcomes to support evaluation decisions. Alternative approaches to the traditional assessment
systems like the written licensure examinations in Physical Therapy (PT) should be explored to ensure valid
measurement of achievement of these terminal outcomes.

Objective: The study aimed at establishing equivalence of two summative assessments new to PT in
measuring achievement of the PT outcomes: work-based assessment using Entrustable Professional
Activities (EPA) and knowledge-based assessment using Context-Dependent Item Sets (CDIS).

Methodology: Thirty-two newly graduated PT's underwent a one-week EPA assessment and took a 102-item CDIS test (based on 14 clinical vignettes). Qualitative data from blueprint review, group face-to-face
interviews with participants and assessors, and field notes from observations, and quantitative data from EPA entrustment decisions and CDIS scores were utilized to ascertain their comparability in terms of Purpose, Administration, Quality and Decisions. This was used to determine the extent of equivalence of the two assessments.

Results: Review of both blueprints show alignment with PT outcomes, with integrative content motivating participants towards professional development. Administration were equally acceptable to users, though EPA had more practice opportunities with a longer assessment time. Entrustment decisions in EPA had a high inter-rater reliability, while CDIS had low reliability, with most items having poor discriminative power. Decisions of “pass” or “fail” had good concordance when high prevalence indices were considered.

Conclusion: There is high extent of equivalence in purpose of EPA and CDIS but are not equivalent in terms of administration. There is moderate equivalence in quality and decisions, with potential for increased concordance if improved quality of CDIS is attained.