Validity Evidence for the Physician Oral Language Observation Matrix as a Measure of Medical Spanish Proficiency

Background

The lack of a standardized language assessment process for medical students and physicians communicating in a non-English language threatens healthcare quality and safety.

Objective

To evaluate the validity of a new rating tool, the Physician Oral Language Observation Matrix (POLOM)™, in assessing medical students’ oral communication with Spanish-speaking standardized patients (SPs).

Design

POLOM scores were compared to measures of student medical Spanish proficiency to examine convergent validity and to measures of clinical performance to examine concurrent/criterion validity.

Participants

Forty-two students at two schools completed SP encounters between January 2021 and April 2022, and POLOM raters scored the videorecorded performances between January and June 2022.

Main Measures

Two approaches to generating POLOM total scores were investigated: rater average and strict consensus. Convergent validity was examined via the POLOM’s correlations with (1) the phone-based Clinician Cultural and Linguistic Assessment (CCLA) and (2) the self-rated Interagency Language Roundtable scale for healthcare (ILR-H). Concurrent/criterion validity was examined via correlations with (1) the Comunicación y Habilidades Interpersonales (CAI) scale, (2) a checklist completed by the SP, and (3) a faculty rating of the student’s post-encounter clinical note. Pearson’s correlations of r ≥ 0.5 and r ≥ 0.2 were considered evidence of convergent validity and concurrent/criterion validity, respectively.

Key Results

Both rater average and strict consensus POLOM scores were strongly correlated with ILR-H (r = 0.72) and CCLA (r ≥ 0.60), providing evidence of convergent validity. The POLOM was substantially correlated with the CAI (r ≥ 0.29), the SP Checklist (r = 0.32), and the faculty scoring of the student’s clinical note (r ≥ 0.24), providing concurrent/criterion validity evidence.

Conclusions

The POLOM has demonstrated evidence of convergent and concurrent/criterion validity as a measure of medical students’ Spanish proficiency during SP encounters. Additional research is needed to evaluate how the POLOM can be implemented with resident and practicing physicians, applied to other health professions, and adapted to other languages.