Test Evaluation Practices

In this area of research, we have published the results of two studies. The first explored the test evaluation practices of researchers in a sample of four testing-oriented journals. Findings suggest that psychological researchers employ a variety of procedures and rationales when analyzing the psychometric properties of their test data, some sound, but many apparently not grounded in a coherent test evaluation logic. More specifically, researchers appear, in general, to handle the assessment of external test score validity quite well, but the internal test score validity and score precision assessment practices remain inconsistently or insufficiently handled. Moreover, the extent to which researchers coherently integrate evaluation of different psychometric properties suggests that many have only a superficial understanding of the important ways in which these seemingly distinct psychometric properties are interrelated (Slaney, Tkatchouk, Gabriel, & Maraun, 2009).

In the second paper, we examined whether the test evaluation practices and rationales employed by researchers differ substantially between researchers with some expertise or interest in quantitative psychology versus those lacking such expertise and interests. The findings of this study indicate that, contrary to expectations, articles published in measurement-oriented journals, as compared with general journals, generally may not reflect better psychometric analysis and reporting practices or sounder test-analytic rationales on the part of the researchers. Moreover, we found that although most researchers will evaluate either score precision or validity (typically external score validity, as opposed to internal or “structural” validity), they seldom evaluate both, suggesting that researchers may not appreciate well the importance of conducting a full and coherent data-based test analysis whenever a measure is employed (Slaney, Tkatchouk, Gabriel, Ferguson, Knudsen, & Legere, 2010). We have applied this logic in a practical application within the field of risk assessment (Slaney, Storey, & Barnes, 2011).

Relevant Publications:

Slaney, K. L., Storey, J. E., & Barnes, J. (2011). “Is my test valid?”: Guidelines for the practicing psychologist for evaluating the psychometric properties of measures. International Journal of Forensic Mental Health10, 261-283 (target article).http://dx.doi.org/10.1080/14999013.2011.627086

Slaney, K. L., Tkatchouk, M., Gabriel, S.M., Ferguson, L.P., Knudsen, J.R.S., & Legere, J.C. (2010). A review of psychometric assessment and reporting practices: An examination of measurement-oriented versus non-measurement-oriented domains. Canadian Journal of School Psychology, 25(3), 246-259. DOI: 10.1177/0829573510375549

Slaney, K. L., Tkatchouk, M., Gabriel, S.M., & Maraun, M. D. (2009). Psychometric assessment and reporting practices: Incongruencies between theory and practice. Journal of Psychoeducational Assessment, 27(6), 465-476. DOI: 10.1177/0734282909335781