Improving inter-rater reliability of the enhancing assessment of common therapeutic factors (ENACT) measure through training of raters
Accepted: 15 May 2022
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.
Background. As evidence supports task-shifting approaches to reduce the global mental health treatment gap, counselor competency evaluation measures are critical to ensure evidence-based therapies are administered with quality and fidelity. Objective. This article describes a training technique for evaluating lay counselors’ competency for mental health lay practitioners without rating scale experience.
Methods: Mental health practitioners were trained to give the Enhancing Assessment of Common Therapeutic Factors (ENACT) test to assess counselor proficiency in delivering the Common Elements Treatment Approach (CETA) in-person and over the phone using standardized video and audio recordings. A two-day in-person training was followed by a one-day remote training session. Training includes a review of item scales through didactic instructions, active learning by witnessing and scoring role-plays, peer interactions, and trainer observation and feedback. The trainees rated video and audio recordings, and ICC values were calculated.
Results: The training technique presented in this research helped achieve high counselor competency scores among lay providers with no prior experience using rating scales. ICC rated both trainings satisfactory to exceptional (ICC: .71 - .89).
Conclusions. Raters with no past experience with rating scales can achieve high consistency when rating counselor competency through training. Effective rater training should include didactic learning, practical learning with trainer observation and feedback, and video and audio recordings to assess consistency.
Santomauro DF, Herrera AM, Shadid J, et al. Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic. Lancet. 2020; 398. Available from: https://doi.org/10.1016/S0140-6736(21)02143-7
Rhem J, Shield KD. Global Burden of Disease and the Impact of Mental and Addictive Disorders. Curr. Psychiatry Rep. 2019; 21(10). Available from: https://doi.org/10.1007/s11920-019-0997-0
Wainberg ML, Scorza P, Shultz JM, et al. Challenges and Opportunities in Global Mental Health: a Research-to-Practice Perspective. Curr Psychiatry Rep. 2017; 19(5): 28-34. Available from: doi:10.1007/s11920-017-0780-z
Lund C, Alem A, Schneider M, et al. Generating evidence to narrow the treatment gap for mental health disorders in sub-saharan Africa: rationale, overview and methods of AFFIRM. Epidemiol Psychiatri Sci. 2015; 24: 233-240. Available from: Doi:10.1017/S2045796015000281
Branson A, Myles P, Mahdi M, Shafran R. The relationship between competence and patient outcome with low intensity cognitive behavioural interventions. Behav cogn psychother. 2018; 46(1): 101-114. Available from : https://doi.org/10.1017/S1352465817000522
Liness S, Beale S, Lea S, et al. Multi professional IAPT CBT Training: Clinical Competence and Patient Outcomes. Behav and Cogn Psychother. 2019. Available from: DOI: 10.1017/S1352465819000201
Brown RC, Southern-Gerow M, McLeod BD, et al. The Global Therapist Competence Scale for Youth Psychosocial Treatment: Development and Initial Validation. J. Clin. Psychol. 2019; 74(4): 649-664. Available from: Doi:10.1002/jclp.22537
McLeod BD, Southam-Gerow MA, Rodriguez A, et al. Development and Initial Psychometrics for a Therapist Competence Instrument for CBT for Youth Anxiety. J Clin Child Adolesc Psychol. 2018; 47(1): 47-60. Available from: doi:10.1080/15374416.2016.1253018
Muse K, McManus F. A systematic review of methods for assessing competence in cognitive-behavioural therapy. Clin. Psychol. Rev. 2013; 33(3): 484-499. Available from: Doi: 10.1016/j.cpr.2013.01.010
Bjaastad JF, Haugland BSM, Fjermestad KW, et al. Competence and Adherence Scale for Cognitive Behavioral Therapy (CAS-CBT) for Anxiety Disorders in Youth: Psychometric Properties. Psychol. Assess. [Advance online publication]. 2015. Available from: http://dx.doi.org/10.1037/pas0000230
Ottman K, Kohrt BA, Pedersen G, Schafer A. Use of Role-plays to Assess Therapist Competency and its Association with Client Outcomes in Psychological Interventions: A Scoping Review and Competency Research Agenda. 2019. Available from: https://www.sciencedirect.com/science/article/pii/SOOO5796719302177
Kohrt BA, Jordans MJD, Rai S, et al. Therapist competence in global mental health: Development of the Enhancing Assessment of Common Therapeutic factors (ENACT) rating scale. Behav Res and Ther. 2015; 69: 11-21
Glanzman AM, Mazzone ES, Young SD, et al. Evaluator Training and Reliability for SMA Global Nusinersen Trials. J. Neuromuscul Dis. 2018; 5: 159-166. Available from: DOI 10.3233/JND-180301
Schmidt LK, Andersen K, Nielsen AS, Moyers TB. Lessons learned from measuring fidelity with the Motivational Interviewing Treatment Integrity code (MITI 4). J. Subst. abuse treat. 2019; 97: 59-67
Kohrt BA, Schafer A, Willhoite A, et al. Ensuring Quality in Psychological Support (WHO EQUIP): developing a competent global workforce. World Psychiatry. 2020; 19: 115-6 doi: 10.1002/wps.20704
Murray LK, Dorsey S, Haroz E, et al. A common elements treatment approach for adult mental health problems in low- and middle-income countries. Cogn. Behav. Pract. 2014; 21: 111-123. Available from: https://doi.org/10.1016/j.cbpra.2013.06.005
Murray LK, Kane JC, Glass N, et al. Effectiveness of the Common Elements Treatment Approach (CETA) in reducing intimate partner violence and hazardous alcohol use in Zambia (VATU): A randomized controlled trial. PLoS Med. 2020; 17(4). Available from: https://doi.org/10.1371/journal.pmed.1003056
United Nations, Department of Economic and Social Affairs, Population Division. World Population Prospects. 2019. Volume 2: Demographic Profiles (ST/ESA/SER.A/427)
Munakampe NM. Strengthening Mental Health Systems in Zambia. Int. J. Ment. Health Syst. 2020; 14(28). Available from: https://doi.org/10.1186/s13033-020-00360-z
Kohrt BA, Mutamba BB, Luitel NP, et al. How competent are non-specialists trained to integrate mental health services in primary care? Global perspectives from Uganda, Liberia, and Nepal. Int. Rev. Psychiatry. 2018; 30(6): 182-98. Doi: 10.1080/09540261.2019.1566116
Kohrt BA, Ramaiya MK, Rai S, et al. Development of a scoring system for non-specialist ratings of clinical competence in global mental health: a qualitative process evaluation of the Enhancing Assessment of Common Therapeutic Factors (ENACT) scale. GMH. 2015; 2(23): 1-16. Available from: Doi:10.1017/gmh.2015.21
Koo TK, Li MY. a Guideline of Selecting and Reporting Intra-class Correlation Coefficients for Reliability Research. J. Chiropr. Med. 2016; 15: 155-163
Sadler ME, Yamamoto RT, Khurana L, Dallabrida SM. the impact of rater training on clinical outcomes assessment data: a literature review. Int. J. Clin. Trials. 2017; 4(3): xxx-xx
Shweta, Bajpai RC, Chaturvedi HK. Evaluation of Inter-Rater Agreement and Inter-Rater Reliability for Observational Data: An Overview of Concepts and Methods. J. Indian Acad. Appl. Psychol. 2015; 41(3) Special issue: 20-27
West MD, Daniel DG, Opler M, et al. Consensus Recommendations on Rater Training and Certification. Innov Clin Neurosci. 2014; 11(11): 10-13
Rohan KJ, Rough JN, Evans M, et al. A protocol for the Hamilton Rating Scale for Depression: Item Scoring Rules, Rater Training, and Outcome Accuracy with Data on its Application in a Clinical Trial. J. Affect. Disord. 2016; 200: 111-118. Available from: Doi:10.1016/j.jad.2016.01.051
Kobak K, Williams JBW, Engelhardt N, Lipsitz J. Rater Training in Multicenter Clinical Trials. J. Clin. Psychopharmacol. 2004. Available from: doi:10.1097/01.jcp.0000116651.91923.54 PubMed
Opler MGA, Yavorsky C, Daniel DG. Positive and Negative Syndrome Scale (PANSS) Training: Challenges, Solutions, and Future Directions. Innov Clin Neurosci. 2017; 14(11): 77-81
Asan O, Montague E. Using video-based observation research methods in primary care health encounters to evaluate complex interactions. Informatics in primary care. 2014; 21: 161-170. Available from: Doi:10.14236/jhi.v21i4.72
Gulgin H, Hoogenboom B. The Functional Movement Screening (FMS): An inter-rater reliability study between raters of varied experience. Int. J. Sports Phys. Ther. 2014; 9(1): 1-7
Sajatovic M, Gaur R, Tatsuoka C, et al. Rater training for a multi-site, international clinical trial: What mood symptoms may be most difficult to rate? Psycho pharmacol Bull. 2011; 44(3): 5-14
Copyright (c) 2022 Mwamba M. Mwenge, Caleb J. Figge, Kristina Metz, Jeremy C. Kane, Brandon A. Kohrt, Gloria A. Pedersen, Izukanji Sikazwe, Stephanie Skavenski Van Wyk, Saphira M. Mulemba, Laura K. Murray
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.