Evaluation Overview

Supportive Supervision and Resiliency

The QIC-WD evaluation was conducted with the support of the Ohio Department of Job and Family Services to determine if a Supportive Supervision and Resiliency intervention, known as Coach Ohio, was effective in improving workforce and child welfare outcomes. 

Research Questions

The site-level evaluation for Ohio was designed to understand implementation of and outcomes related to the Coach Ohio intervention and its component parts: (1) the ACCWIC Coaching Model for managers and supervisors to introduce the key principles of supportive supervision and (2) Resilience Alliance (RA) groups for supervisor and frontline workers to enhance coping in the face of exposure to trauma. 

Relationships among intervention components, outputs, and outcomes were mapped out in Ohio’s logic model. Initial research questions of interest included how well the components of the intervention were executed with fidelity, in the form of dosage, adherence, quality, and participant engagement. The efficacy evaluation addressed a substantial number of research questions about short- and medium-term outcomes and included:

  • To what extent did the intervention lead to a workforce with higher levels of: (1) emotional regulation, (2) coping, (3) optimism, (4) resilience, (5) work-life balance, and (6) perceived social support among those in the intervention vs. comparison condition? And to what extent did the intervention lead to changes in these variables pre to post intervention?
  • To what extent did the intervention lead to a workforce with lower levels of: (1) secondary trauma and (2) burnout? And to what extend did the intervention lead to changes in these variables pre to post intervention? 
  • To what extent did the intervention lead to a workforce with higher levels of: (1) job satisfaction and (2) intent to stay and lower levels of: (3) intent to leave and related leaving behaviors, and (4) absenteeism among those in the intervention vs. comparison condition? And to what extent did the intervention lead to changes in these variables pre to post intervention period? To what extent did the intervention affect organizational culture and climate, particularly in the targeted areas of rigidity, resistance, engagement and stress? 
  • How did uptake of Coach Ohio and changes in the attitudes and behaviors described above vary by demographic and personality characteristics? 

The evaluation also addressed longer-term research questions about the effect of the Coach Ohio intervention compared to those in the comparison condition on turnover and child and family engagement and permanency/safety outcomes.

Evaluation Design

The process evaluation was descriptive in nature and assessed: (1) intervention fidelity of the ACCWIC Coaching Model training through observations and facilitator self-assessments and (2) RA intervention fidelity including dosage (e.g., participant attendance rates across the 24 RA sessions), adherence to the content of each RA session by facilitators (using periodic observations and facilitator self-ratings of each session), and participant satisfaction with the content of RA and quality of facilitation of RA sessions (every three weeks). 

A quasi-experimental design was employed to evaluate the effectiveness of Coach Ohio (see diagram). Staff in four counties and half of the staff in a large urban county received the intervention while staff in three other counties and the second half of staff in the large urban county did not and served as part of the comparison group (see the Site Overview for a list of counties). Both survey and administrative data from child welfare and human resources systems were collected for intervention and comparison participants.

Counties that agreed to be in study


Measures included personality and attitude scales that might mediate uptake or outcomes as well as all short- and medium-term outcomes of perceived work stress, secondary traumatic stress (STS) and burnout, as well as emotional regulation, coping behaviors, optimism, resilience, perceived social support and organizational culture and climate. Long-term outcomes of workforce turnover and case-based family engagement as well as child safety and permanency were measured using child welfare and human resource administrative data extracts. Survey data were collected either in-person or using a secure online platform. Survey data were linked together and with administrative data for mediation analyses. All data were stored and managed in a secure location, accessible only to the QIC-WD evaluation team and the site data coordinator. 

Timeline

A baseline organizational culture and climate context was administered to over 550 frontline staff in the nine participating counties in January 2018 which informed creation of the intervention.  Pre-intervention baseline surveys were administered between late January and early March 2019.  The mid-RA survey was administered in September 2019.  The immediate post-follow-up survey was administered in December 2019. The second follow-up survey to assess organizational culture and climate and additional outcomes was administered between February 2020 and July 2020. The third and final follow-up survey was administered between May 2021 and July 2021. Administrative data were transmitted to the evaluation team periodically starting in the fall of 2020. Both HR and child welfare data were extracted from the two years before the QIC-WD engaged the site 2016, 2017, the three years the site was engaged to conduct the needs assessment, develop the intervention, roll out the intervention, implement the intervention and follow-up on the intervention- 2018, 2019 and 2020- and the two years following the intervention 2021 and 2022.