Designing Rigorous Tests of Workforce Interventions in Complex Environments
Public child welfare agencies continually struggle with how to address issues of staff recruitment and retention. Furthermore, very little evidence exists regarding what works to attract, select, and retain talented staff. Much of what agencies do to address workforce issues is never formally evaluated and successes are not published widely. Thus, agencies confronted with urgent retention needs have little to go on to identify an intervention that is likely to succeed. The Quality Improvement Center for Workforce Development (QIC-WD) was funded by the Children’s Bureau to begin to address this issue, in part through carefully selected, and rigorously evaluated interventions in a variety of sites across the country.
The QIC-WD is testing workforce interventions in eight partner sites across the country with the goal of improving staff retention and, ultimately, child welfare outcomes. The QIC-WD is working with a diverse group of agency partners including a tribal child welfare agency, a large urban county, groups of counties in state-supervised, county-administered child welfare systems, and state-administered systems. The workforce strategies being tested are also diverse. Strategies include telework, mobile technologies, competency-based employee selection, job redesign, onboarding, and interventions focused on secondary traumatic stress, supportive supervision, and organizational culture and climate. Furthermore, sites vary widely in readiness to implement rigorous evaluation.
Each site worked with the QIC-WD to select an intervention and the QIC-WD evaluation team designed site-specific studies based on the intervention and structure of the site. A variety of evaluation approaches are being used in the partner sites, ranging from randomized controlled trials to rigorous multi-method studies. The evaluation team also selected several tools that measure variables shown to be predictors of turnover. These tools were identified in the research literature and allow the evaluation team to use common definitions and methods to synthesize findings across QIC-WD sites.
The process of designing the evaluations highlighted challenges with data collection, building relationships, navigating local contextual factors (e.g., history, culture, resources, politics, geography, other initiatives), and the complexities of collecting and working with existing and institutional data across multiple systems with varying resources and requirements. The QIC-WD also encountered challenges conducting research using human resources (HR) data, which are often managed outside the child welfare agency and use different identifiers than those used to identify workers in case-based data systems. For example, only one out of the 26 jurisdictions that applied to work with the QIC-WD had a common identifier to enable evaluators to link child welfare practice data with HR data. QIC-WD teams also found that some sites had multiple initiatives underway, some of which target the same set of workforce outcomes the QIC-WD project is trying to address. These challenges can threaten the validity of conclusions drawn regarding the selected interventions.
The QIC-WD team developed a range of strategies to address these challenges.
Site teams: QIC-WD evaluators work closely with site implementation teams which include a site implementation manager (SIM) and data coordinator (DC). These team members help with the logistics of data collection and the production of administrative data files for HR and child welfare data.
Common Measures: The QIC-WD evaluation team also developed templates to facilitate building a common set of variables, drawn from sites’ administrative data, that allow for cross-site analyses of the relationship between workforce turnover and child and family outcomes. The QIC-WD is using an extract, transfer and load (ETL) tool to efficiently combine site-specific data extracts into a single database.
Risk Management: Evaluators regularly engage site implementation teams in identifying emerging issues that may pose a risk to the QIC-WD’s ability to draw valid conclusions from the evaluation. The team helps sites develop risk management plans that address issues such as fidelity, attrition due to turnover in study populations, and diffusion of aspects of the intervention to control groups. For example, a risk management plan in one site addresses turnover in the intervention (experimental) units by filling in with a worker who has already completed training and is eligible to carry a full caseload. When this is not possible, a team of trainers and coaches meets individually with the new staff member to provide information and support around (1) child welfare work, in general, and (2) their specific roles and responsibilities related to the intervention.
The field of child welfare needs a systematic, data-informed approach to testing workforce interventions. However, it is not easy to conduct rigorous studies in complex, dynamic, public child welfare agencies. The QIC-WD evaluation team is using a range of evaluation approaches to better understand the underlying causes of workforce problems in child welfare, and the effectiveness of interventions targeted to specific agency environments, needs, and capacities, while navigating these complex systems.