About the Study
In 2019, the Chief Evaluation Office (CEO) partnered with the Employment and Training Administration, Office of Unemployment Insurance (OUI) to conduct the Behavioral Interventions to Improve Work Search Among Unemployment Insurance Claimants project. The unemployment insurance (UI) program requires claimants to be actively looking for work while they receive benefits to encourage a rapid return to work. However, multiple barriers—such as program complexity and lack of knowledge of program processes and requirements—may reduce claimants’ compliance with this mandate, which can result in inaccurate benefits payments. As part of CEO’s broader Behavioral Interventions in Labor Programs Evaluation portfolio project with Mathematica, these impact evaluations aimed to assess the effectiveness of behaviorally-informed communications – such as a pop-up alert and emails – in increasing UI claimants’ compliance with work search requirements.
From 2019 to 2021, researchers developed behavioral interventions to test with UI claimants in two states, Washington and North Carolina, that were motivated to experiment and learn how behavioral interventions might be used to reduce improper payment rates. The studies undertook a multi-method, multi-phased approach to support iterative learning. In Washington, a randomized control trial (RCT) involving 26,967 claimants compared knowledge and work search behaviors between a treatment group who received the intervention and a control group who did not. Outcomes were assessed using administrative data as well as qualitative information from phone interviews with staff.
In North Carolina, an RCT involving 24,416 claimants explored the communications’ influence on self-attested work search compliance, while a series of quasi-experimental design (QED) studies explored other outcomes. Outcomes were assessed using administrative data as well as qualitative information from phone interviews with staff and in-person focus groups with staff and claimants.
The COVID-19 pandemic and related policy changes shortened data collection periods in both states and caused disruptions to study plans, collaborative efforts, and workforce priorities across the United States. Quantitative findings suggest limited impacts. However, qualitative findings on implementation suggest work search may nonetheless be a promising area for continued testing of behavioral insights.
This Department of Labor (DOL)-funded study was a result of the annual process to determine the DOL’s research priorities for the upcoming year. It contributes to the labor evidence-base to inform various programs and policies and addresses Departmental strategic goals and priorities.
- The effect of the COVID-19 pandemic limited what we were able to learn in these behavioral studies. With the onset of the COVID-19 pandemic in March 2020, many states paused work search requirements. This limited some aspects of data collection in the studies and prevented the possibility of iterative design and further testing that is typical in behavioral trials.
- Quantitative data suggests that the interventions had limited impact. Administrative data in both states show no statistically significant impact on claimants’ knowledge of work search requirements, as measured through self-attestation of work search or work search behavior. Although most participants were exposed to the main intervention (in North Carolina, a weekly pop-up alert or hyperlink to a tip sheet outlining what counts as valid employer contacts and adequate proof for different modes of contact; in Washington, a one-time email sent to claimants), only a small share clicked on embedded links to access additional information.
- Qualitative findings indicate that the UI work search presents a promising area for additional testing of behavioral interventions. Claimant focus groups in North Carolina mentioned several behavioral barriers, including low understanding of how to log work search efforts and provide supporting documentation, and suggested additional barriers (e.g., maintaining morale) which could inform future research. This feedback may not be generalizable to all UI claimants, however, since the focus groups consisted of individuals attending mandated UI appointments and their views may differ from those of individuals who did not attend their appointments.
- These trials demonstrate the feasibility of quickly deploying behavioral interventions whose effectiveness is measured with administrative data. Many systems that facilitate the UI claims process and claimant communications can be customized for targeted deployment and make data accessible, which makes rigorous evaluation feasible. Study designs in North Carolina and Washington had to balance rigor with the states’ goals of quick turnaround times and interest in scaling the interventions.
- State administrative data can be useful but requires investment to assess potential. Given sample size and other limitations with the Benefits Accuracy Measurement (BAM) data, other sources of state administrative data were used to measure some work search knowledge and behavioral outcomes. However, to determine data options, quality, benefits, and limitations, the study team had to build trust with the states and spend time with state staff to understand data collection and validation processes.
- Early engagement activities with program staff as well as users (non-staff) may improve intervention design. In both states, early site visits allowed researchers to contextualize the spectrum of opportunities for possible interventions, and produced new insights. Site visits also forged new relationships with researchers and allowed access to data and key program stakeholders. However, investing in gathering user (non-staff) perspectives early and at other junctures may also be important to assess whether programmatic assumptions and hypotheses are correct.
- State partners reported the collaborative design process to apply behavioral principals was useful. In North Carolina, state partners scaled up the use of the behavioral intervention materials from the trial with new claimants and have demonstrated ongoing interest in applying behavioral insights to improve operations. In Washington, staff reported that learning to apply behavioral insights has increased their ability to quickly revise communications going forward.
Amin, S., Chojnacki, G., Congdon, B., Davis, S., Langan, A., Deutsch, J., Welch, E., Spitzer, A., Johnson, A. (2022). Mathematica. Behavioral Interventions to Improve Work Search Among UI Claimants: Results from North Carolina and Washington. Chief Evaluation Office, U.S. Department of Labor.
The Department of Labor’s (DOL) Chief Evaluation Office (CEO) sponsors independent evaluations and research, primarily conducted by external, third-party contractors in accordance with the Department of Labor Evaluation Policy. CEO’s research development process includes extensive technical review at the design, data collection and analysis stage, including: external contractor review and OMB review and approval of data collection methods and instruments per the Paperwork Reduction Act (PRA), Institutional Review Board (IRB) review to ensure studies adhere to the highest ethical standards, review by academic peers (e.g., Technical Working Groups), and inputs from relevant DOL agency and program officials and CEO technical staff. Final reports undergo an additional independent expert technical review and a review for Section 508 compliance prior to publication. The resulting reports represent findings from this independent research and do not represent DOL positions or policies.