As mentioned in our Customer Experience (CX) Principles, USDOL is working with states to identify pain points claimants experience when applying for unemployment insurance benefits, including technical issues that could prevent them from successfully filing a claim.
We are identifying these pain points through a suite of tools, including (but not limited to) quantitative data analysis, survey design and deployment, and user research. While quantitative data (obtained through analytics platforms) and survey data can provide large datasets for analysis, we also recommend exploring qualitative user experience research methods like usability testing or observational research to bring extra detail, granularity, and depth to your data.
When planning for research, it is crucial to design the study in a way that is sustainable for your staff and potential research participants. We recently collaborated with the Illinois Department of Employment Security to conduct observational research with UI claimants in Illinois, while creating a repeatable process for continued research in the future.
Identify research goals
USDOL collaborated with the IDES Assistant Director for Service Delivery to identify our research goals. After implementing a new customer experience survey and looking at app analytics, IDES leadership still had questions about where claimants were experiencing the most critical issues. The team distilled the remaining questions down to these core research questions:
1. Which specific touchpoints in the UI claim filing experience are burdensome and confusing for claimants, preventing them from finishing their application (or from applying correctly)?
2. What aspects of claimants’ experiences matter most to successfully completing, submitting, and receiving UI benefits?
3. Prior to applying, where and how do claimants seek information about the initial UI application process and UI benefits in general?
With these research goals in mind, the team decided it made the most sense to identify specific pain points, and learn about claimants' overall preparation process, by observing actual UI customers going through the claim filing process in real-time.
Recruitment
We took two approaches in reaching out to claimants and inviting them to participate in the research; we recruited select visitors from benefits.gov, and also collaborated with IDES field offices to recruit claimants who were already scheduling an appointment to receive assistance filing claims. The latter strategy proved to be most successful, with five field offices working with us to recruit claimants.
USDOL and IDES also collaborated to develop an informed consent form, ensuring research participants understood that:
1. Participating in the research was completely voluntary
2. Participating (or not participating) would not impact the outcome of their UI claim
3. Personally identifiable information (PII) captured by the state’s recordings of the research sessions would be protected and the PII would be removed from the recordings by the state
4. They could opt out at any point in the research session, whether or not their session was complete
The baseline target was to recruit five participants for research in the span of a week—with the plan of observing each claimant go through the online claim filing process from start to finish. With the support from IDES field offices, thirteen participants were recruited for observation and research. This is an especially strong result for an initial recruiting effort, and more than enough claimants for us to learn valuable insights and continue refining the application experience.
Conduct the research
USDOL and IDES condensed the research into a single week—each day collaborating with a different field office to recruit participants in-person. After arriving at the field office for their scheduled appointment, prospective participants were asked if they would be interested in participating in the research and given some time to review and sign the informed consent form. Claimants then filed their UI application from an IDES computer (With an IDES field office staff member stood by to answer questions), while additional IDES staff members and USDOL researchers observed remotely.
For the most part, field office staff guided claimants through the claim filing process step by step, answering claimant questions when necessary. Claimants’ questions were captured by observing researchers, along with key moments where claimants were unable to proceed without staff intervention. These questions and observations directly linked back to the core research questions identified by the team above, helping IDES and DOL to pinpoint key friction points in the IDES claim filing process.
The value of observational research with unemployment insurance claimants
In the process of observing UI claimants going through the claim application process, USDOL researchers and IDES leadership were able to answer the core research questions posed above, bringing detail to support existing quantitative data and survey data and providing actionable insights. These insights are directly informing IDES’ IT modernization efforts, providing some clarity into which CX investments might provide the most immediate benefit for both claimants and the IDES organization. We are very grateful for our partnership with the Illinois Department of Employment Security, and especially grateful for the dedicated field office staff in Peoria, Rockford, Harvey, Pilsen, and Mt. Vernon.
Resources like the DHS Usability Testing Kit and the 18F UX Guide are an excellent place to start for state workforce agencies looking to find more guidance around the mechanics and best practices for usability testing and observational research in the public sector. Usability testing, along with other CX practices like identifying core metrics and survey design, will ensure your agency is actively building and maintaining IT systems that efficiently and actively serve the needs of your claimants.