By: Justin Lai - April 7, 2023
Project team members
- Justin Lai – Design Research Lead, Office of Unemployment Insurance Modernization, USDOL
- Jennifer Phillips – Assistant Deputy Director for Service Delivery, IDES
- Dave Guarino – Technology Advisor, Office of Unemployment Insurance Modernization, USDOL
Introduction
In the Fall of 2022, the USDOL Office of Unemployment Insurance Modernization (OUIM) consulted with the Illinois Department of Employment Security (IDES) to provide hands-on support with IT modernization and customer experience strategy.
One of our primary discussions focused on making informed product, service, and business decisions based on qualitative and quantitative data— how might IDES leverage existing data streams to identify the most pressing technology issues in their unemployment insurance system, and how might IDES act upon this information in a timely and impactful manner?
Analyzing existing survey data
IDES already has an active voluntary satisfaction survey that pops up immediately following new UI claim submission. During the pandemic, IDES didn’t have the capacity to analyze the survey data of approximately 86,000 survey responses collected from 2020 – 2022, and survey data consisted of a mixture of Likert-scale style responses, multiple choice, and free-form open responses. IDES supplied the survey data to USDOL without any PII.
After initial review, the free-form response data seemed to be most compelling; answers to the question “What did you dislike about the online filing experience” were specific and actionable, but also vast—it would take months to manually sort through the 86,000+ responses to organize and identify the most commonly occurring claimant experience issues.
To quickly analyze the text at scale, the OUIM team used spaCy, a natural language processing library in Python. We wrote a python script looking for the most commonly occurring strings of adjectives + nouns in the survey data, creating a ranked list of the most frequently mentioned phrases by claimants in response to “What did you dislike about the online filing experience.”
The analysis output immediately identified a variety of specific claimant issues, ranging from question confusion, frustration with lack of mobile-device support, and prior technical issues that prevented some claimants from successfully filing their claims online. While IDES was already aware of many of these issues and had already taken steps to address them, the survey data analysis provided objective and complementary evidence to justify medium- to long-term IT investments.
Survey redesign
While natural language processing is a useful tool to capture insights on large-scale qualitative data, the most meaningful responses come from well-designed survey questions. IDES and OUIM worked together to redesign the new UI claims satisfaction survey, thinking about how the new questions and available responses might be tailored to help IDES better identify the most pressing claimant experience issues.
You’ll find the updated IDES new UI claims satisfaction survey below, with some rationale and explanation behind the survey design:
| Question | Answer type/scale | Rationale | |
| Q1 | How difficult was it to apply for unemployment insurance benefits online? | Very hard, hard, somewhat easy, easy, very easy | |
| Q2 | Is this your first time filing for unemployment insurance benefits? | Yes/No | This question is intended to identify any specific issues for first-time UI claim filers. For future data analysis, IDES can, in theory ,identify claimant experience issues that might disproportionately impact new UI claimants. |
| Q3 | Is this your first time filing online?
| Yes/No | The rationale for this question is like Q2, but focused on identifying claimant experience issues that might disproportionately impact UI claimants who are filing online for the first time. |
| Q4 | Where did you learn about how to apply for unemployment insurance benefits online? | Dropdown plus ‘other’ option Selection options:
| Rationale focuses on where and what sources claimants are using to learn about UI and how to apply. IL has a hunch it may not be the IDES website, and they want to learn more about the best ways to reach claimants. |
| Q5 | If you filed online at an Illinois workNet/American Job Center or an IDES local office, were you satisfied with the assistance you received? Conditional if yes Second level Q – what location? dropdown Third level Q – open comment (200 characters) | Y/N Location filed – dropdown (see excel file for list)
| Rationale includes being able to look at geographic information. Because Illinois has separate agencies for WIOA Title I and III, this data may help IDES think about current and future co-location opportunities. Right now, UI offices and Job Centers are not always co-located. Secondarily, we would like to know about the claimants actual experience if they filed online but at a physical office. |
| Q6 | Please share any part of the application process you liked | Open text box (200 character limit) | |
| Q7 | Please share any part of the application process you did not like. | Open text box (200 character limit) | This question was retained from the initial satisfaction survey, as the responses highlighted specific claimant experience issues. |
| Q8 | Were there any questions you did not understand? | Open text box (200 character limit) | This question was designed to capture any issues around plain language, looking to see if any specific questions might have confused claimants. |
Q9
| May we contact you to obtain additional information regarding your responses to this survey? | Y/N | This question was included to create an active participant pool for future user research exercises, such as usability testing, semi-structure interviews, etc. |
Outcomes and takeaways
The satisfaction survey redesign will be pushed to production in early February, and our hope is that the new survey data will be more representative of the claimant experience and actionable for IDES staff. Moving forward from this experience, we also have recommendations for other states who are looking to implement or improve existing surveys to evaluate customer experience:
- Use your program goals to inform your survey design: What are you trying to learn about claimants? What pieces of information might be most helpful in bringing clarity to your IT modernization strategy? Likert-style questions (scale of 1-5) might make analysis more straightforward, but practically mean little to claimants.
- Survey timeliness matters: Claimants who see a satisfaction survey immediately after filing haven’t yet received a decision on their claim-- they are unlikely to know if they filled out their information correctly. Use this opportunity to assess key performance indicators on successful online claim filing, such as time to complete, any major technical issues that might have prevented claimants from filing successfully, ease of use, etc. General questions about ease of use might be less productive if claimants filed their claim incorrectly but aren’t aware.
- Utilize open-ended questions to surface new issues: While surveys are a useful research tool to passively capture claimant data at scale, they can be more narrow and confined to pre-determined questions. Use free-form text responses/open-ended questions to give claimants the opportunity to surface new issues.
Below, you’ll find a testimonial from Jennifer Phillips, the IDES Assistant Deputy Director for Service Delivery. While the primary focus of this work was survey design and analysis, the ultimate goal here was to help states solicit claimant feedback and evaluate customer experience in a lightweight, immediate, and repeatable fashion.
How USDOL Office of Unemployment Insurance Modernization (OUIM) added value to IDES
- Access to expert help: OUIM provided a level of expertise on user experience that IDES doesn't currently have in house.
- Strategic thinking: OUIM provided a sounding board and consultative guidance on the strategic underpinnings of the survey itself (the why of which questions to ask) and the outcomes (what to do with this information).
- Making UI mod a priority: OUIM has been a conduit for sharing UI modernization best practices nationally and in other states. Through its offer to help Illinois, it pushes improvements to equitable user experience to the forefront and is enhancing IDES strategic thinking and implementation of Tiger Team and UI Equity Grant recommendations.
Interested in research help?
The USDOL OUIM team has resources on standby to support other state UI agencies like IDES who are interested in quickly expanding their capabilities in evaluative research methods (e.g., survey design and analysis), customer experience strategy, and IT modernization. Inquire at ui.modernization@dol.gov.