Design Options for an Impact Evaluation of the Employee Benefits Security Administration’s Participant Assistance Program Final Report
Related Tags
Topic
Research Methods
Study Population
DOL Partner Agency
Country
About the Report
The Chief Evaluation Office (CEO) at the U.S. Department of Labor (DOL) contracted with Mathematica Policy Research to conduct a study to explore potential research designs for determining the impact of the participant assistance program administered by the Office of Outreach, Education, and Assistance (OEA) within DOL’s Employee Benefits Security Administration (EBSA). Through its participant assistance program, trained Benefits Advisors (BAs) work in the agency’s field offices to provide outreach and direct assistance to employees, employers, benefit plan sponsors, service providers, and other stakeholders who contact EBSA with benefits-related issues.
The study involved development of three key components: (1) a program logic model, (2) designs for an objective evaluation to study the impact of the program, and (3) suggestions for refinements to OEA’s performance measurement process. The report focuses on two potential impact evaluation designs that would test—in slightly different ways—the impact of receiving a referral to EBSA’s website compared with receiving BA assistance through the telephone hotline, which is currently the predominant mode inquirers use to access services. OEA has indicated a desire to increase participants’ use of its website to acquire information to resolve inquiries on their own and to submit web inquiries for further assistance. If this approach were successful at enabling participants to resolve relatively simple questions on their own without BA assistance, this could allow BAs to focus on inquiries that require specialized knowledge of benefits-related laws. In addition, inquiries submitted through the website require less data entry on the part of BAs, resulting in less time spent per inquiry. This report discusses implementation considerations for this type of evaluation, data collection needs, potential evaluation sample sizes, and the strengths and drawbacks of the two service delivery options.
Research Questions
- What is the impact of referring inquirers to the website (either with or without a telephone follow-up option) on their customer satisfaction and knowledge of and access to their entitled pension/health benefits, compared with receiving services through the telephone hotline?
Research Gaps
- An impact evaluation of these models would answer the research question "What is the impact of referring inquirers to the Employee Benefits Security Administration website on their customer satisfaction and knowledge of and access to their entitled pension/health benefits, compared with receiving Benefit Advisor assistance through the telephone hotline?" (page vii)
Citation
Mastri, A., Bellotti, J., Bruch, J., Roemer, G., Puri, B. (2013). Mathematica. Design Options for an Impact Evaluation of the Participant Assistance Program: Final Report. Chief Evaluation Office, U.S. Department of Labor.
The Department of Labor’s (DOL) Chief Evaluation Office (CEO) sponsors independent evaluations and research, primarily conducted by external, third-party contractors in accordance with the Department of Labor Evaluation Policy and CEO’s research development process.